Netflix Eureka bug on elastic ip disassociate

I was checking out Eureka to see if we can apply it to Hoiio’s microservice. I downloaded the war from maven central, unzip it, tweak a few parameters and deployed it to aws. The version I was using was 1.4.4

Eureka recommended that we use EIPs for the server, but I didn’t want to expose my Eureka servers to public and since all of our servers were in one AWS region so I put them in my VPC and wrote a script to update the Route 53 configuration on instance launch instead.

So everything went smooth. After Eureka was deployed, I implemented a few services, registered them with Eureka and let them talk to each others. Until one day, I wanted to change several configs and restarted the Eureka servers. Things started to went wrong.

One of our elastic ip was suddenly disassociated. We didn’t know why, so we went to Cloudtrail to find the author of such action. At first we though it was a mistake made by some body in the team. Turn out it was a shutdown hook on Eureka causing the event. On shutdown, Eureka server tried to disassociate the elastic ip being bound with the instance. The eip is obtained by calling DiscoverAddressRequest by AWS SDK using the param of the instance’s public ip which in my case was null since I didn’t allow public ip on my servers. The result of a DiscoverAddressRequest with null public ip is not empty but a list of all elastic ips (link)! So in the end, instead of ignoring the disassociation for private instance, Eureka unbound the first EIP on the list!

First action was to remove EIP permission from the instance profile of Eureka. Then, I created a pull request on Eureka’s github. If the pull request is rejected, we will consider to bind unused EIPs to these instance, or worst case rebuild the Eureka server WAR file.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s