Graceful shutdown ec2 instance in aws autoscaling group

I have this experience today that I think worth sharing.

So we have this microservice system based on message passing on rabbitmq. For the deployment of each module, I wrote a few scripts to make sure each version of a module is placed on a different autoscaling group.

For each deployment, we keep the 2 versions of the module running for a period of time to make sure the new version does not break the system, then we stop the old version by scaling down the autoscaling group of the old version to 0.

The problem is I set rabbitmq timeout pretty high, and when autoscaling group terminates an instance on scaling down, the tcp connection from that instance to rabbitmq does not close until a timeout. RabbitMQ still delivers message to that dead connection, causing timeout for the user request since there are no consumer at the other end.
Continue reading

Advertisements

Make your life easier with unix symbolic link

Although it might seem straight forward, for a man who started using computer with Windows as the OS, I cannot help feeling amazed by the convinience of symbolic link on unix brings.

Most of the time I work on the shell, including browsing files. Symbolic link helps the browsing so much faster while keeping the directory structure so clean. The best thing is we can also use the symbolic link in path name to feed to the application without any issue. For example, I have this setup of 32GB SSD where I install the OS and the application packages and a 512GB HDD where I store data. The problem is I store pretty much data on the /home/ directory where I also have some binaries like .rvm/ or caches that I want to run on SSD. After sometimes, the the little 32GB SSD fills up and I have no space left so I simply migrate the /home directory to the HDD, then link the ~/.rvm, ~/.cache,… to /home/bin/ where I put the actual binaries files. It works without any issues.
Continue reading