For several years LiveRamp has operated applications in AWS with a combination of cloudformation stacks, bash/python scripts, and clicking around the management console. While these methods can work in some cases, it did not work for us in the long run — especially if we ever needed to recreate an entire VPC, configuration management stack, application stack, or to update an AMI to the latest packages. This inspired fear and loathing of our AWS stacks. No one wanted to touch or make changes to AMIs, ELBs, and other parts of AWS infrastructure unless it was absolutely necessary. Cloudformation stacks were confusing and our scripts had little error checking. We had no way to easily revert any changes to a known good state. To alleviate these concerns, we landed on a combination of HashiCorp’s terraform and packer.
Terraform the Cloud
At LiveRamp, we have started using terraform to manage some AWS layers including our VPCs, inhouse services, and applications. Terraform is declarative using configuration files. In other words, the configuration files describe what the infrastructure should be (examples). If those configurations are modified and applied, terraform determines how to transition the infrastructure from the current state to the configuration file defined state. These configurations are stored in git allowing us to collaborate on infrastructure changes via pull requests by which we can record changes, and then revert them if necessary. Terraform also allows for a “dry-run”, and can tell you all the changes that it is going to make, allowing you to then apply those reviewed changes to your infrastructure. Features such as these have given us greater confidence in making changes to our AWS infrastructure without incidentally causing outages.
In practice, terraform is easier to use than most tools out there. It does, however have a few drawbacks. For instance, not everything can be defined via configuration files, such as saving the state files to a remote location. We have created wrappers around terraform to add this necessary functionality until terraform does it natively. Additionally, it can be difficult to debug changes or find the source of the error when terraform has an issue with our configuration files. Despite these shortcomings, terraform is working well for us for the few services that we have migrated to AWS.
Bake Those AMIs
Packer is a tool to create reusable images for EC2 instances or templates for virtual machines. We use it to create AMIs for our EC2 instances with our application and OS configurations baked in via chef cookbooks. For each major release of a chef cookbook i.e. 1.2.3 to 2.0.0, packer starts an instance using the previous AMI, runs the chef cookbook, creates a new AMI based on this instance, and tags the new instances with chef cookbook version. For minor and patch releases, the cookbooks are updated on the local chef servers and updated using chef-clients. The baked in cookbook allow us to bring instances up quickly when used in auto-scaling groups and to revert major changes if there is an unforeseen issue with the current AMI. So far, we have not encountered any notable errors with packer, although our use-case is trivial for packer.
We are pleased with terraform and packer in our workflow, and continuously iterate and develop it so that our team can create and modify AWS infrastructure for their services, confidently and without worry. Using this workflow, we successfully deployed an internal service named hank. In future posts, we will dive deeper into our organisation of terraform configurations and our infrastructure workflow.
Let us know in the comments section if you are interested.