Having the opportunity to be part of this year's PuppetConf was a tremendous experience. Being surrounded by so many like-minded individuals all working towards automating infrastructures in repeatable and reliable ways was truly inspiring. While much of what I learned was validation for the goals I envision for my current work environment, the details of the process were very edifying.
The general theme of the conference that I came away with was all about workflows and automated testing both of which are key to environments that instill confidence in their deployments. Without these two items there is a lack of consistency and reliability to the deployments that leads to a number of problems. Technical debt, lack of confidence in the deployment or operations teams, general unhappiness throughout an organization, reduction in deliverability velocity and reduction in profits. Configuration changes should be able to be rolled out frequently, quickly and with high confidence in their reliability. The only way to accomplish this is through robust, consistent workflows and automated testing of the results of the configuration changes in production-like environments.
There are a number of products that are emerging or maturing to provide these two key aspects of infrastructure development, but two in particular stood out throughout the conference - Beaker for automated acceptance testing and R10K for providing a workflow around deploying configuration changes through development, staging & production environments.
Delivering configuration changes quickly is great but only if the changes actually work without breaking existing services in unexpected ways. This is where acceptance testing comes into play. Acceptance testing is the process of ensuring that changes are applied in predictable ways and that the state of the system after the configuration changes is as it is expected to be. Ideally each and every configuration change that is committed to version control should be tested so that changes that break existing systems are identified as quickly as possible. A proper acceptance testing framework includes comprehensive feedback that provides the developer with sufficient information to easily determine what change broke the environment. By using a framework, the process and results of the testing is made consistent across the entire development team. The testing process is incorporated into the workflow process that every developer uses and successful testing runs are expected before the configuration changes are submitted for peer review.
Beaker is a framework built on top of other acceptance testing frameworks including Serverspec, RSpec and other established frameworks. The benefit of using Beaker is to wrap these other frameworks in a consistent layer that provides for portable testing infrastructures that are shipped with the configuration code being tested. This consistent configuration is provided via the use of a YAML file along with Bundler for defining Ruby dependencies, etc. The combination of these files provides every other developer with all the information they need to setup the environment exactly as it was tested in by the original developer. A peer reviewer is able to run a consistent series of commands (the same for any module changes) and run through the test suite to verify that the changes are applied in a predictable and that the environment is setup as expected. Additionally, configurations changes that break legacy configurations will show up as regression issues because of previous written tests that are carried forward. This allows the developer to identify areas that either need to be accounted for in the new configuration code or changes that will be required to legacy systems to deploy the code into legacy environments.
This type of framework does put a certain onus on the developer of the configuration changes to also develop their own tests ensuring the changes are applied as they expect. Often this is done through a test-driven development process where tests are generated first asserting a certain result and then the configuration logic is written to satisfy the tests.
By extending this framework into a continuous integration environment, all changes to configurations can be automatically tested prior to deployment but after peer review and approval. This adds yet another layer of confidence to the deployment process and facilitates the fast delivery of changes without the level of risk typical of manually reviewed/tested changes.
The process of getting configuration changes from development to production requires a consistent workflow that includes testing at various levels depending on the environment the changes are deployed into. Configurations changes are "hardened" as they travel through the layers beginning in a fairly fluid state in development until they finally reach production where they are considered solid. Testing at the development layer is highly automated and fast while staging requires some level of manual validation. Production is validated, in effect, by the customers using the system. Obviously once in production it is expected that there are no issues and the changes are considered static.
Puppet configurations need to be treated with the same type of workflow. In essence the configurations managed by Puppet are code that follow a development workflow as described above. R10K is a tool that helps facilitate the transition of configurations from one stage to the next through the heavy use of version control branches and tooling to manage these branches. Configurations in the development stage are committed to a development branch and, using R10K, are deployed to a development Puppet environment. Once these configuration changes are validated and approved, they are migrated through the use of a merge into a staging environment. The staging branch is then deployed to a Puppet staging environment and again validated at this level. Finally configurations are merged into a production branch where they are again deployed via R10K to a production Puppet environment.
With the R10K framework each of these steps can be automated through a continuous integration service that evaluates feedback of the validation steps and the promotion of the code can progress up to production with minimal human interaction.
While testing and workflow were a huge part of the theme of the conference another theme emerged - teamwork. With proper workflows and testing in place, teamwork is being promoted as a common goal. Consistency promotes cooperation as the entire team is focused on improving and streamlining the process for delivering changes. Testing builds confidence throughout the system allowing teams to feel comfortable with the contributions of each individual. Management can be confident in stable and reliable products being delivered in a timely manner.
It's great to see configuration management being viewed and used as software development projects with comprehensive workflows and testing built in. Reviews and team involvement brings the infrastructure into the same domain as the application where the entire infrastructure can work together in better harmony as a full product. Silos and artificial walls begin to fall as everyone begins speaking in similar languages towards a common goal of producing a profitable product.