Learn the best practices to properly start an infrastructure as code project, from the source code to collaboration exercises and tools.<\/b>
\nJoin the DZone community and get the full member experience. Infrastructure as Code (IaC) has been a trend for many years. While some standards are being defined today, we continue to hear about new methods or tools to improve and make our lives easier by automating our daily tasks as much as possible. Fortunately or unfortunately, today, we have the choice between multiple automation tools like Ansible, Pulumi, Terraform, etc, each one having its own benefits and disadvantages. Thus, choosing the right tool is not the easiest part. It requires a team collaboration to identify, test, and define the right tool. This collaboration is the key to success, it is important to ensure the engagement of the other teams to the IaC project to properly automate the process of each team. Choosing the right tool is a challenge, but defining the architecture of the project is another one. Apart from the war of mono or multi-repository, the focus in the definition of an IaC project architecture should still be collaboration. The purpose should be to aim for simplicity to facilitate engagement. The purpose of this post is to present an opinionated infrastructure-as-code project architecture used multiple times with different automation tools in different environments. But, let\u2019s start with a quick introduction to what an IaC project is. Infrastructure as Code represents the management of any infrastructure component (Cloud architecture, networks, virtual or physical servers, load balancers, etc) in a descriptive model. Based on the practices of software development, it emphasizes consistent, repeatable routines for provisioning and changing systems and their configuration. Similar to the principle that the same source code generates the same binary, an IaC model generates the same environment every time it is applied. IaC is a key DevOps practice. Basically, that means IaC is a methodology of provision, configuring, and managing an IT infrastructure through machine-readable definition files to easily version the state of an entire infrastructure. This methodology is not new, it has been improved over the years and continues to be improved today. An IaC project could potentially have no end since it is obviously linked to the business requirements of the company. There are multiple benefits of an IaC project. The main ones are: Based on this, IaC coupled with the right tool and an appropriate project architecture can be powerful for every engineer in a company. Automation (and IaC) is not reserved for the DevOps team, so when it is time for you to make some choice, have in mind the entire infrastructure to not only improve your daily tasks but also the tasks of your colleagues. Keep in mind that collaboration is the key to success! This is an opinionated architecture that proved its efficiency in multiple contexts. It has obviously been improved over iteration and can probably be improved again. Keep in mind that an IaC project cannot be defined correctly from the first time, it requires several iterations to adapt to the company\u2019s work methodology. Here is a quick explanation of the directory architecture: This directory architecture can be used to logically separate the provisioning code from the configuration code without having to manage multiple repositories which, in some cases, can be a benefit. That way, full automation can be easily handled by the same project, like the provisioning of a virtual machine with Terraform and automatically configuring it with a local provisioner like Ansible. Collaboration is the key to success, like a great proverb says: \u201cAlone we go faster, together we go further!\u201d Developing an IaC project as a team avoids someone developing an architecture that nobody else can understand, and also avoids choosing the wrong automation tools. Keep in mind that an IaC project should be used by anyone in the engineering team of a company to automate its processes. The main goal of the DevOps methodology is to reduce the gap between the operation and the developers. IaC can help, but to achieve that, the project requires involvement from everybody. Obviously, an IaC developed with different teams requires project management to divide it into roadmap, tasks, sub-tasks, and to follow the progression over time. This post doesn\u2019t aim to explain how a project should be managed, but what teams can do to collaborate easier. This step is often forgotten by the operation team, but keep in mind that an IaC project should be seen as any other software project. Version control is the practice of tracking and managing changes to source code over time to prevent concurrent work from conflicting. It also introduces release management to quickly roll back to a previous version when needed. A convention has to be defined and potentially managed automatically by a continuous delivery pipeline to automatically tag a repository, a state file, or a bucket. This convention can be as simple as: In other words, the convention for an IaC project could look like: Another best practice from the development side that an IaC project should follow, the usage of branches. Branches, in any source control software, allow people to diverge from the production version of code to fix a bug or add a feature. Basically, branches allow you to develop your changes without having any impact on production or others’ work. It is highly recommended to use an explicit name while creating a branch to ensure that someone else can refer to it and quickly understand if this branch is still in development or not. A best practice is to name the branch with the ticket number of the current task to quickly refer to an identifier in the project manager. Another best practice is to maintain the main branch for the production code and a dedicated branch for each environment. That way, a workflow can be defined to first deploy every change in the development environment\/branch then a staging environment\/branch then release the changes in the production\/main branch. Another best practice often forgotten by Operation teams is the formatting of the commit messages to ensure a good understanding of the changes. Again, it is highly recommended to define a commit message convention to extract useful information from each update. Here is an example of a commit message convention composed of information that can be used to analyze a project and quickly get insight into the maturity: In other words, the convention is composed of 3 components: a type, a scope, and a summary. The type defines the global purpose of the commit: The scope defines which sub-component of the project is impacted, and the summary quickly describes the update in less than 80 characters. The usage of a convention requires using small commits to easily find the type of commits to use. If you don\u2019t know what pattern to use, that could mean that your commit must be divided into multiple commits. A Github project named \u201c git-semantic-commits \u201d is a good introduction to automate the format of commit messages created in the command line. To ensure that every contributor follows the format, a pre-commit rule can be easily applied by any source controller software. Commit messages are important for the collaboration of multiple people on the same project. Used properly, it can help in many ways, specifically during bug fixes and rollback processes. The main important point of collaboration is the code review, a software quality assurance activity in which people check the IaC code mainly by viewing and reading parts of it, and decide after to continue or interrupt the implementation. A basic process should require at least one approval to merge the code in the main (aka production) branch. Collaboration means involvement and one pain point of it is probably the onboarding process, the time required for a contributor to get the philosophy of the project, understanding the best practices, and committing its first update. This obviously depends on multiple factors, the quality of the documentation, the quality of the code, and the time required to set up a local environment to start the development of an update. Setting up an environment can take time as it requires installing or updating tools, configuring them, downloading dependencies, config files, etc. To make it easier for anyone interested to contribute to the IaC project is to take advantage of automation tools to also configure the \u201cbase\u201d profile required by any contributor. That way, everybody can share the same local environment configuration (version of tools, dependencies, etc.) and easily onboard on the project. No waste of time on research, meetings to set up the environment with the correct version or command line execution, only one playbook to run to be ready to jump in the IaC adventure. In the current architecture presented, this was managed by Ansible to ensure the installation of Python, Terraform, Kubectl, Helm, Mitogen, etc. all at once to easily configure multiple environments (Linux, MacOS, etc.). We cannot talk about infrastructure as code without mentioning tools. So here is a quick list of tools that deserve specific attention when it\u2019s time to start an IaC project. Terraform is an open-source tool developed by HashiCorp that allows DevOps engineers to programmatically provide the resources needed to run an application. Based on the HashiCorp Language (aka HCL), it allows anyone to easily maintain the state of an entire infrastructure by provisioning and re-provisioning it across multiple cloud and on-premises data centers. Thanks to different provisioners and modules, Terraform is able to manage almost every resource required by an application. Terraform is designed to maintain up to date and share multiple states at the same time, but working as a team can become complicated over time if a DevOps process is not followed properly. Fortunately, an external tool has been developed to easily manage code review and automatic merge of any update in a Terraform project, this tool is named Atlantis. Definitely, software to use to catch errors or mistakes before pushing something in production. Testing is an important piece of the DevOps methodology, and an IaC project is not exempt. A great tool has been developed to automate the test of Terraform resources. This project is named Terratest. This Go library, developed by Gruntwork, helps to create and automate tests for an IaC written with Terraform, Packer for IaaS providers like Amazon, Google, or for a Kubernetes cluster. A good step to add to an automated pipeline. Ansible is a tool that provides powerful automation for cross-platform computer support. It can be used by any IT professionals to manage application deployment, updates on workstations and servers, cloud provisioning, configuration management, and nearly anything a systems administrator does on a weekly or daily basis. Ansible, with its idempotency, has dramatically improved the scalability, consistency, and reliability of IT environments. Ansible can be coupled or not with Terraform to easily manage the physical resources required by any application. Even though today, Terraform can do more than provisioning, it has been coupled with Ansible for a long time to manage the provisioning only while Ansible was used to configure components with more flexibility. Like any programmatic language, Ansible definition files must be tested and reviewed before moving to production. Two different tools can be used to automate it, Molecule and Ansible Test. Both are good and easy to use to run unit tests on Ansible playbooks to validate the behavior before reviewing the code. To improve Ansible performances, a great Python library, named Mitogen, can be added to Ansible to dramatically reduce the compression and the traffic generated by Ansible to run any playbooks. Highly recommended using it for an IaC project based on Ansible. As explained in a previous section, an IaC project must be flexible to easily change, if necessary, methodology, architecture, or tool. Every year, new tools are developed and used by the community and aim to compete with the best ones in the market. This is the case of Pulumi, another IaC tool focused on a programmatic language approach, thus, if you are familiar with a programmatic language like Python, Go, etc, and you are about to start an IaC project, take a look at the Pulumi project. It definitely deserves attention. An IaC project must be seen as an application project. It must follow the best practices regarding the development of the definition file and specifically the test step. In certain cases, it is better to test an update locally before testing it in a development or testing environment. Fortunately, multiple tools exist today to help DevOps engineers to test their code before moving it to production. Vagrant, Docker, Podman, Buildah, Minikube, Kind, MicroK8s, etc. are different tools that can be used to emulate specific environments to test a piece of an IaC project. Vagrant for example is a great open-source tool to create a virtual environment (based on Virtualbox for example) to test the deployment of software on any specific operating system. Docker, Podman, and Buildah can be used to do the same thing in a container environment while Minikube, Kind, MicroK8s can be used to do it in a local Kubernetes cluster. Depending on the use case, we have today multiple approaches to easily introduce testing in an IaC project to be proactive on potential issues instead of being reactive. An IaC project is usually composed of a \u201cbase\u201d profile regarding the computed resources. A base profile or a common profile is a list of roles that the automation tool needs to apply on any compute resources to ensure consistency across resources. This base profile can be composed of the configuration of the root password, the NTP servers, the SMTP servers, the deployment of the monitoring tools, etc\u2026 Sometimes, when it comes to improving the IaC project by reducing the time to spin up a new machine, the base profile becomes a problem. One known option to improve this is to reduce the base profile by creating a custom base image of an operating system. To do that in a programmatic way, an open-source tool named Packer can be used. Packer is an open-source tool, developed by HashiCorp, to create identical machine images for multiple platforms from a single source configuration. The images created can then be uploaded and used in the Cloud or by any on-premise virtual orchestrator platform. Packer can be used locally by Vagrant to generate a virtual machine based on a specific custom image definition and test locally a piece of the IaC project. Definitely a good combination to improve the deployment of any custom resources. This step is obviously optional and depends on the infrastructure requirements. Emulating Cloud services locally is important to reduce the cost of an IaC project. Instead of generating an entire environment to test a piece of code, some tools can be used to emulate locally some Cloud services before testing the code in production. For the AWS users, a great project named LocalStack has been developed and is, today, really powerful to test Terraform or Ansible code locally. LocalStack can be easily deployed on a local virtual environment (virtual machine or container) to mock many AWS endpoints. For GCP users, the gcloud command has some experimental features to emulate limited endpoints for now. This article is an opinionated presentation of infrastructure as a code architecture to easily and correctly start a project. There are probably thousands of other ways to do this and this architecture can probably be improved, so feel free to share your ideas in the comments! For more information: Semantic versioning definition Semantic-release tools to help automatic release Conventional commits Published at DZone with permission of Nicolas Giron. See the original article here. Opinions expressed by DZone contributors are their own.<\/p>\n