Building Fully-Auditable IaC w/ Atlantis, Terraform & Git

We love HashiCorp’s Terraform, but have one big reservation with how it gets used in ways that don’t respect proper GitOps practices. Most notably, the requirement that your system-level changes are handled in code and peer reviewed, with fully-auditable results.

The only viable answer is to automate Terraform itself. At first glance, that might seem redundant—doesn’t Terraform already automate the provisioning on infrastructure using code?—but what we’re really talking about is automating how your people execute Terraform to deploy the IaC you’ve developed.

What if, instead of using the CLI tooling or ClickOps, you had a single runtime for automating Terraform through pull requests? You might think there’s no way you can prioritize that over the hundreds of other items on your engineering TODO list, but it’s not as complicated as you might think.

That’s exactly what we’ve built with kubefirst, which glues Terraform, Atlantis, Infrastructure as Code (IaC), and GitOps into a single, synchronized platform. You can deploy kubefirst today in just a few steps, or you can build this automation yourself—but before we dig into either option, let’s talk about how you benefit from automating Terraform in the first place.

Why automate Terraform at all?

Let’s say your team has spent a few weeks writing and perfecting your IaC code using Terraform. You’re excited to have finally codified your infrastructure into standardized, composable, and reusable components, which is a big step toward your long-term goal of treating your clusters like cattle, not pets.

As your teams start using your IaC to update or deploy infrastructure, they run Terraform in one of two ways, both of which are antithetical to GitOps:

• The Terraform CLI directly on an engineer’s laptop, which doesn’t provide version control and auditability.
• Through GitHub’s “ClickOps” features, which let anyone create new repositories and CI/CD pipelines with GitHub Actions, building inconsistency in how IaC manages your infrastructure.

As engineers and developers apply Terraform plans, they can’t see every possible outcome directly from their laptop or in GitHub Actions’ output logs. It’s trivially easy to run into “simple” problems, like collisions on names and descriptions or a half-applied plan requiring mitigation. The likelihood of these issues amplifies as your people continue to apply plans in Terraform without complete visibility into what their peers are doing simultaneously.

You quickly lose control over the current state of your infrastructure. Is the latest commit on main the code that’s actually deployed on your cluster? Has someone pushed a manual fix and not followed up with a proper pull request to make that change permanent? Do those five new repositories created just yesterday have the proper IaC and role-based access control (RBAC) policy?

At Kubefirst, we hate these kinds of questions and the worry, headache, and lost time that goes into answering them. We hate that when you make a positive, proactive improvement like implementing IaC, you’re seemingly forced to take a step back on your GitOps culture.

We’re also all about building the glue that brings the best in cloud native together and helps you get the most out of GitOps, which is why we’ve integrated Atlantis, an open source and self-hosted application for automating Terraform via pull requests, directly into our kubefirst platform.

How Atlantis automates your Terraform IaC

Atlantis wraps around Terraform to create an automated, auditable system for ensuring that your IaC repositories are accurate and up-to-date representations of your deployed infrastructure. Through pull requests and your Git provider of choice, it shows you exactly what your proposed changes will do and abstracts away the responsibility of executing Terraform plans.

Here’s the step-by-step once you have automated Terraform:

1. You create a PR that adds or modifies a Terraform configuration (.tf) file, which calls a Atlantis webhook running inside your infrastructure.
2. Atlantis receives the call and runs terraform plan against your infrastructure.
3. Atlantis adds a comment to the PR with the full log output from terraform plan, which ensures you and your reviewers can see the full implications of your changes.
4. You and your peers engage in the standard code collaboration and review process you have established.
5. One or more peers approves your PR.
6. You run atlantis apply directly in the comments of your Git platform.
7. Atlantis runs terraform apply in the background and returns the output of that command to the comments.
8. If your changes are deployed as expected, the PR is automatically merged and closed.

No CLI tooling, no ClickOps-ing.

If you want to automate your IaC to resolve these conflicts between Terraform and GitOps, you have two options: build the system yourself, or use our free and open source tool kubefirst.

Putting the GitOps and IaC pieces together

Deploying Atlantis yourself

If you take the manual route, you’re in for a bit of back-and-forth of webhooks, credentials, and Kubernetes manifests.

The Atlantis folks have a great installation guide to walk you through everything, starting with generating access tokens for your Git platform of choice—GitHub, GitLab, Bitbucket, or Azure DevOps—and creating webhook secrets.

You then need to deploy Atlantis in your infrastructure, which you can do with a Helm chart, raw Kubernetes manifest, Kustomize, or a custom module for certain providers, like Google Compute Engine (GCE).

As long as you get that running properly, you configure webhooks on your Git platform to ensure it can listen to and respond to pull requests as expected, then configure your providers with the necessary credentials to run Terraform commands directly.

Deploying Atlantis (and a whole lot more) with kubefirst

In a single command, kubefirst deploys a Kubernetes cluster managed with IaC, GitOps asset management, Argo CD, Vault secrets management, and much more. That includes Terraform and Atlantis, which lets you focus on onboarding your team to a complete application delivering ecosystem rather than worrying about the nuts and bolts.

We’ve even assembled a complete after-installation walkthrough using GitLab and an example microservice called Metaphor to demonstrate your new IaC workflow.

We’ve also thoroughly documented our Terraform+Atlantis glue to address the most common use cases and questions, including how to extend this functionality to all new repositories created by your team.

The great thing about the cloud native ecosystem is that no matter which route you choose, automating Terraform with Atlantis unlocks huge benefits for doing IaC and GitOps properly:

• All system-level changes are handled in code.
• All changes are peer-reviewed before applied, which allows you to enable branch protections, like requiring >1 approval before merging.
• Atlantis locks the directory or workspace until other open pull requests are approved and closed, ensuring changes are applied in the order you expect.
• The complete output of terraform plan is visible to all reviewers in the pull request, helping your team catch issues that might have otherwise gone unnoticed.
• Your approved and merged pull requests become a true system audit log from the day you started using IaC to deploy your cluster.
• No more “it worked on my machine” by running terraform plan or terraform apply directly from laptops or CI/CD pipelines.
• You improve the security of your deployment process as you don’t have to distribute Terraform credentials to every engineer because you can now open pull requests on GitHub that run Terraform via Atlantis, improving your security posture.

We hope you join us in exploring what’s possible when people start doing IaC and Terraform in a way that improves—not hinders—the best of Kubernetes GitOps.

If you end up going the easy way with kubefirst and have questions, our team is happy to chat and help in the Kubefirst Slack community—we'd also love your feedback on your cluster deployment and IaC experience!

Photo by Eric Krull on Unsplash