Skip to content

Instantly share code, notes, and snippets.

@apolloclark

apolloclark/Terraform Open-Source Deployment Platform.md

Last active January 24, 2025 18:26
Show Gist options
  • Save apolloclark/6d4b6178e2e50a8f2f99818d382aa262 to your computer and use it in GitHub Desktop.
Save apolloclark/6d4b6178e2e50a8f2f99818d382aa262 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Terraform Open-Source Deployment Platform.md

Terraform Open-Source Deployment Platform

Overview

Terraform open-source (TFOSS) is a very powerful CI/CD tool for deploying cloud resources, however it is missing some key features in the open-source version that are only available in the paid vendor-managed versions called Terraform Cloud (TFC) and the Terraform Enterprise (TFE). Building a deployment platform with only Terraform open-source (TFOSS), versus buying Terraform Cloud (TFC) or Terraform Enterprise (TFE), is like building a kit-car versus buying a sports car from an established brand. It's not impossible, but there are a lot of considerations, listed below.

  1. Module Versioning

  2. Submodule Versioning

  3. Runners

  4. Workspaces

  5. Stacks

  6. Secrets Management

  7. Security Policy as Code

Other features missing in both the open-source and enterprise versions:

  1. Cannot configure a singular Terraform Provider (ex: AWS) to run across multiple Cloud Accounts and Cloud Regions.

Alternative Vendors:

Terraform Frameworks:

Notable guides:

1. Module Versioning

Overview

Terraform has a concept of modules which are a collection of cloud resources which can be configured together, for example when configuring AWS VPC Networking. Modules should contain more than 10 resources but less than 100 resources. Having an excessive amount of cloud resources within a single module can lead to very long Terraform Run times, which increase the likelihood of a broken deployment.

Terraform has a variety of ways of referencing Modules Sources:

  • Local Paths
    • This is useful, but Sub-modules should be used instead for easier code maintenance.
    • This lacks native semantic versioning.
  • HTTP URL
    • This lacks native semantic versioning.
  • AWS S3 Buckets
    • This lacks native semantic versioning.
  • GCP GCS Buckets
    • This lacks native semantic versioning.
  • Modules in Package Sub-directories
    • This is useful, but Sub-modules should be used instead for easier code maintenance.
    • This lacks native semantic versioning.
  • Git
# select a specific tag
module "vpc" {
  source = "git::https://example.com/vpc.git?ref=v1.2.0"
}
module "consul" {
  source = "app.terraform.io/example-corp/k8s-cluster/azurerm"
  version = "1.1.0"
}

Terraform Registry is the only Module Source that natively supports versioning.

Terraform Private Module Registry (PMR)

This is a an enterprise feature natively supported in TFC and TFE and implements the module registry protocol. However, there are multiple open-source systems which support this too.

Terraform Module versioning in a Git URL is a long-standing issue dating back to 2015

From Martin Atkins (apparentlymart) the co-creator of Terraform:

  • "The version argument is only relevant when you are installing a module via a module registry, such as Terraform Registry or a third-party implementation of the same protocol. The registry protocol adds an extra level of indirection over the physical source addresses which allows Terraform to ask the registry which versions are available and then select one based on a version constraint."

Third-party systems supporting the Terraform PMR:

2. Submodule Versioning

A collection of multiple modules can be called a "Parent Module" / Meta-module with "Child Modules" / Submodules. Submodules can be nested multiple levels deep, but it is advised not to do this since it becomes very tedious to debug later for a variety of reasons including inconsistent naming of values passed between modules. The difficulty of using a Meta-Module with a static submodule version is that there is a "Multiplicative Explosion of Complexity" (also the name of my Electro Indie Pop Synth band) caused by the different versions of submodules.

meta-module A
↳ aws-vpc 1.0
↳ aws-rds 1.0
↳ aws-eks 1.0

meta-module B
↳ aws-vpc 1.0
↳ aws-rds 1.0
↳ aws-eks 1.1

Notice how the submodule "aws-eks" is 1.0 in the "meta-module A" versus 1.1 in "meta-module B" while the other submodules "aws-vpc" and "aws-rds" are both 1.0. This is a difficult problem because the Terraform submodule version is not dynamic when using any of the Terraform Module Sources which are not the Terraform Private Module Registry (PMR). This problem can be solved by using the Terraform Private Module Registry (PMR) protocol since it enables dynamically changing the submodule version. Here's what the code for this looks like:

[meta-module.tfvars]
variable "aws-vpc-version" {
 type = string
 default = "1.1"
 description = "Version for the aws-vpc Module"
}
variable "aws-rds-version" {
 type = string
 default = "1.1"
 description = "Version for the aws-vpc Module"
}
variable "aws-eks-version" {
 type = string
 default = "1.1"
 description = "Version for the aws-vpc Module"
}


[meta-module.tf]
module "aws-vpc" {
  source = "app.terraform.io/example-corp/aws-vpc"
  version = var.aws-vpc-version
}
module "aws-rds" {
  source = "app.terraform.io/example-corp/aws-rds"
  version = var.aws-rds-version
}
module "aws-eks" {
  source = "app.terraform.io/example-corp/aws-eks"
  version = var.aws-eks-version
}

3. Runners

Terraform Cloud (TFC) and Terraform Enterprise (TFE) offer an environment to perform Terraform Runs both within the TFC or TFE environment or on remote Cloud Accounts. When using Terraform open-source (TFOSS) you need to designate where that execution environment will be.

4. Workspaces

Terraform open-source (TFOSS) doesn't have a concept of a Terraform Run persistence. Terraform Cloud (TFC) and Terraform Enterprise (TFE) have a Workspace feature which is where the history of a singular Terraform Statefile is stored for running the same Terraform code over time.

Notably Terraform Workspaces have a concept of Run Triggers, announced on 2020-05-20, which allows for triggering multiple Terraform Runs across other Workspaces, enabling easy separation of multiple Terraform Runs based on the cloud resource type and limiting blast radius versus monolithic large Terraform Runs.

5. Stacks

Terraform Stacks are the collection of multiple Workspace Runs, announced on 2024-10-15.

6. Secrets Management

Terraform open-source (TFOSS) doesn't have native functionality for secrets management. Terraform Cloud (TFC) and Terraform Enterprise (TFE) have limited functionality for secrets management, but the best solution is to use a dedicated Secrets Management system, such as:

  • HashiCorp Vault
  • Thycotic
  • CyberArk
  • AWS Secrets Manager (AWS SSM)

7. Security Policy as Code

Terraform open-source (TFOSS) doesn't have native functionality for security policy as code. Terraform Cloud (TFC) and Terraform Enterprise (TFE have a feature called Sentinel, announced on 2021-02-09.

Terraform Cloud (TFC) and Terraform Enterprise (TFE) include Sentinels rules:

TFC and TFE added support for Open Policy Agent (OPA) on 2023-01-31.

OPA is open-source and free, but the Ruleset is not free. There is a third-party company called Fugue which offers hosted-OPA for security and includes large Ruleset packages, and was acquired by Snyk on 2022-02-17.

Atlantis has support for running OPA through a system called Conftest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment