Ephemerality in Terraform

db.tf

provider "aws" {
  region = "us-west-2"
}

ephemeral "random_password" "db_password" {
  length = 16
}

resource "aws_secretsmanager_secret" "db_password" {
  name = "db-password"
}

resource "aws_secretsmanager_secret_version" "db_password" {
  secret_id                = aws_secretsmanager_secret.db_password.id
  secret_string_wo         = ephemeral.random_password.db_password.result
  secret_string_wo_version = 1
}

ephemeral "aws_secretsmanager_secret_version" "db_password" {
  secret_id = aws_secretsmanager_secret_version.db_password.secret_id
}

resource "aws_db_instance" "example" {
  instance_class      = "db.t3.micro"
  allocated_storage   = "5"
  engine              = "postgres"
  username            = "example"
  skip_final_snapshot = true

  password_wo         = ephemeral.aws_secretsmanager_secret_version.db_password.secret_string
  password_wo_version = aws_secretsmanager_secret_version.db_password.secret_string_wo_version
}

Ideally what some of the SMEs have been thinking is that the wo_version should reference a var everywhere it’s used - otherwise you run the risk of bumping it (for example) for the AWS Secret getting updated, but not the DB itself, and they can get out of sync.

Important: If an input argument to an ephemeral resource references a value that is not yet known (but will be during or after the plan), Terraform defers the execution of the ephemeral resource to the apply stage instead of running it during the plan.

In this example, the ephemeral resource aws_secretsmanager_secret_version references an input argument that is initially unknown during the first plan. Because of this, Terraform defers its execution to the apply stage, incorporating it into the dependency graph and ensuring it runs in the correct order. This allows Terraform to first create the secret using the ephemeral random_password, then retrieve it using the ephemeral aws_secretsmanager_secret_version resource, and finally pass it into the write-only attribute of the aws_db_instance managed resource.

Ideally what some of the SMEs have been thinking is that the wo_version should reference a var everywhere it’s used - otherwise you run the risk of bumping it (for example) for the AWS Secret getting updated, but not the DB itself, and they can get out of sync.

Thanks for sharing this, @jweigand! 👍 I left that part out to keep the example a bit simpler, but great call-out!

Update: I've updated the example to include a reference between the two write-only argument versions.

This is very cool, no doubt.

I'd like to use this but sadly opentofu doesn't support ephemeral yet opentofu/opentofu#1996. Once they do, this is on our list!

@bschaatsbergen - In your example, if aws_secretsmanager_secret_version.db_password.secret_string_wo_version was incremented to 2, aws_secretsmanager_secret_version.db_password.secret_string_wo would change from the current value X, to a new value Y that is read from ephemeral.random_password.db_password.result.

And because aws_db_instance.example.password_wo_version would also be incremented to 2, aws_db_instance.example.password_wo would also be updated as well.

However, what I'm not so sure about is whether aws_db_instance.example.password_wo would be updated to the new value Y, or would it attempt to update it to the value X (which would effectively be a no-op in the end)?

Let's say we've applied your configuration successfully and then we apply the following configuration with the change of incrementing secret_string_wo_version from 1 to 2

provider "aws" {
  region = "us-west-2"
}

ephemeral "random_password" "db_password" {
  length = 16
}

resource "aws_secretsmanager_secret" "db_password" {
  name = "db-password"
}

resource "aws_secretsmanager_secret_version" "db_password" {
  secret_id                = aws_secretsmanager_secret.db_password.id

  # This will be updated, let's refer to the current value as "X", and the new value that
  # is read out from `ephemeral.random_password.db_password.result` as "Y"
  secret_string_wo         = ephemeral.random_password.db_password.result

  # Changing this from 1 -> 2 will trigger an update for secret_string_wo
  secret_string_wo_version = 2
}

# Unclear: Will this be "X" or will this be "Y"?
ephemeral "aws_secretsmanager_secret_version" "db_password" {
  secret_id = aws_secretsmanager_secret.db_password.id
}

resource "aws_db_instance" "example" {
  instance_class      = "db.t3.micro"
  allocated_storage   = "5"
  engine              = "postgres"
  username            = "example"
  skip_final_snapshot = true

  # What value would this be? "X" or "Y"?
  password_wo         = ephemeral.aws_secretsmanager_secret_version.db_password.secret_string

  # Changing this from 1 -> 2 will trigger an update for password_wo
  password_wo_version = aws_secretsmanager_secret_version.db_password.secret_string_wo_version
}

Can you confirm if in this case that the password "update" for aws_db_instance will remain consistent with the password "update" for aws_secretsmanager_secret_version?

Great question, @ktham! The value would be “Y” (the newly updated secret version). When we increment the secret_string_wo_version to 2, we’re effectively accepting a new value generated by the ephemeral random password. It might seem a bit confusing, but ephemeral resources run during every plan and apply, performing the same work each time. This means that a new ephemeral random password is generated with every plan and apply. However, the aws_secretsmanager_secret_version doesn’t use the new password until we increment the secret_string_wo_version to a new revision (2, 3, 4). When we increment the version, it triggers a series of actions: we update the AWS Secrets Manager secret, and similarly the ephemeral aws_secretsmanager_secret_version resource, which runs every time, fetches the new secret, and it then updates the aws_db_instance because we also incremented the write-only version argument there.

Here is the output of terraform plan when we increment secret_string_wo_version to a new revision (2):

ephemeral.random_password.db_password: Opening...
ephemeral.random_password.db_password: Opening complete after 0s
aws_secretsmanager_secret.db_password: Refreshing state... [id=arn:aws:secretsmanager:us-west-2:000000000000:secret:db_password_v1-LuOGiP]
aws_secretsmanager_secret_version.db_password: Refreshing state... [id=arn:aws:secretsmanager:us-west-2:000000000000:secret:db_password_v1-LuOGiP|terraform-20250303214456989400000001]
ephemeral.aws_secretsmanager_secret_version.db_password: Opening...
ephemeral.aws_secretsmanager_secret_version.db_password: Opening complete after 0s
aws_db_instance.example: Refreshing state... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI]
ephemeral.aws_secretsmanager_secret_version.db_password: Closing...
ephemeral.random_password.db_password: Closing...
ephemeral.aws_secretsmanager_secret_version.db_password: Closing complete after 0s
ephemeral.random_password.db_password: Closing complete after 0s

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # aws_db_instance.example will be updated in-place
  ~ resource "aws_db_instance" "example" {
        id                                    = "db-4EUVRROTPHKUWL4NSX3Q55O5VI"
      ~ password_wo_version                   = 2 -> 3
        tags                                  = {}
        # (70 unchanged attributes hidden)
    }

  # aws_secretsmanager_secret_version.db_password must be replaced
-/+ resource "aws_secretsmanager_secret_version" "db_password" {
      ~ arn                      = "arn:aws:secretsmanager:us-west-2:000000000000:secret:db_password_v1-LuOGiP" -> (known after apply)
      ~ has_secret_string_wo     = true -> (known after apply)
      ~ id                       = "arn:aws:secretsmanager:us-west-2:000000000000:secret:db_password_v1-LuOGiP|terraform-20250303214456989400000001" -> (known after apply)
      ~ secret_string_wo_version = 2 -> 3 # forces replacement
      ~ version_id               = "terraform-20250303214456989400000001" -> (known after apply)
      ~ version_stages           = [
          - "AWSCURRENT",
        ] -> (known after apply)
        # (4 unchanged attributes hidden)
    }

Plan: 1 to add, 1 to change, 1 to destroy.

And when applying this plan:

ephemeral.random_password.db_password: Opening...
ephemeral.random_password.db_password: Opening complete after 0s
aws_secretsmanager_secret_version.db_password: Destroying... [id=arn:aws:secretsmanager:us-west-2:000000000000:secret:db_password_v1-LuOGiP|terraform-20250303214456989400000001]
aws_secretsmanager_secret_version.db_password: Destruction complete after 0s
aws_secretsmanager_secret_version.db_password: Creating...
aws_secretsmanager_secret_version.db_password: Creation complete after 1s [id=arn:aws:secretsmanager:us-west-2:000000000000:secret:db_password_v1-LuOGiP|terraform-20250304083930381200000001]
ephemeral.aws_secretsmanager_secret_version.db_password: Opening...
ephemeral.aws_secretsmanager_secret_version.db_password: Opening complete after 0s
aws_db_instance.example: Modifying... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI]
aws_db_instance.example: Still modifying... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI, 10s elapsed]
aws_db_instance.example: Still modifying... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI, 20s elapsed]
aws_db_instance.example: Still modifying... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI, 30s elapsed]
aws_db_instance.example: Still modifying... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI, 40s elapsed]
aws_db_instance.example: Still modifying... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI, 50s elapsed]
aws_db_instance.example: Still modifying... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI, 1m0s elapsed]
aws_db_instance.example: Still modifying... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI, 1m10s elapsed]
aws_db_instance.example: Still modifying... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI, 1m20s elapsed]
aws_db_instance.example: Still modifying... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI, 1m30s elapsed]
aws_db_instance.example: Modifications complete after 1m32s [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI]
ephemeral.random_password.db_password: Closing...
ephemeral.aws_secretsmanager_secret_version.db_password: Closing...
ephemeral.aws_secretsmanager_secret_version.db_password: Closing complete after 0s
ephemeral.random_password.db_password: Closing complete after 0s

Apply complete! Resources: 1 added, 1 changed, 1 destroyed.

Thank you for explaining!

similarly the ephemeral aws_secretsmanager_secret_version resource, which runs every time, fetches the new secret, and it then updates the aws_db_instance because we also incremented the write-only version argument there.

I understand that the ephemeral aws_secretsmanager_secret_version resource runs every time on each plan/apply and fetches a new value each time. My concern is about when it fetches the new value during the TF apply-phase. In particular, is there a possibility that it runs/fetches the new value before the aws_secretsmanager_secret_version.db_password resource gets recreated?

In your TF apply, I see that ephemeral.aws_secretsmanager_secret_version.db_password: is "opened" after aws_secretsmanager_secret_version.db_password is re-created in your particular example, which is what we'd want. However, I just want to confirm that this will always be the case?

Normally, Terraform set implicit dependencies between resources based attribute access, but in this case, the ephemeral aws_secretsmanager_secret_version is only dependent on aws_secretsmanager_secret and not the aws_secretsmanager_secret_version, so I wasn't sure if I can correctly assume that the ephemeral aws_secretsmanager_secret_version resource will always run after aws_secretsmanager_secret_version.db_password resource gets recreated

Actually, I will retract my last comment, I realized that the code in your Gist is actually an old version of your code.

I see that in hashicorp/terraform#36605, instead of

ephemeral "aws_secretsmanager_secret_version" "db_password" {
  secret_id = aws_secretsmanager_secret.db_password.id
}

you have

ephemeral "aws_secretsmanager_secret_version" "db_master" {
  secret_id = aws_secretsmanager_secret_version.db_password.secret_id
}

This resolves the "timing" concern I mentioned above with respect to when the ephemeral resource attempts to read the value for a secret version, and which version to read (which is whatever AWSCURRENT is pointing to after TF creates/re-creates the aws_secretsmanager_secret_version resource)

I have one suggestion to make it more explicit and clear in https://github.com/hashicorp/terraform/pull/36605/files#r1980138389

Thank you for catching the outdated reference, @ktham, and ++ for finding the updated documentation improvement I had submitted! I noticed I had shown you the plan and apply output of the correct example, but the Gist did not match that. Thanks again!

This is how the dependency graph should look like, using:

ephemeral "aws_secretsmanager_secret_version" "db_master" {
  secret_id = aws_secretsmanager_secret_version.db_password.secret_id
}