Ephemerality in Terraform

db.tf

provider "aws" {
  region = "us-west-2"
}

ephemeral "random_password" "db_password" {
  length = 16
}

resource "aws_secretsmanager_secret" "db_password" {
  name = "db-password"
}

resource "aws_secretsmanager_secret_version" "db_password" {
  secret_id                = aws_secretsmanager_secret.db_password.id
  secret_string_wo         = ephemeral.random_password.db_password.result
  secret_string_wo_version = 1
}

ephemeral "aws_secretsmanager_secret_version" "db_password" {
  secret_id = aws_secretsmanager_secret_version.db_password.secret_id
}

resource "aws_db_instance" "example" {
  instance_class      = "db.t3.micro"
  allocated_storage   = "5"
  engine              = "postgres"
  username            = "example"
  skip_final_snapshot = true

  password_wo         = ephemeral.aws_secretsmanager_secret_version.db_password.secret_string
  password_wo_version = aws_secretsmanager_secret_version.db_password.secret_string_wo_version
}

Great question, @ktham! The value would be “Y” (the newly updated secret version). When we increment the secret_string_wo_version to 2, we’re effectively accepting a new value generated by the ephemeral random password. It might seem a bit confusing, but ephemeral resources run during every plan and apply, performing the same work each time. This means that a new ephemeral random password is generated with every plan and apply. However, the aws_secretsmanager_secret_version doesn’t use the new password until we increment the secret_string_wo_version to a new revision (2, 3, 4). When we increment the version, it triggers a series of actions: we update the AWS Secrets Manager secret, and similarly the ephemeral aws_secretsmanager_secret_version resource, which runs every time, fetches the new secret, and it then updates the aws_db_instance because we also incremented the write-only version argument there.

Here is the output of terraform plan when we increment secret_string_wo_version to a new revision (2):

ephemeral.random_password.db_password: Opening...
ephemeral.random_password.db_password: Opening complete after 0s
aws_secretsmanager_secret.db_password: Refreshing state... [id=arn:aws:secretsmanager:us-west-2:000000000000:secret:db_password_v1-LuOGiP]
aws_secretsmanager_secret_version.db_password: Refreshing state... [id=arn:aws:secretsmanager:us-west-2:000000000000:secret:db_password_v1-LuOGiP|terraform-20250303214456989400000001]
ephemeral.aws_secretsmanager_secret_version.db_password: Opening...
ephemeral.aws_secretsmanager_secret_version.db_password: Opening complete after 0s
aws_db_instance.example: Refreshing state... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI]
ephemeral.aws_secretsmanager_secret_version.db_password: Closing...
ephemeral.random_password.db_password: Closing...
ephemeral.aws_secretsmanager_secret_version.db_password: Closing complete after 0s
ephemeral.random_password.db_password: Closing complete after 0s

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # aws_db_instance.example will be updated in-place
  ~ resource "aws_db_instance" "example" {
        id                                    = "db-4EUVRROTPHKUWL4NSX3Q55O5VI"
      ~ password_wo_version                   = 2 -> 3
        tags                                  = {}
        # (70 unchanged attributes hidden)
    }

  # aws_secretsmanager_secret_version.db_password must be replaced
-/+ resource "aws_secretsmanager_secret_version" "db_password" {
      ~ arn                      = "arn:aws:secretsmanager:us-west-2:000000000000:secret:db_password_v1-LuOGiP" -> (known after apply)
      ~ has_secret_string_wo     = true -> (known after apply)
      ~ id                       = "arn:aws:secretsmanager:us-west-2:000000000000:secret:db_password_v1-LuOGiP|terraform-20250303214456989400000001" -> (known after apply)
      ~ secret_string_wo_version = 2 -> 3 # forces replacement
      ~ version_id               = "terraform-20250303214456989400000001" -> (known after apply)
      ~ version_stages           = [
          - "AWSCURRENT",
        ] -> (known after apply)
        # (4 unchanged attributes hidden)
    }

Plan: 1 to add, 1 to change, 1 to destroy.

And when applying this plan:

ephemeral.random_password.db_password: Opening...
ephemeral.random_password.db_password: Opening complete after 0s
aws_secretsmanager_secret_version.db_password: Destroying... [id=arn:aws:secretsmanager:us-west-2:000000000000:secret:db_password_v1-LuOGiP|terraform-20250303214456989400000001]
aws_secretsmanager_secret_version.db_password: Destruction complete after 0s
aws_secretsmanager_secret_version.db_password: Creating...
aws_secretsmanager_secret_version.db_password: Creation complete after 1s [id=arn:aws:secretsmanager:us-west-2:000000000000:secret:db_password_v1-LuOGiP|terraform-20250304083930381200000001]
ephemeral.aws_secretsmanager_secret_version.db_password: Opening...
ephemeral.aws_secretsmanager_secret_version.db_password: Opening complete after 0s
aws_db_instance.example: Modifying... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI]
aws_db_instance.example: Still modifying... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI, 10s elapsed]
aws_db_instance.example: Still modifying... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI, 20s elapsed]
aws_db_instance.example: Still modifying... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI, 30s elapsed]
aws_db_instance.example: Still modifying... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI, 40s elapsed]
aws_db_instance.example: Still modifying... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI, 50s elapsed]
aws_db_instance.example: Still modifying... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI, 1m0s elapsed]
aws_db_instance.example: Still modifying... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI, 1m10s elapsed]
aws_db_instance.example: Still modifying... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI, 1m20s elapsed]
aws_db_instance.example: Still modifying... [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI, 1m30s elapsed]
aws_db_instance.example: Modifications complete after 1m32s [id=db-4EUVRROTPHKUWL4NSX3Q55O5VI]
ephemeral.random_password.db_password: Closing...
ephemeral.aws_secretsmanager_secret_version.db_password: Closing...
ephemeral.aws_secretsmanager_secret_version.db_password: Closing complete after 0s
ephemeral.random_password.db_password: Closing complete after 0s

Apply complete! Resources: 1 added, 1 changed, 1 destroyed.

Thank you for explaining!

similarly the ephemeral aws_secretsmanager_secret_version resource, which runs every time, fetches the new secret, and it then updates the aws_db_instance because we also incremented the write-only version argument there.

I understand that the ephemeral aws_secretsmanager_secret_version resource runs every time on each plan/apply and fetches a new value each time. My concern is about when it fetches the new value during the TF apply-phase. In particular, is there a possibility that it runs/fetches the new value before the aws_secretsmanager_secret_version.db_password resource gets recreated?

In your TF apply, I see that ephemeral.aws_secretsmanager_secret_version.db_password: is "opened" after aws_secretsmanager_secret_version.db_password is re-created in your particular example, which is what we'd want. However, I just want to confirm that this will always be the case?

Normally, Terraform set implicit dependencies between resources based attribute access, but in this case, the ephemeral aws_secretsmanager_secret_version is only dependent on aws_secretsmanager_secret and not the aws_secretsmanager_secret_version, so I wasn't sure if I can correctly assume that the ephemeral aws_secretsmanager_secret_version resource will always run after aws_secretsmanager_secret_version.db_password resource gets recreated

Actually, I will retract my last comment, I realized that the code in your Gist is actually an old version of your code.

I see that in hashicorp/terraform#36605, instead of

ephemeral "aws_secretsmanager_secret_version" "db_password" {
  secret_id = aws_secretsmanager_secret.db_password.id
}

you have

ephemeral "aws_secretsmanager_secret_version" "db_master" {
  secret_id = aws_secretsmanager_secret_version.db_password.secret_id
}

This resolves the "timing" concern I mentioned above with respect to when the ephemeral resource attempts to read the value for a secret version, and which version to read (which is whatever AWSCURRENT is pointing to after TF creates/re-creates the aws_secretsmanager_secret_version resource)

I have one suggestion to make it more explicit and clear in https://github.com/hashicorp/terraform/pull/36605/files#r1980138389

Thank you for catching the outdated reference, @ktham, and ++ for finding the updated documentation improvement I had submitted! I noticed I had shown you the plan and apply output of the correct example, but the Gist did not match that. Thanks again!

This is how the dependency graph should look like, using:

ephemeral "aws_secretsmanager_secret_version" "db_master" {
  secret_id = aws_secretsmanager_secret_version.db_password.secret_id
}