Route 53 + CloudFormation + CDK: a survival runbook (orphan hosted zones, skip-retain trap, recovery procedures)

route53-cdk-runbook.md

Route 53 + CloudFormation + CDK runbook

Reference for PublicHostedZone operations, CDK stack/stage renames, and skip-retain situations.

---

Mechanics

DeleteHostedZone blocks on non-default records

Succeeds only when the zone has just NS + SOA. Blocked by:

• A, AAAA, ALIAS
• MX, TXT, SPF, DKIM, CAA
• Custom NS records (child-zone delegation)

CDK's PublicHostedZone RemovalPolicy.DESTROY is ignored — the API call fails. The stack enters DELETE_FAILED.

Skip-retain leaves silent orphans

aws cloudformation delete-stack --retain-resources <logical-id>
aws cloudformation continue-update-rollback --resources-to-skip <logical-id>

The resource is abandoned with no audit entry, no tag, no scheduled cleanup. The orphan keeps serving traffic. Same shape applies to S3 buckets with content and similar retain-capable resources.

---

Before renaming a CDK stack/stage that owns a hosted zone

Rename ⇒ stack delete + create. Old zone is orphaned unless emptied first.

aws route53 list-resource-record-sets --hosted-zone-id <Zid>
# delete every non-NS/SOA record:
aws route53 change-resource-record-sets --hosted-zone-id <Zid> --change-batch ...
# verify only 2 records remain (NS + SOA)
# deploy rename
aws route53 list-hosted-zones | grep <dns-name>   # should show only the new zone

---

Before --retain-resources / --resources-to-skip

aws cloudformation describe-stack-resource --stack-name <stack> --logical-resource-id <id>

For Route53::HostedZone / S3::Bucket, tag the resource before skipping:

aws route53 change-tags-for-resource \
  --resource-type hostedzone \
  --resource-id <Zid> \
  --add-tags 'Key=orphaned-by,Value=<name>' \
             'Key=orphaned-on,Value=<date>' \
             'Key=orphaned-reason,Value=<short>'

---

Before deleting a hosted zone

rg <Zid> .                                              # IaC references
aws route53 list-tags-for-resource --resource-type hostedzone --resource-id <Zid>
# look for aws:cloudformation:stack-id / stack-name → live CFN-managed
aws route53 list-resource-record-sets --hosted-zone-id <parent-Zid> \
  --query "ResourceRecordSets[?Name=='<dns-name>.'].ResourceRecords"
# NS values identify which zone is live

When multiple zones share a DNS name: delete by physical ID, never filter by name alone.

---

Recovery

DELETE_FAILED on a hosted-zone stack

aws cloudformation describe-stack-events --stack-name <stack> --max-items 20

Clear records → delete-stack. --retain-resources only as a last resort with the tagging step above.

UPDATE_ROLLBACK_FAILED referencing a missing zone

R53 hosted zone NS values are unrecoverable once the zone is deleted.

aws cloudformation continue-update-rollback --stack-name <stack> --resources-to-skip <logical-id>

Do not retry the same in-place update. Reconcile drift first: rebuild the stack, or cloudformation create-change-set --change-set-type IMPORT.

Custom::CrossRegionExportWriter involved

CDK's cross-region SSM ref. If failed, skip the writer's logical ID alongside the zone.

---

CDK design

Custom-domain wiring behind a config flag

Gate addCustomDomain(...) and the DNS/cert stacks on optional customDomain?: { host: string }. Stages without it skip the plumbing.

Permanent this.exportValue(...) retainers

this.exportValue(resource.attr);

Keep it permanently, not as a transient compat step. The export survives consumers disappearing. Comment at the call site.

Avoid stack/stage renames in live accounts

Iterate layout in a sandbox first.

Daily drift detection on hosted-zone stacks

aws cloudformation detect-stack-drift --stack-name <stack>

Alert on drift.

Tag CFN-managed hosted zones

CFN doesn't tag Route 53 zones automatically with aws:cloudformation:stack-id. Add it via CDK or a custom resource. Cleanup scripts grep -v CFN-tagged zones.

---

Cleanup pre-flight

aws route53 list-hosted-zones > /tmp/zones-snapshot-$(date +%s).json
for zid in $(jq -r '.HostedZones[].Id' /tmp/zones-snapshot-*.json); do
  aws route53 list-resource-record-sets --hosted-zone-id "$zid" > "/tmp/records-${zid##*/}.json"
done

• Identify by physical ID, not by DNS name.
• Check aws:cloudformation:stack-id tags before each delete.
• list-hosted-zones order is not deterministic; when N zones share a name, only one is live.

---

TL;DR

• Hosted zones with records can't be CFN-deleted. Stack/stage renames orphan them.
• --retain-resources / --resources-to-skip silently complete the orphan.
• Dedup by DNS name picks a random survivor. Use physical ID + tags.
• Gate custom-domain wiring on a config flag.
• Permanent this.exportValue retainers on producers.
• Daily drift detection on hosted-zone stacks.