AWS Describe/Get actions as Terraform Data Sources

NOTES.md

AWS Describe and Get actions as Terraform Data Sources

Beyond the read-only resources that existed in Terraform before data resources were an official feature, I'd like to make room for a new pattern for re-usable modules where they accept only the minimum set of ids they need as variables and look up other data via data sources.

Here's a motivating example:

variable "aws_subnet_id" {
    description = "Id of the subnet where the EC2 instance will be created"
}

data "aws_subnet" "target" {
    id = "${var.aws_subnet_id}"
}

data "aws_vpc" "target" {
    id = "${data.aws_subnet.target.id}"
}

resource "aws_security_group" "main" {
    vpc_id = "${data.aws_subnet.target.vpc_id}"

    ingress {
        from_port = 22
        to_port = 22
        protocol = "tcp"
        cidr_blocks = ["${data.aws_vpc.target.cidr_block}"]
    }

    # ...
}

resource "aws_instance" "main" {
    # ...

    vpc_security_group_ids = ["${aws_security_group.main.id}"]
    subnet_id = "${var.aws_subnet_id}"
}

output "aws_instance_id" {
    value = "${aws_instance.main.id}"
}

In the above example we have a Terraform module that takes just an AWS subnet id and uses it to find the corresponding VPC so that a VPC security group can be created along with an EC2 instance.

In principle just about every AWS action that starts with Describe or Get could become a data source. In practice, there are probably just a small set that would be useful in 90% of cases where today a module must take as arguments a whole sea of ids at different levels of abstraction but instead can now just take the "leafiest" ids and derive the rest.

Along with simplifying module interfaces, this pattern could be applied to reduce coupling between Terraform configs by passing a resource from one to another using AWS tagging rather than terraform_remote_state: instead of having a state that spells out which VPC is used for a particular deployment environment, one could simply ask Terraform to look up the one tagged with Environment = production.

The remainder of this describes a suggested pattern for building a data source around any AWS Describe or Get endpoint, in the hope that we can present a consistent pattern across many different data sources.

Mapping Request and Response Attributes

Many of AWS's "read" actions -- all of the ones in EC2, certainly -- are structured such that the request contains a set of constraints and the response is a set of resources matching those constraints, possibly with pagination where many results are returned.

There are a few different kinds of constraint:

• The resource's own id. In this case, the API returns the specific items requested.
• Some other queryable id or attribute, like the account id that owns the resource, or the id of the parent object it belongs to.
• Various filters, which usually allow constraining by many more criteria than just the queryable attributes but at the expense of needing to do more requests since the filter is applied as a post-processing step after the response has already been paginated on the server side.

All of the above can become Optional or Required attributes on the corresponding data source, depending on whether the underlying API requires them.

These AWS APIs return lists of results, but with Terraform's currently-rather-limited support for lists of complex types it is more pragmatic to build "singular" resources that return just one item that matches the given criteria, apart from some key exceptions like DescribeAvailabilityZones where the result is only useful as a list.

In the AWS API these are often presented as sets where the caller can provide multiple values that should be matched, but within Terraform I think it's better to present them as single values so that they can be consistent with the attributes on any corresponding managed resource and so that the resulting values can be placed into the same attributes as the query constraints. So the hypothetical data source for DescribeSubnets might have attributes like this:

• id (Optional, Computed): if provided in config, sent as the only member of SubnetIds request attribute
• availability_zone (Optional, Computed): if provided in config, sent as availabilityZone filter
• cidr_block (Optional, Computed): if provided in config, sent as cidrBlock filter
• vpc_id (Optional, Computed): if provided in config, sent as vpc-id filter
• state (Optional, Computed): if provided in config, sent as state filter
• tags (Optional, Computed): a mapping of strings to strings that becomes a tag:key=value filter for each entry

The above attribute names intentionally align with the equivalent attributes in the aws_subnet resource.

They are all "Optional" and "Computed", so the user is free to set any combination of these in the configuration and then when the data source is read the remaining unspecified attributes will be populated from the API response. This creates a model of "specify as much as you know and we will figure out the rest".

In some cases a particular constraint is supported both as a queryable attribute and as a filter. In that case we can prefer the queryable attribute.

Custom Filters

In some cases the user may wish to do a more complex filter than can be expressed by our single-value attributes above. For any Describe API that supports filters we can additionally provide a filter block that is "Optional" and not "Computed", that can be used like this:

filter {
    name = "availabilityZone"
    values = ["us-west-1a", "us-west-1b"]
}

These then get added as additional filters to the query, without any further processing on Terraform's part.

If there is a compelling use-case for one of the queryable attributes to accept lists in Terraform then we can potentially produce a plural alternative to the singular argument for that, allowing the user to populate either.

From Many Objects to Just One Resource

As noted above, it's more convenient with Terraform's current limited support for complex-type list attributes to make the data sources return a single value. Since the underlying API endpoints may return multiple values, we have two options for how to deal with this:

• If a particular type of object has one or more "natural" sorting criteria, like the time the object was created, an additional select argument can be supported to specify how to choose a single item from a list. For example, aws_ami might support select = "latest" to specify that the user wishes to retrieve the most recently-created image matching the given criteria.
• Otherwise, it's probably best to treat more than one result as an error and encourage the user to put more constraints in the configuration to reduce it to a single item, to avoid accidents where the user thinks she is selecting only one resource but in fact ends up arbitrarily selecting one of many.

In cases like aws_availability_zones where it is the list that is primarily interesting and individual items not so much, it's most convenient to make a result attribute that includes just a list of ids so that it's easier to work with in Terraform's interpolation language. This might be paired with a singular data source that makes it possible to retrieve details about a single item from that list. For (contrived) example:

data "aws_availability_zones" "us-west-2" {
    states = ["available", "information", "impaired"]
}

// (see https://github.com/hashicorp/terraform/pull/6672 )
resource "random_shuffle" "azs" {
    input = ["${data.aws_availability_zones.us-west-2.names}"]
    result_count = 2
}

data "aws_availability_zone" "primary" {
    name = "${random_shuffle.azs.result[0]}"
}
data "aws_availability_zone" "secondary" {
    name = "${random_shuffle.azs.result[1]}"
}

provider "aws" {
    alias = "primary_az"
    region = "${data.aws_availability_zone.primary.region}"
}
provider "aws" {
    alias = "secondary_az"
    region = "${data.aws_availability_zone.secondary.region}"
}

# ... and then some resources in those AZs

Love this idea, it would solve some of the issues we were having. I attempted to do something very similar to this by hacking together a module that would execute a local_exec and pass the describe/get outputs to a template & then leverage the rendered template as an output. This had a host of issues and race conditions so we abandoned the idea entirely.

Sorry for creeping on your Gists, but I am always interested to see how you are trying to move the platform forward as your general ideas align closely with some of our use cases.

Somewhat off topic, but your example at the end isn't entirely accurate, since the region will always actually be the same between the AZ primary and secondary but we get the gist :)