Dynamically populating manifests from AWS tags

Motivation

We’d like to re-use the from_tags pattern eventually for all of our imports to dynamically populate fields. The goal is to make easy to add a from_tags field to our other types.

Examples

subnets:
  from_tags:
    org: bi
    type: public

vpc:
  from_tags:
    org: bi

cluster:
  from_tags:
    org: bi
    team: analytics

Proposals

workload.go

package manifest

type Tags map[string]StringSliceOrString

type Subnets struct {
    IDs      []string
    FromTags Tags `yaml:"from_tags"`
}

type dynamicSubnets struct {
    cfg    *Subnets
    client EC2
}

// load populates the subnet's IDs field if the client is using tags.
func (dyn *dynamicSubnets) load() error {
    if len(dyn.cfg.IDs) > 0 {
        return nil
    }
    // Call ids := dyn.client.SubnetIDs()... and override.
    dyn.cfg.IDs = ids
}

We will start by introducing a new type for each field with tags type dynamicXXX struct that has a load() error method to populate its fields dynamically. The key observation is that .FromTags is never going to be used by the stack package and clients can rely on .IDs to be always populated.

loader.go

// EC2 is the AWS EC2 interface.
type EC2 interface {
    SubnetIDs(filters ...ec2.Filter) ([]string, error)
}

// DynamicLoadBalancedWebService represents a *LoadBalancedWebService 
// that can be dynamically populated.
type DynamicLoadBalancedWebService struct {
    Mft *LoadBalancedWebService

    // Clients required to dynamically populate.
    EC2 EC2
}

// DynamicBackendService represents a *BackendService 
// that can be dynamically populated.
type DynamicBackendService struct {
    Mft *BackendService

    // Clients required to dynamically populate.
    EC2 EC2
}

// Load dynamically populates all fields in the manifest.
func (dyn *DynamicLoadBalancedWebService) Load() error {
    loaders := []Loader{
        &dynamicSubnets{
            cfg: &dyn.Mft.Network.VPC.Placement.PlacementArgs.Subnets,
            client: dyn.EC2,
        },
    }
    return loadAll(loaders)
}

func (dyn *DynamicBackendService) Load() error {
    // Similar to above.
}


// loader is the interface to dynamically populate manifest fields.
type loader interface {
    load() error
}

func loadAll(loaders []loader) error { 
    // call in a for-range loader.Load()...
}

Then each manifest type like LoadBalancedWebService or BackendService can have a wrapper type type DynamicXXX that can be dynamically populate and surface a Load() error method.

efekarakus/proposal.md

Dynamically populating manifests from AWS tags

Motivation

Examples

Proposals