multiple-kustomizations.md

The Issue

Currently, many options within kustomization.yaml are global, such as commonLabels, namespace, etc. This can lead to issues when you want to create an overlay which includes multiple resources, some sharing values such as the namespace, but then some monitoring resources also needing to be created in a different namespace. I'll provide an example to illustrate my problem.

Example

I'm currently trying to define a "monitoring" overlay which is composed of multiple applications: prometheus-operator, grafana, etc. I also have an overlay for each K8s environment I manage: example, dev, staging and production each have their own overlay. Each environment overlay would include the monitoring overlay.

# environments/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../overlays/monitoring

# overlays/monitoring/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: monitoring
resources:
  - namespace.yaml
  - prometheus-operator
  - grafana

The trouble is when I want to also include things which are monitoring related, but need to live in kube-system, for example coredns. The coredns resources needs to be created in kube-system to define a Service and ServiceMonitor that match the coredns pods that run in kube-system.

# overlays/monitoring/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: monitoring
resources:
  - namespace.yaml
  - prometheus-operator
  - grafana
  # I can't just add the coredns resources/base here. It the needs namespace to be kube-system.
  # - coredns

So far the only way I've been able to do what I want is one of two choices. I can put the coredns definition in a different overlay. This defeats the purpose of defining a monitoring overlay which has all the monitoring related resources defined in it. The other option is defining another kustomization.yaml which has the options I need.

However, because my monitoring overlay already has a kustomization.yaml I need to create another directory for this file, say overlays/monitoring/coredns/kustomization.yaml. This is also a problem though, because I can't add monitoring/coredns into my monitoring/kustomization.yaml because to the namespace field set to monitoring.

So instead I end up with something like this:

# overlays/monitoring/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - main
  - coredns

# overlays/monitoring/main/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: monitoring
resources:
  - namespace.yaml
  - prometheus-operator
  - grafana

# overlays/monitoring/coredns/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: kube-system
resources:
  - coredns

This applies to all transformers, as they're all global in nature. I run into this issue particularly when I want to do namespaces, common labels, and annotations. Usually I just avoid it by not applying as much metadata within the upper level kustomizations, and instead duplicate the transformers in the leaf kustomizations.

Proposed Solution (multiple kustomizations in a single file)

Generally, I would like to be able to just do something like this:

# overlays/monitoring/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: main

namespace: monitoring
resources:
  - namespace.yaml
  - prometheus-operator
  - grafana

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: coredns

namespace: kube-system
resources:
  - coredns

The idea is that you can define multiple kustomizations in a single kustomization.yaml, and this would basically be a list of kustomizations to process. You could also change metadata.name to name to make it top-level, or it could be optional or skipped in the initial version. A name might be nice for errors, but you could also output the index of the list of kustomizations in the file.

Alternative Solution (multiple kustomizations in a directory)

I can think of an alternative where a new field was added to kustomization.yaml, which allows referencing another kind: Kustomization file. This isn't mutually exclusive with the proposed solution, but I think it's more complex and a still a bit too verbose, but could still be useful in situations where you're doing many kustomizations in a single directory, and don't want a single really long kustomization.yaml.

# overlays/monitoring/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

kustomizations:
  - monitoring-kustomization.yaml
  - coredns-kustomization.yaml

# overlays/monitoring/monitoring-kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: monitoring
resources:
  - namespace.yaml
  - prometheus-operator
  - grafana

# overlays/monitoring/coredns-kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: kube-system
resources:
  - coredns

Related issues

kubernetes-sigs/kustomize#2168 Is similar, and if combined with this functionality could make a few big improvements in allowing one to define transformations separately from resources, allowing more complex combinations of transformers without duplicating them. However, splitting up resources requires more kustomization.yaml's, and thus more directories, so it doesn't fix the root issue discussed. Additionally, the alternate solution seems similar to this issue, as they both allow a Kustomization-like file to be referenced in a Kustomization.

Interesting! One alternative might be to use inline patches to define namespaces instead of using Kustomize's namespace built-in functionality, then at least you have more control over how the namespaces are applied.

For example:

1. Create labels that can be used to select coredns and monitoring resources (and distinguish them from the normal app resources). e.g.

• In monitoring/kustomization.yaml define a default label that is applied to all of the monitoring and coredns resources:

  commonLabels:
    monitoring: monitoring

• Then define a separate label on all of the coredns resources (can't be monitoring because that will get overwritten), e.g. coredns: coredns
• Of course if you define the label on each resource explicitly then you won't need to worry about overwriting labels

2. Then use inline patches to apply the namespace on each of the monitoring/coredns resources, adding the following to monitoring/kustomization.yaml:

patches:
  - patch: |-
      - op: replace
        path: /metadata/namespace
        value: monitoring
    target:
      labelSelector: "monitoring=monitoring"
  - patch: |-
      - op: replace
        path: /metadata/namespace
        value: coredns
    target:
      labelSelector: "coredns=coredns"

3. Then in your top-level kustomization, e.g. dev/kustomization.yaml you can apply a similar patch to all resources that don't match the new labels:

patches:
  - patch: |-
      - op: replace
        path: /metadata/namespace
        value: dev
    target:
      labelSelector: "monitoring!=monitoring,coredns!=coredns"

... So a little bit verbose, but seems to work. Not sure if it would be possible to create a transformer to do this?

It may be possible with your suggestion, but it defeats the point. I don't want to write patches like this, I just want to use the transformers. Also, this applies to all transformers, not just namespaces. I'll clarify that in the description.

Additionally, in this case, the coredns resources aren't managed by kustomize, they're pre-installed in EKS, so I can't use kustomize to just apply labels like that.

I suspect I'll create an issue for this either way. Thanks for your suggestion.