PodPreset Ramblings

PodPreset Ramblings

Some aspects that influence my thinking on this:
- having PodPresets in a Binding is causing problems
- first, because we'll be constantly chasing the real PodPreset definition
- second, because we actually don't want a full PodPreset with all of its features,
  we just need the bits related to Secrets
- third, PodPresets (as of now) are immutable
- we need to allow for people to specify more than one LabelSelector at a time to support
  upgrading of apps
- as long as the LabelSelector is immutable and embedded in a Binding we should separate the
  idea of asking for creds from linking the creds to an app to allow for changing LabelSelectors
  w/o having to create a new Binding each time.
- as much as I would really like to reduce the number of resources for a better UX we may
  have to give up on that to not limit the flexibility that Kube allows
- we need to support asking for credentials w/o binding to an app (cf calls these ServiceKeys)

I see a few options going forward - minimalist ones first:

1:
=============================================================================================
- Remove PodPreset from Bindings
- Rename Binding to something else that implies its just there to ask for the creds and
  create a Secret (eg. ServiceKey). But if this is too radical I can live with
  calling it Binding, but it just feels odd since we're not "binding" anything any more.
- User can then create as many PodPresets (or anything else) they want to inject/link the
  Secret to their "app"
- we leave app-id blank for the OSB API CreateBinding() calls since we never know what "app"
  it will be bound to.

UX flow:
- kubectl create -f serviceInstance.yaml
- kubectl create -f serviceKey.yaml (implicitly creates a Secret)
- kubectl create -f podpreset1.yaml (to inject the secret into our "app")
- kubectl apply -f podpreset1.yaml (to inject the secret into 2nd definition of "app" - e.g. during an upgrade)


2:
=============================================================================================
- Modify PodPreset subresource in Binding so that has just the bits of PodPreset that relate to
  Secrets being injected into containers (probably do not call it podpreset - SecretInjection?)
- Allow for this subresource to be updated (and even deleted)
- Allow for any number of these subresource to be specified so we can support binding to
  more than one definition of an app at the same time
- Upon modification of these subresources we create/delete/update PodPresets accordingly
- The user is then expected to update the Binding's subresources as they modify their
  definition of "the app"
- app-id can then be the ID of the Binding because its the same no matter how the definition
  of "the app" changes.
- Note that if the Binding doesn't have any subresources then this becomes like a ServiceKey.

UX flow:
- kubectl create -f serviceInstance.yaml
- kubectl create -f serviceBinding.yaml (implicitly creates a Secret and PP's based on the subresources)
- kubectl apply -f serviceBinding.yaml (e.g. during an app migration and the labelSelector changes)



Comments:
- the 1st approach feels more Kube-ish because it exposes everything
- the 2nd approach implicitly groups multiple definitions of "an app" into one thing (the
  Binding). This is different from the 1st approach which almost encourages the same set
  of Creds to be linked with any random resource. This might be important because the 1st
  approach doesn't lend itself to defining a clear idea of when a new set of creds are
  supposed to be asked for - meaning for each new "app".
- while the 2nd approach tries to present a better UX, there is nothing stopping
  people from using it like the 1st approach, meaning people can just not specify any of
  subresources and use the secret for any purpose. In that respect, option 2 might be more robust
  because it keeps the simple use cases simple but allows for the more advanced scenarios as well.
- Neither option helps in the usecase I've previously mentioned where we may want to 
  inject all creds into one env var (VCAP_SERVICES) to help CF apps migrate.


3:
=============================================================================================
- Follow-on to option #1 but also create a new resource like Paul suggested: PodSecretBinding
- I think Binding should be renamed ServiceKey in this case too since it doesn't Bind anything.
- This resource has a ref to a ServiceKey, and a set of subresources similar to
  #2 above. So, a subset of a podpreset - just the secret stuff.
- The app-id sent to the broker would be the id of this resource
- Allow for more than one (or zero) of these subresources
- Allow for more than one ref to ServiceKeys so we have a shared app-id across
  all OSB API Bindings for the same app.
- Allow for PodSecretBinding, and its subresources, to be modified.
- app-id would be the id of the PodSecretBinding.

UX flow:
- kubectl create -f serviceInstance.yaml
- kubectl create -f serviceCredentialRequest.yaml (creates a secret)
- kubectl create -f podsecretbinding.yaml (creates a podpreset based on the subresources)
- kubectl apply -f podsecretbinding.yaml (to allow for a changing definition of "our app")

Comments:
- I think its important to support an MxN relationship in this resource because w/o that
  it doesn't provide any value over just asking people to create a PodPreset themselves.
- This tries to get people to think about "an app" and how it will related to the
  secret/binding - so its like #2, but not as "wild west" as #1.
- unlike #2 it doesn't try to put too much into the Binding resource, but on the downside
  it requires an extra resource that will feel a lot like a PodPreset in many cases, and 
  could cause confusion. IOW, PodSecretBinding is optional and might be viewed as a
  subset/duplicate of just creating a PodPreset. Which it does do under the covers.
- We could add VCAP_SERVICES migration support with this option.


4:
=============================================================================================
- With the introduction of Initializers, what if we had an initializer that looked for
  an annotation on a resource (app) being created and that annotation referenced a 
  serviceInstance.  We could then create the ServiceKey, Secret, PodPreset, etc.
  (or modify the pod spec directly ourselves w/o podpreset)
- We then get out of the business of asking people to create a duplicate set of LabelSelectors
  that must remain in-sync with their real app definition's.
- It could even reference a serviceClass instead of an Instance, then we could create the
  Instance for them
- Two "apps" pointing to the same serviceInstance (by name) share that instance.
- More than one annotation in a single resource means they're binding to more than one instance.
- app-id in this case would be the id of the resource on which the annotation was attached.
- Unbind would be done by removing the annotation.

UX flow:
- kubectl create -f deployment.yaml (references an instance)
- kubectl apply -f deployment.yaml (update app as they do today, no change)

Comments:
- CF supports this model.
- This is probably the most UX friendly option of them all.
- This doesn't help with associating more than one resource/app as "the same app", unless
  we ask for some extra info to link them.  Which might feel annoying.
- We can not do just #4 w/o one of the others because we will need to support creating
  ServiceKeys w/o binding them to an app.  So, minimally, we'd need #1 too.
- Does support the VCAP_SERVICES migration usecase.




NET:
=============================================================================================
- These are not all mutually exclusive.
- In terms of an MVP, option #1 is the best choice as that introduces the bare minimum needed
  to do the basic OSB API operations.
- Options #2 and #3 try to add some UX helper functions.
- IMO, the real question is whether we need to do option #2 or #3 if we go beyond #1.

My proposal: do #1 and #4 and then see based on real-world usage if we need #2 or #3.
Or just #1 if people want time to think about #4 since its an optional UX thing, sort of.