K8S On-call Role Elevation

Intro

Scenario: you are a developer on-call, there is an issue with the cluster that you need to investigate and correct.

sudo for kubectl was the basic idea (Mattz):

- assume admin role
- ideally still have constraints as to which namespaces it can impact 
     (i.e. stay out of kube-system, monitoring, etc...)
- has a fixed time/window for use (different than sudo -- like an expiration time or timeout)
- has additional logging/audit constraints around it for tracking
    - maybe fires off a mail to admin/mgmt that it has been invoked?
    - definitely shows in the logs

Deny!

It'd be great if there was a deny in kubernetes, specifically for kube-system namespace! Not going to happen though according to github issues. See discussion here

Impersonation

Idea -- use impersonation as it already exists in the system for a big chunk of what we want above.

kubectl --as <user-to-impersonate> ...
kubectl --as <user-to-impersonate> --as-group <group-to-impersonate> ...

Some links and examples (more for my reference than anything):

Also, this covers some of our audit ask:

Kubernetes impersonation is well designed regarding audit trails, as API calls get logged with
full original identity (user) and impersonated user (impersonatedUser).

That covers the role assumption, but it doesn't cover our desire to control namespace access.

Ok, impersonation but how to CLI easier

What about a kubectl sudo plugin? Kubectl plugins in kubernetes are pretty simple things. From the docs:

A plugin is nothing more than a standalone executable file, whose name begins with kubectl-

They can be created with a bash script. Here's an existing kubectl sudo that we could use as a reference with us just changing the user or group used:

https://github.com/postfinance/kubectl-sudo/blob/master/bash/kubectl-sudo

Usage of this script first without the sudo:

$ kubectl get nodes
Error from server (Forbidden): nodes is forbidden: User "bofh" cannot list nodes at the cluster scope

and then using it:

$ kubectl sudo get nodes
NAME                     STATUS   ROLES    AGE   VERSION
kubelet1.example.com     Ready    <none>   96d   v1.11.2
kubelet2.example.com     Ready    <none>   96d   v1.11.2

Looking at the script, it just wraps the impersonate subcommand we talked about previously. Substitute in the role created for your use-case and off you go. Okay, one more piece to go then...

Protect Namespaces

How then can we restrict access (verbs) on our more sensitive namespaces? What specifically does that look like in terms of an RBAC RoleBinding or ClusterRoleBinding?

Digging around, I found this kubernetes operator:

https://github.com/FairwindsOps/rbac-manager

What is it?

This is an operator that supports declarative configuration for RBAC with new custom 
resources. Instead of managing role bindings or service accounts directly, you can specify 
a desired state and RBAC Manager will make the necessary changes to achieve that state.

So an operator that runs in the cluster that takes one set of YAML and applies it to the RoleBindings, ClusterRoleBindings, etc... of the cluster to get to your desired state.

What was most interesting with this for our use case is the ability to control access to namespaces that have certain labels

apiVersion: rbacmanager.reactiveops.io/v1beta1
kind: RBACDefinition
metadata:
  name: dev-access
rbacBindings:
  - name: dev-team
    subjects:
      - kind: Group
        name: dev-team
    roleBindings:
      - clusterRole: edit
        namespaceSelector:
          matchLabels:
            team: dev

In the example above, Role Bindings would automatically get created for each Namespace 
with a team=dev label. This supports the same functionality as other Kubernetes label selectors...

We would have some tag created for the namespaces then, oncall, that would allow access to these resources.

On top of that, we'd have to still allow this Role the verbs get, list, and watch for kube-system and any other namespaces we're protecting.

Put it all together

install RBACManager on cluster
create oncall RoleBinding via RBACManager such that it has full access to all namespaces tagged as oncall, can't manipulate namespaces at all, has limited access to kube-system and other protected, and whatever else we come up with on review.
install kubectl plugin, kubectl sudo, and customize to impersonate oncall role just created
could also just link or alias that same plugin bash script as ksudo or something like that or set up an alias
if it is only on certain jumpboxes additional logging could be added, alerting, etc... as appropriate for the host (fire to syslog, write to some other log file, send off email)