Bosh system certificate rotation

bosh-certs-rotate.txt

On director
Task kick off to rotate some certs

nats, blobstore, system ones
shows 'updating instance' calls but it's not a deploy
maybe say 'configuring instance blah/xxx'

Applying instance configuration acb/123
-- Generating: exchange key pair, new certificates
-- Encrypting and sending certificates
-- Decrypting and connecting ...
-- Connected
-- Deleting deprecated artifacts

no need to recreate vm

current
create_vm places generated certs on vm metadata
agent loads and connects to bus on bootstrap
recreate needed to reload agent and transfer cert

instead
securely send certs to agent to reload settings and reconnect

re-render nats config file? or are certs paths
director reloads gnatsd? is reload required - ya
with two CAs concatenated
new ca is active immediately. in case anything created during this time
director lock? maybe no other actions can be performed by an operator? so only rez
nah. make create-env be a thing still. doesn't take that long.
or rerender nats config and SIGHUP reload

director generates leaves
sends to agent
agent accepts new certs over nats - new message of reload_certs or something - or if existing method
makes another connection to test and ping back (director wait until ready)
then switch over to new certs connection
connection is done in code. can change agent to add new command to update_bosh_certs or something
if second connection can't be made, fail

bosh create-env - two concat CAs
bosh update-system-certificates -d zookeeper - leafs and CAs out
...
bosh create-env - new CA only
bosh update-system-certificates -d zookeeper

if director is made to rerender nats config and SIGHUP it
bosh env-ca-rotate [--nats-ca --blobstore-ca] --vars-store creds.yml [-l new values to use if not letting cli generate]
- cli generates here, what about credhub values being used in creds: pass in new values as same key

risks
missed communications
heartbeats make hm go balistic
update hm to accept yml config for 'guards' [deployment update, bosh system update]
compromise certs

security
transferring new creds over a connection that has been compromised
agent generates a keypair
transfers public key over to director
director generates new leaves from the ca
encrypt with agent pub key
send over to agent
agent decrypts and installs certs
test connection to notify completion with director
delete old certs off agent
write back settings
delete keys off director and agent

performance
kind of intense with the keys and stuff?
threads properly dispersed over workers
should be real fast