Skip to content

Instantly share code, notes, and snippets.

@aricart

aricart/README.md

Created December 12, 2024 21:59
Show Gist options
  • Save aricart/90a40dada68f17a890aaf643a0c459ef to your computer and use it in GitHub Desktop.
Save aricart/90a40dada68f17a890aaf643a0c459ef to your computer and use it in GitHub Desktop.
Download ZIP
Recover Operator
Raw
README.md

Reissue

nsc has a reissue operator command that changes the operator's subject (main identity) with a new key and then traverses all known accounts (must be in the nsc data directory), and re-issues them with the new operator key.

If the --convert-to-signing-key option is used, the initial subject for the operator is added as a signing key. Ensuring that accounts in the wild (not found in the nsc data directory) will continue to work.

If the -K or --private-key option is specified along with a private operator key, the command will use the specified identity as the new subject and sign accounts with it.

How to use:

# point nsc to some test directory - so we can play with adding junk
aricart@mac /tmp> set -x XDG_CONFIG_HOME /tmp/nsc/config
aricart@mac /tmp> set -x XDG_DATA_HOME /tmp/nsc/data

# Add an operator
aricart@mac /tmp> nsc add operator O
[ OK ] generated and stored operator key "OCWZWRN7S2KLCYKCZCERAMM42MD4CYVTJ5EGIVM5XFUSL6RBKMFHXSAF"
[ OK ] added operator "O"
[ OK ] When running your own nats-server, make sure they run at least version 2.2.0
# set some server urls so we can push accounts etc.
aricart@mac /tmp> nsc edit operator --service-url nats://localhost:4222 --account-jwt-server-url nats://localhost:4222
[ OK ] set account jwt server url to "nats://localhost:4222"
[ OK ] added service url "nats://localhost:4222"
[ OK ] edited operator "O"

# Add a system account
aricart@mac /tmp> nsc add account SYS
[ OK ] generated and stored account key "ACIFUINKM263Y7Q64DMQS4JF5K2P5QB3MG2RK23VLBNYCDDS2LKBQTHU"
[ OK ] added account "SYS"

# Set it as system
aricart@mac /tmp> nsc edit operator --system-account SYS
[ OK ] set system account "ACIFUINKM263Y7Q64DMQS4JF5K2P5QB3MG2RK23VLBNYCDDS2LKBQTHU"
[ OK ] edited operator "O"

# Add a regular account
aricart@mac /tmp> nsc add account A
[ OK ] generated and stored account key "AARSZE42ERPBLNKVHZ2OJTHBXV4TOSFCYGTUC32MHKEPHOH2XD5RCEMP"
[ OK ] added account "A"

# List the keys so we can see the identities
aricart@mac /tmp> nsc list keys -A
+------------------------------------------------------------------------------------------+
|                                           Keys                                           |
+--------+----------------------------------------------------------+-------------+--------+
| Entity | Key                                                      | Signing Key | Stored |
+--------+----------------------------------------------------------+-------------+--------+
| O      | OCWZWRN7S2KLCYKCZCERAMM42MD4CYVTJ5EGIVM5XFUSL6RBKMFHXSAF |             | *      |
|  A     | AARSZE42ERPBLNKVHZ2OJTHBXV4TOSFCYGTUC32MHKEPHOH2XD5RCEMP |             | *      |
|  SYS   | ACIFUINKM263Y7Q64DMQS4JF5K2P5QB3MG2RK23VLBNYCDDS2LKBQTHU |             | *      |
+--------+----------------------------------------------------------+-------------+--------+

# Describe the account so we can see the issuer
aricart@mac /tmp> nsc describe account
+--------------------------------------------------------------------------------------+
|                                   Account Details                                    |
+---------------------------+----------------------------------------------------------+
| Name                      | A                                                        |
| Account ID                | AARSZE42ERPBLNKVHZ2OJTHBXV4TOSFCYGTUC32MHKEPHOH2XD5RCEMP |
| Issuer ID                 | OCWZWRN7S2KLCYKCZCERAMM42MD4CYVTJ5EGIVM5XFUSL6RBKMFHXSAF |
| Issued                    | 2024-12-12 21:07:33 UTC                                  |
| Expires                   |                                                          |


# Create a nats-server configuration file
nsc generate config --nats-resolver --sys-account SYS --config-file /tmp/server.conf
[ OK ] wrote server configuration to `/tmp/server.conf`
Success!! - generated `/tmp/server.conf`

# Edit the resolver's `dir` configuration to point somewhere reasonable

# Push the accounts, now we have a system that is running
aricart@mac /tmp> nsc push -A
[ OK ] push to nats-server "nats://localhost:4222" using system account "SYS":
       [ OK ] push A to nats-server with nats account resolver:
              [ OK ] pushed "A" to nats-server NC3I3ONY3THJI2SAUW6LW2W6JYQ5A35KSKHZBQVZRZMNTDPALS6NONFK: jwt updated
              [ OK ] pushed to a total of 1 nats-server
       [ OK ] push SYS to nats-server with nats account resolver:
              [ OK ] pushed "SYS" to nats-server NC3I3ONY3THJI2SAUW6LW2W6JYQ5A35KSKHZBQVZRZMNTDPALS6NONFK: jwt updated
              [ OK ] pushed to a total of 1 nats-server
       
# The dir for the resolver should have our accounts
aricart@mac /tmp> tree /tmp/jwt/
/tmp/jwt/
├── AARSZE42ERPBLNKVHZ2OJTHBXV4TOSFCYGTUC32MHKEPHOH2XD5RCEMP.jwt
└── ACIFUINKM263Y7Q64DMQS4JF5K2P5QB3MG2RK23VLBNYCDDS2LKBQTHU.jwt


# If you followed along, now we have a system that has one operator, two accounts.
# Time to recreate the disaster
aricart@mac /tmp> nsc env
+----------------------------------------------------------------------------------------------------------------+
|                                                NSC Environment                                                 |
+--------------------------+-----+-------------------------------------------------------------------------------+
| Setting                  | Set | Effective Value                                                               |
+--------------------------+-----+-------------------------------------------------------------------------------+
| $XDG_CONFIG_HOME         | Yes | /tmp/nsc/config                                                               |
| $XDG_DATA_HOME           | Yes | /tmp/nsc/data                                                                 |

# Gulp!
aricart@mac /tmp [1]> rm -Rf /tmp/nsc

# Make a directory to put a bunch of random JWTs
aricart@mac /tmp> mkdir disaster

# Copy the operator from the resolver configuration, save it in a file
# Copy each of the JWTs in the resolver preload - format is:
# public account: jwt
# Hopefully it is only one account the SYS
# Copy all the JWTs in the directory where the resolver put JWTs, to disaster
aricart@mac /tmp> tree disaster
disaster
├── AARSZE42ERPBLNKVHZ2OJTHBXV4TOSFCYGTUC32MHKEPHOH2XD5RCEMP.jwt
├── ACIFUINKM263Y7Q64DMQS4JF5K2P5QB3MG2RK23VLBNYCDDS2LKBQTHU.jwt
├── account.jwt
└── operator.jwt


# The fix command to the rescue... (nsc fix --help)

nsc fix --in disaster --out fixed
[WARN] :
       [ OK ] Find NKeys
       [WARN] Find Operators:
              [WARN] error loading /tmp/disaster/account.jwt: illegal base64 data at input byte 86
              [ OK ] operator OCWZWRN7S2KLCYKCZCERAMM42MD4CYVTJ5EGIVM5XFUSL6RBKMFHXSAF:
                     [ OK ] loaded from /tmp/disaster/operator.jwt
       [WARN] Find Accounts:
              [ OK ] account AARSZE42ERPBLNKVHZ2OJTHBXV4TOSFCYGTUC32MHKEPHOH2XD5RCEMP:
                     [ OK ] loaded from /tmp/disaster/AARSZE42ERPBLNKVHZ2OJTHBXV4TOSFCYGTUC32MHKEPHOH2XD5RCEMP.jwt
              [ OK ] account ACIFUINKM263Y7Q64DMQS4JF5K2P5QB3MG2RK23VLBNYCDDS2LKBQTHU:
                     [ OK ] loaded from /tmp/disaster/ACIFUINKM263Y7Q64DMQS4JF5K2P5QB3MG2RK23VLBNYCDDS2LKBQTHU.jwt
              [WARN] error loading /tmp/disaster/account.jwt: illegal base64 data at input byte 86
       [WARN] Find Users:
              [WARN] error loading /tmp/disaster/account.jwt: illegal base64 data at input byte 86
       [ OK ] Generate:
              [ OK ] operator O [OCWZWRN7S2KLCYKCZCERAMM42MD4CYVTJ5EGIVM5XFUSL6RBKMFHXSAF]:
                     [ OK ] stored operator
                     [ OK ] account A [AARSZE42ERPBLNKVHZ2OJTHBXV4TOSFCYGTUC32MHKEPHOH2XD5RCEMP]:
                            [ OK ] stored account
                     [ OK ] account SYS [ACIFUINKM263Y7Q64DMQS4JF5K2P5QB3MG2RK23VLBNYCDDS2LKBQTHU]:
                            [ OK ] stored account
stored 1 operators
stored 2 accounts
stored 0 users
stored 0 nkeys

# Point nsc to the recovered tree
set -x XDG_DATA_HOME /tmp/fixed
set -x XDG_CONFIG_HOME /tmp/fixed_config

# This will give an error, but created the tree it is looking for
aricart@mac /tmp> nsc list operators
no operators defined in store dir "/tmp/fixed/nats/nsc/stores"
# Move the contents of "operators" directory to "/tmp/fixed/nats/nsc/stores/"
# ie - stores/O, move "O" to /tmp/fixed/nats/nsc/stores

# Because no keys were found when we recovered the operator,
# the operator is managed, we need to fix that
vim /tmp/fixed/nats/nsc/stores/O/.nsc

# Change it so `managed` is set to `false`
# {"managed":false,"name":"O","kind":"operator","version":"1"}     

# Create a new operator key - we lost ours
aricart@mac /tmp> nsc generate nkey -o
SOAGAK6DXRF3T25S2MEFYRNP2FKSFRM247PJYN5SPK5H6HQJ37RVZXV2OU
OCDYW3VJPASS2HNLVLKU4HZE4ESSFWY4ALLG65UV7MVG6W4EC66IGE3Q

# Use the reissue with the specified key
aricart@mac /tmp> nsc reissue operator -K SOAGAK6DXRF3T25S2MEFYRNP2FKSFRM247PJYN5SPK5H6HQJ37RVZXV2OU
[ OK ] operator "O" successfully changed identity to: OCDYW3VJPASS2HNLVLKU4HZE4ESSFWY4ALLG65UV7MVG6W4EC66IGE3Q
[ OK ] account "A" re-signed
[ OK ] account "SYS" re-signed
all jobs succeeded


# The identity for the operator now is the new key
+------------------------------------------------------------------------------------------+
|                                           Keys                                           |
+--------+----------------------------------------------------------+-------------+--------+
| Entity | Key                                                      | Signing Key | Stored |
+--------+----------------------------------------------------------+-------------+--------+
| O      | OCDYW3VJPASS2HNLVLKU4HZE4ESSFWY4ALLG65UV7MVG6W4EC66IGE3Q |             | *      |
|  A     | AARSZE42ERPBLNKVHZ2OJTHBXV4TOSFCYGTUC32MHKEPHOH2XD5RCEMP |             |        |
|  SYS   | ACIFUINKM263Y7Q64DMQS4JF5K2P5QB3MG2RK23VLBNYCDDS2LKBQTHU |             |        |
+--------+----------------------------------------------------------+-------------+--------+

# And the accounts are issued by the new operator ID
aricart@mac /tmp> nsc describe account A
+--------------------------------------------------------------------------------------+
|                                   Account Details                                    |
+---------------------------+----------------------------------------------------------+
| Name                      | A                                                        |
| Account ID                | AARSZE42ERPBLNKVHZ2OJTHBXV4TOSFCYGTUC32MHKEPHOH2XD5RCEMP |
| Issuer ID                 | OCDYW3VJPASS2HNLVLKU4HZE4ESSFWY4ALLG65UV7MVG6W4EC66IGE3Q |


# If the keys for the different accounts are lost, I have a different PR that has never been merge
# that we would need to investigate if it works - hopefully the disaster is not that bad

# If we don't have the system account, keys we have to generate a new one - we don't have the key here
# Add a new system account
aricart@mac /tmp> nsc add account SYS2
[ OK ] generated and stored account key "AAGWWC37BVO4XUZTCWYOTZLA3C6UPVAUOURIRKGV4FXEZ35VW3WGXMFW"
[ OK ] added account "SYS2"
# Set it with the operator
aricart@mac /tmp> nsc edit operator --system-account SYS2
[ OK ] set system account "AAGWWC37BVO4XUZTCWYOTZLA3C6UPVAUOURIRKGV4FXEZ35VW3WGXMFW"
[ OK ] edited operator "O"
# Delete the old one
aricart@mac /tmp> nsc delete account SYS
[ OK ] expired account "SYS"
[ OK ] deleted account
[ OK ] deleted account directory

# Let's regenerate the resolver config with the new settings
nsc generate config --nats-resolver --sys-account SYS2 --config-file /tmp/server2.conf
[ OK ] wrote server configuration to `/tmp/server2.conf`
Success!! - generated `/tmp/server2.conf`

# Edit the resolver configuration to be a DIFFERENT directory
aricart@mac /tmp> vim /tmp/server2.conf

# Restart the server with the new config
nats-server -c server2.conf

# Add all the accounts we recovered which are the only accounts we know
aricart@mac /tmp> nsc push -A
[ OK ] push to nats-server "nats://localhost:4222" using system account "SYS2":
       [ OK ] push A to nats-server with nats account resolver:
              [ OK ] pushed "A" to nats-server NDQIGKLWNYZ544ZMUB5EJVNN7XEA5JDAWHAE75TKD3DRXUJ75KRR5VXU: jwt updated
              [ OK ] pushed to a total of 1 nats-server
       [ OK ] push SYS2 to nats-server with nats account resolver:
              [ OK ] pushed "SYS2" to nats-server NDQIGKLWNYZ544ZMUB5EJVNN7XEA5JDAWHAE75TKD3DRXUJ75KRR5VXU: jwt updated
              [ OK ] pushed to a total of 1 nats-server
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment