Skip to content

Instantly share code, notes, and snippets.

@alena1108

alena1108/disableEncryption

Created October 31, 2019 21:18
Show Gist options
  • Save alena1108/f55e6d50b347ed320b3fa2e48e423b62 to your computer and use it in GitHub Desktop.
Save alena1108/f55e6d50b347ed320b3fa2e48e423b62 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
disableEncryption
**What kind of request is this (question/bug/enhancement/feature request):** bug
**Steps to reproduce (least amount of steps as possible):**
- Enable custom config in an exiting rke cluster.
- Edit cluster, Add this in the YAML file
```
services:
kube-api:
secrets_encryption_config:
enabled: true
custom_config:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: k-fw5hn
secret: RTczRjFDODMwQzAyMDVBREU4NDJBMUZFNDhCNzM5N0I=
- identity: {}
```
- On clicking Save, cluster is provisioned again, goes into Updating state and comes up Active.
- The secrets are seen encrypted on the etcd
```
/ # ETCDCTL_API=3 etcdctl get /registry/secrets/default/test | hexdump -C
00000000 2f 72 65 67 69 73 74 72 79 2f 73 65 63 72 65 74 |/registry/secret|
00000010 73 2f 64 65 66 61 75 6c 74 2f 74 65 73 74 0a 6b |s/default/test.k|
00000020 38 73 3a 65 6e 63 3a 61 65 73 63 62 63 3a 76 31 |8s:enc:aescbc:v1|
00000030 3a 6b 2d 66 77 35 68 6e 3a b0 9f 5e 03 cb c6 0e |:k-fw5hn:..^....|
00000040 d7 f8 34 a3 0a 98 47 25 a0 64 2f c5 45 9f 77 0c |..4...G%.d/.E.w.|
00000050 d8 52 9f bc 2e 48 7f e1 4f b1 bf f2 bd 73 31 51 |.R...H..O....s1Q|
00000060 f2 3d 8d 00 0a 2b 44 79 b2 ee ec 04 2f 01 f9 a5 |.=...+Dy..../...|
00000070 20 00 e2 bb 4e 24 a0 c5 fe 2b ca bc 48 b8 d5 00 | ...N$...+..H...|
00000080 7b 17 00 a9 7d b0 a1 df 30 f1 9b bb 3f 5d 00 c3 |{...}...0...?]..|
00000090 cb 1a 6a 3a 55 7c cc b8 c7 41 76 7a c6 c0 48 e4 |..j:U|...Avz..H.|
000000a0 05 9e 5c d7 27 5e 98 75 a7 de 26 65 91 03 69 e7 |..\.'^.u..&e..i.|
000000b0 2e ea f0 9f 7c 54 7e 2f b6 a1 4b 78 f6 9e 40 e3 |....|T~/..Kx..@.|
000000c0 c0 9e 9b 09 d4 eb bd 9b 3c 43 7a 2b ea 71 51 6f |........<Cz+.qQo|
000000d0 81 ec 0e c0 a4 94 d5 94 a0 05 3e 5d 5b d6 e1 bc |..........>][...|
000000e0 c1 74 60 cc 82 fd 1e 4f e3 a7 d3 7d 8f c1 42 1a |.t`....O...}..B.|
000000f0 0e f8 cc 24 fa 6b 19 fb 97 5d f2 df a7 d3 64 a3 |...$.k...]....d.|
00000100 dd 80 df 83 f2 e0 8f 86 3a e0 4d 4b 35 f6 5c 2b |........:.MK5.\+|
00000110 12 01 93 be f5 3d 11 54 32 f7 c8 22 dc 12 2c 45 |.....=.T2.."..,E|
00000120 ca e2 e0 8b 69 6d ea fa 17 1d 79 1c 67 65 17 4d |....im....y.ge.M|
00000130 95 27 30 1f d4 b2 cf 28 52 9b c5 bb af 76 15 d1 |.'0....(R....v..|
00000140 e8 f4 a8 16 06 fc bc 18 0e 9c 4e 5f 26 49 a9 9e |..........N_&I..|
00000150 7f 22 85 f6 28 5c 6f 76 85 a5 fb 4e f9 8d f3 60 |."..(\ov...N...`|
00000160 09 0d e9 9e 29 d0 09 9c 37 95 23 99 e2 fa f8 5b |....)...7.#....[|
00000170 b7 b4 7f 50 ce 41 02 13 b4 28 4b a6 8d 2c d6 a2 |...P.A...(K..,..|
00000180 bc 51 72 5b 8c 29 2f a1 e8 0a |.Qr[.)/...|
0000018a
```
- `encryption.yaml` file on the control plane node is
```
root@santest114c1:/etc/kubernetes/ssl# cat encryption.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- providers:
- aescbc:
keys:
- name: k-fw5hn
secret: RTczRjFDODMwQzAyMDVBREU4NDJBMUZFNDhCNzM5N0I=
- identity: {}
resources:
- secrets
```
- Now Move identity: {} to the top of the providers list, so your config looks as follows:
```
secrets_encryption_config:
custom_config:
api_version: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- Providers:
- Identity: {}
- AESCBC:
Keys:
- Name: k-fw5hn
Secret: RTczRjFDODMwQzAyMDVBREU4NDJBMUZFNDhCNzM5N0I=
AESGCM: null
Identity: null
KMS: null
Secretbox: null
- AESCBC: null
AESGCM: null
KMS: null
Secretbox: null
Resources:
- secrets
enabled: true
```
- Click on Save. cluster is provisioned again, goes into Updating state and comes up Active.
- The secrets are seen encrypted on the etcd
```
/ # ETCDCTL_API=3 etcdctl get /registry/secrets/default/test | hexdump -C
00000000 2f 72 65 67 69 73 74 72 79 2f 73 65 63 72 65 74 |/registry/secret|
00000010 73 2f 64 65 66 61 75 6c 74 2f 74 65 73 74 0a 6b |s/default/test.k|
00000020 38 73 3a 65 6e 63 3a 61 65 73 63 62 63 3a 76 31 |8s:enc:aescbc:v1|
00000030 3a 6b 2d 66 77 35 68 6e 3a b0 9f 5e 03 cb c6 0e |:k-fw5hn:..^....|
00000040 d7 f8 34 a3 0a 98 47 25 a0 64 2f c5 45 9f 77 0c |..4...G%.d/.E.w.|
00000050 d8 52 9f bc 2e 48 7f e1 4f b1 bf f2 bd 73 31 51 |.R...H..O....s1Q|
00000060 f2 3d 8d 00 0a 2b 44 79 b2 ee ec 04 2f 01 f9 a5 |.=...+Dy..../...|
00000070 20 00 e2 bb 4e 24 a0 c5 fe 2b ca bc 48 b8 d5 00 | ...N$...+..H...|
00000080 7b 17 00 a9 7d b0 a1 df 30 f1 9b bb 3f 5d 00 c3 |{...}...0...?]..|
00000090 cb 1a 6a 3a 55 7c cc b8 c7 41 76 7a c6 c0 48 e4 |..j:U|...Avz..H.|
000000a0 05 9e 5c d7 27 5e 98 75 a7 de 26 65 91 03 69 e7 |..\.'^.u..&e..i.|
000000b0 2e ea f0 9f 7c 54 7e 2f b6 a1 4b 78 f6 9e 40 e3 |....|T~/..Kx..@.|
000000c0 c0 9e 9b 09 d4 eb bd 9b 3c 43 7a 2b ea 71 51 6f |........<Cz+.qQo|
000000d0 81 ec 0e c0 a4 94 d5 94 a0 05 3e 5d 5b d6 e1 bc |..........>][...|
000000e0 c1 74 60 cc 82 fd 1e 4f e3 a7 d3 7d 8f c1 42 1a |.t`....O...}..B.|
000000f0 0e f8 cc 24 fa 6b 19 fb 97 5d f2 df a7 d3 64 a3 |...$.k...]....d.|
00000100 dd 80 df 83 f2 e0 8f 86 3a e0 4d 4b 35 f6 5c 2b |........:.MK5.\+|
00000110 12 01 93 be f5 3d 11 54 32 f7 c8 22 dc 12 2c 45 |.....=.T2.."..,E|
00000120 ca e2 e0 8b 69 6d ea fa 17 1d 79 1c 67 65 17 4d |....im....y.ge.M|
00000130 95 27 30 1f d4 b2 cf 28 52 9b c5 bb af 76 15 d1 |.'0....(R....v..|
00000140 e8 f4 a8 16 06 fc bc 18 0e 9c 4e 5f 26 49 a9 9e |..........N_&I..|
00000150 7f 22 85 f6 28 5c 6f 76 85 a5 fb 4e f9 8d f3 60 |."..(\ov...N...`|
00000160 09 0d e9 9e 29 d0 09 9c 37 95 23 99 e2 fa f8 5b |....)...7.#....[|
00000170 b7 b4 7f 50 ce 41 02 13 b4 28 4b a6 8d 2c d6 a2 |...P.A...(K..,..|
00000180 bc 51 72 5b 8c 29 2f a1 e8 0a |.Qr[.)/...|
0000018a
```
- `encryption.yaml` file looks like
```
root@santest114c1:/etc/kubernetes/ssl# cat encryption.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- providers:
- identity: {}
- aescbc:
keys:
- name: k-fw5hn
secret: RTczRjFDODMwQzAyMDVBREU4NDJBMUZFNDhCNzM5N0I=
- {}
resources:
- secrets
```
- Remove secrets_encryption_config directive from cluster.yml
- The cluster goes into updating state and fails to come up.
Error: `Error getting secret: illegal base64 data at input byte 3`
<img width="1290" alt="Screen Shot 2019-11-01 at 2 11 02 AM" src="https://user-images.githubusercontent.com/26032343/67985613-39920e80-fc4f-11e9-90cd-989944265ac7.png">
- encryption.yaml file is deleted on the control plane now.
- And the secrets are still see encrypted
```
/ # ETCDCTL_API=3 etcdctl get /registry/secrets/default/test | hexdump -C
00000000 2f 72 65 67 69 73 74 72 79 2f 73 65 63 72 65 74 |/registry/secret|
00000010 73 2f 64 65 66 61 75 6c 74 2f 74 65 73 74 0a 6b |s/default/test.k|
00000020 38 73 3a 65 6e 63 3a 61 65 73 63 62 63 3a 76 31 |8s:enc:aescbc:v1|
00000030 3a 6b 2d 66 77 35 68 6e 3a b0 9f 5e 03 cb c6 0e |:k-fw5hn:..^....|
00000040 d7 f8 34 a3 0a 98 47 25 a0 64 2f c5 45 9f 77 0c |..4...G%.d/.E.w.|
00000050 d8 52 9f bc 2e 48 7f e1 4f b1 bf f2 bd 73 31 51 |.R...H..O....s1Q|
00000060 f2 3d 8d 00 0a 2b 44 79 b2 ee ec 04 2f 01 f9 a5 |.=...+Dy..../...|
00000070 20 00 e2 bb 4e 24 a0 c5 fe 2b ca bc 48 b8 d5 00 | ...N$...+..H...|
00000080 7b 17 00 a9 7d b0 a1 df 30 f1 9b bb 3f 5d 00 c3 |{...}...0...?]..|
00000090 cb 1a 6a 3a 55 7c cc b8 c7 41 76 7a c6 c0 48 e4 |..j:U|...Avz..H.|
000000a0 05 9e 5c d7 27 5e 98 75 a7 de 26 65 91 03 69 e7 |..\.'^.u..&e..i.|
000000b0 2e ea f0 9f 7c 54 7e 2f b6 a1 4b 78 f6 9e 40 e3 |....|T~/..Kx..@.|
000000c0 c0 9e 9b 09 d4 eb bd 9b 3c 43 7a 2b ea 71 51 6f |........<Cz+.qQo|
000000d0 81 ec 0e c0 a4 94 d5 94 a0 05 3e 5d 5b d6 e1 bc |..........>][...|
000000e0 c1 74 60 cc 82 fd 1e 4f e3 a7 d3 7d 8f c1 42 1a |.t`....O...}..B.|
000000f0 0e f8 cc 24 fa 6b 19 fb 97 5d f2 df a7 d3 64 a3 |...$.k...]....d.|
00000100 dd 80 df 83 f2 e0 8f 86 3a e0 4d 4b 35 f6 5c 2b |........:.MK5.\+|
00000110 12 01 93 be f5 3d 11 54 32 f7 c8 22 dc 12 2c 45 |.....=.T2.."..,E|
00000120 ca e2 e0 8b 69 6d ea fa 17 1d 79 1c 67 65 17 4d |....im....y.ge.M|
00000130 95 27 30 1f d4 b2 cf 28 52 9b c5 bb af 76 15 d1 |.'0....(R....v..|
00000140 e8 f4 a8 16 06 fc bc 18 0e 9c 4e 5f 26 49 a9 9e |..........N_&I..|
00000150 7f 22 85 f6 28 5c 6f 76 85 a5 fb 4e f9 8d f3 60 |."..(\ov...N...`|
00000160 09 0d e9 9e 29 d0 09 9c 37 95 23 99 e2 fa f8 5b |....)...7.#....[|
00000170 b7 b4 7f 50 ce 41 02 13 b4 28 4b a6 8d 2c d6 a2 |...P.A...(K..,..|
00000180 bc 51 72 5b 8c 29 2f a1 e8 0a |.Qr[.)/...|
0000018a
```
**Environment information**
- Rancher version (`rancher/rancher`/`rancher/server` image tag or shown bottom left in the UI): 2.3-head - latest
- Installation option (single install/HA): single
<!--
If the reported issue is regarding a created cluster, please provide requested info below
-->
**Cluster information**
- Cluster type (Hosted/Infrastructure Provider/Custom/Imported): rke do
- Kubernetes version (use `kubectl version`):
```
v1.14.8
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment