Secrets-encryption rotation

[dereknola]

Proposed Changes

Introduces new feature: secrets-encryption keys rotation and enable/disable of encryption. The is all controlled via a new sub command k3s secrets-encrypt. Documentation can be found at https://rancher.com/docs/k3s/latest/en/security/secrets_encryption/

NAME:
   k3s secrets-encrypt - Control secrets encryption and keys rotation

USAGE:
   k3s secrets-encrypt command [command options] [arguments...]

COMMANDS:
   status     Print current status of secrets encryption
   enable    Enable secrets encryption
   disable   Disable secrets encryption
   prepare   Prepare for encryption keys rotation
   rotate     Rotate secrets encryption keys
   reencrypt  Reencrypt all data with new encryption key

Types of Changes

New Feature
New Integration Test (covers single node)

Verification

Formal documentation to follow:
You can run k3s secrets-encrypt status at any time to see the current stage/state of encryption
For a single node cluster:

• Start server with k3s server --secrets-encryption
• Create a secret to check

kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
kubectl describe secret secret1 -n default

• Call k3s secrets-encrypt prepare
• Call k3s secrets-encrypt status, note, the addition of a new key
• Kill k3s server and restart with same arguments
• Call k3s secrets-encrypt rotate
• Call k3s secrets-encrypt status, note the keys swap
• Kill k3s server, restart with same arguments
• Call k3s secrets-encrypt reencrypt
• Check that secret is still encrypted

kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
kubectl describe secret secret1 -n default

For external DB:

• Start up a external DB (I used mySQL for this)

• Start up 3 K3s servers joined to mySQL
  k3s server --secrets-encryption --datastore-endpoint "mysql://root:mysql@tcp(192.168.1.200:3306)/k3s"

• Select ONE server (S1 going forward) to perform the rotate on (doing any stage on any server is supported)

• Run k3s secrets-encrypt prepare on S1

• Restart S1 with same arguments

• Once S1 is back up, restart the other two servers

• Call k3s secrets-encrypt status on S1 and S2, info should be the same on both

• Run k3s secrets-encrypt rotate on S1

• Restart S1

• Once S1 is back up, restart the other two servers

• Call k3s secrets-encrypt status on S1 and S2, info should be the same on both

• Run k3s secrets-encrypt reencrypt on S1

• Restart S1

• Once S1 is back up, restart the other two servers

• Call k3s secrets-encrypt status on S1 and S2, info should be the same on both

Linked Issues

#4254
#3407

User-Facing Change

A CLI subcommand to control secrets encryption. Supports key rotation and enabled/disable of secrets encryption.

Further Comments

• Documentation to follow in next PR.
• This feature works on all 3 DB types (single node, HA with embedded etcd, HA with external DB)

[mdrahman-suse]

Verified in k3s with master commit d71b335 and RC with single node and 3 node cluster following the steps described in the ticket and observed the expected test results

3 node cluster

kubectl get nodes,pods -A -o wide
NAME                    STATUS   ROLES                       AGE   VERSION            INTERNAL-IP     EXTERNAL-IP     OS-IMAGE           KERNEL-VERSION   CONTAINER-RUNTIME
node/s1   Ready    <none>                      73m   v1.22.4-rc3+k3s2   <REDACTED>      <REDACTED>     Ubuntu 20.04 LTS   5.4.0-1009-aws   containerd://1.5.8-k3s1
node/s2   Ready    control-plane,etcd,master   77m   v1.22.4-rc3+k3s2   <REDACTED>      <REDACTED>    Ubuntu 20.04 LTS   5.4.0-1009-aws   containerd://1.5.8-k3s1
node/s3   Ready    control-plane,etcd,master   74m   v1.22.4-rc3+k3s2   <REDACTED>      <REDACTED>     Ubuntu 20.04 LTS   5.4.0-1009-aws   containerd://1.5.8-k3s1
node/a1   Ready    control-plane,etcd,master   74m   v1.22.4-rc3+k3s2   <REDACTED>      <REDACTED>   Ubuntu 20.04 LTS   5.4.0-1009-aws   containerd://1.5.8-k3s1

Prepare stage: k3s secrets-encrypt prepare on s1

prepare completed successfully
$ sudo k3s secrets-encrypt status
Encryption Status: Enabled
Current Rotation Stage: prepare
Server Encryption Hashes: hash does not match between s1 and s2

Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey
        AES-CBC   aescbckey-2021-12-15T22:20:54Z

After restarts on all 3 nodes

$ sudo k3s secrets-encrypt status
Encryption Status: Enabled
Current Rotation Stage: prepare
Server Encryption Hashes: All hashes match

Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey
        AES-CBC   aescbckey-2021-12-15T22:20:54Z

Rotate stage: k3s secrets-encrypt rotate on s1

rotate completed successfully
$ sudo k3s secrets-encrypt status
Encryption Status: Enabled
Current Rotation Stage: rotate
Server Encryption Hashes: hash does not match between s1 and s2

Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey-2021-12-15T22:20:54Z
        AES-CBC   aescbckey

After restarts on all 3 nodes

$ sudo k3s secrets-encrypt status
Encryption Status: Enabled
Current Rotation Stage: rotate
Server Encryption Hashes: All hashes match

Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey-2021-12-15T22:20:54Z
        AES-CBC   aescbckey

Reencrypt stage: k3s secrets-encrypt reencrypt on s1

reencryption started
$ sudo k3s secrets-encrypt status
Encryption Status: Enabled
Current Rotation Stage: reencrypt_active
Server Encryption Hashes: hash does not match between s1 and s2

Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey-2021-12-15T22:20:54Z
        AES-CBC   aescbckey

$ sudo k3s secrets-encrypt status
Encryption Status: Enabled
Current Rotation Stage: reencrypt_finished
Server Encryption Hashes: hash does not match between s1 and s2

Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey-2021-12-15T22:20:54Z

After restarts on all 3 nodes

$ sudo k3s secrets-encrypt status
Encryption Status: Enabled
Current Rotation Stage: reencrypt_finished
Server Encryption Hashes: All hashes match

Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey-2021-12-15T22:20:54Z