Improve documentation about securing etcd clusters

[CaptainIRS]

How to categorize this issue?

/area documentation
/kind enhancement

What would you like to be added:

• Details on which keys are expected to be present in the Secrets linked using *SecretRef - ca.crt/bundle.crt, tls.crt, tls.key should be documented.
• The dataKey field in tlsCASecretRef should be documented.
• We seem to be using different defaults for dataKey of the backup-restore CA (bundle.crt) and of client/peer CA (ca.crt), so we should make it consistent in the codebase or document the defaults.
• The independence of the set of certificates for client, peer and backup-restore TLS should be documented as the dev scripts in etcd-wrapper tend to reuse the same certificates for different purposes (client and backup-restore TLS certificates in this case), leading to confusion.
• etcd supports both HTTPS and HTTPS client-cert auth (mTLS) for securing the communication between clients and the etcd server. But, etcd-druid enables client-cert auth when client TLS is enabled and expects a client certificate to be provided for other components like etcd-wrapper and etcd-backup-restore to talk to the etcd server. This limitation should be documented as there may be use-cases where just HTTPS would need to be supported (for instance when an official CA would need to be used to generate the certificates as opposed to a self-signed CA - in which case it would not be feasible to obtain a client certificate for every client that needs to talk to the etcd cluster).
• We should document the minimal set of SANs for the certificates for client, peer and backup-restore TLS. The reference script in etcd-wrapper and the way gardener/gardener setup TLS certificates is with an one-size-fits-all approach for setting SANs (essentially an exhaustive set of SANs are set on the certificate and is reused across client, peer and backup-restore TLS), which might not be a good idea from a security standpoint.
• There is no validation to check if the *SecretRefs actually point to valid certificates, leading to cryptic errors which are hard to trace by checking the Etcd CR's Status. This behaviour could be documented or enhanced later.
• The document doesn't have guidance on how to perform certificate rotation. This leads to some misconceptions as etcd has peculiar behaviour when it comes to updating certificates - i.e the server/client secrets can be updated dynamically (picked up by the etcd process dynamically) whereas updating the CA cert secret requires a rollout (ref: ETCD doesn't automatically load changes to ca bundles for peer-trusted-ca-file or trusted-ca-file etcd-io/etcd#11555)
• The fact that we allow for a transition from non-TLS to TLS enabled cluster and handle it should be documented. Currently, the lack of information leads users to believe that we have to take down the non-TLS cluster and bring up a new cluster with TLS enabled.
• We should document the fact that even if enableEtcdSpecAutoReconcile is set to true, this doesn't trigger reconciliation if the contents of the secret is updated (assume the user creates a mutable secret, unlike how it is done in gardener/gardener with an immutable secret, in which case the reference remains the same whereas the content could be updated).

Why is this needed:

To guide users who wish to secure their etcd clusters by following our documentation.