I was working recently on a small Kubernetes environment, via the GitOps way. I am a big fan of GitOps, everything stored as code in your git and you can only make changes to your setup by updating/adding code instead of changing things manually in the cluster. Even on my CV i have the phrase: If you sh*t isn’t in git, it doest exist. So yes, huge fan. I can tell you all kinds of reason for applying this, but it might be preaching to the choir.

But working on this small Kubernetes environment I stumbled on something that I didn’t want to put into git. In this case it was the generated secret from my CA certificate. In this Kubernetes cluster I have the cert-manager running for automatically generating and signing internal certificates. But I have several tools/apps running in multiple namespaces, that needs this ca certificate (well, its result, the Secret).

So what choices do I have to make this happen:

  • Exporting the secret, have some magic oneliner and with some “kubectl apply” statements to create the secret in all the namespaces I need it. Which is 1, work manually I don’t like to do at all and 2, I have to do this again when the ca certificate is renewed. But I also need document it somewhere and try to not forget it.
  • Again exporting the secret but instead of doing some magic online, I will store and use Sealed Secrets (that I am already using in my setup) in Git. In the many places in the GIT repository I will have to store the encrypted file. But that again means that when the ca certificate is renewed, I have to do this process again and apply the change on all the files in my git repository.

You see, they are both not great. The first one is not GitOps at all, so even thinking about it for a second is actually already too much time spent. The 2nd is more in the GitOps alley, but it still too much work. You still have to monitor when the certificate is renewed, do an export, encrypt it with Sealed Secrets (or store it somewhere else) and apply it. And yes, we do have monitoring of these certificates so that isn’t an issue, but the work to export, encrypt it is to much. There should be a 3rd way right?

I found a tool called reflector, with the code available on https://github.com/emberstack/kubernetes-reflector “Reflector is a Kubernetes addon designed to monitor changes to resources (secrets and configmaps) and reflect changes to mirror resources in the same or other namespaces.”

This is exactly what I need! So let us implement it straight away.

Installation is done, via HELM and is very easy.

$ helm install reflector oci://ghcr.io/emberstack/helm-charts/reflector \
    --create-namespace \
    --namespace support

Yes, I installed it in the support namespace, because I don’t want this in the kube-system or default namespace. And I have other tools installed and running in the support namespace, so that would make sense to add it here.

Personally, I don’t want to create for every tool a dedicated namespace and prefer to combine tools where possible. I know that namespaces doesn’t costs much at all, but it personally feels a mess and unstructured when you create for every tool a dedicated namespace. I know and understand this from a security p.o.v. though, so don’t starting complaining now ;-). Anyways!

Now I can configure the reflector tool with some annotations in my resource manifest for the CA certificate. I have the following right now:

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: our-own-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: "Our Own CA"
  secretName: our-own-ca
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 4096
    rotationPolicy: Always
  issuerRef:
    name: my-ca
    kind: ClusterIssuer
    group: cert-manager.io
  secretTemplate:
    annotations:
      reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
      reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"

With the annotations in the secretTemplate key, I make sure that reflector will make sure that a secrets needs to be synchronised to all namespaces. So in any namespace I now have the our-own-ca secret!

Once applied we can see that the secret is now available in all namespaces.

$ kubectl get secret -A | grep our-own-ca
support.             our-own-ca                                      kubernetes.io/tls               3      42s
cilium-secrets       our-own-ca                                      kubernetes.io/tls               3      42s
default              our-own-ca                                      kubernetes.io/tls               3      42s
elastic-system       our-own-ca                                      kubernetes.io/tls               3      42s
kube-node-lease      our-own-ca                                      kubernetes.io/tls               3      42s
kube-public          our-own-ca                                      kubernetes.io/tls               3      42s
kube-system          our-own-ca                                      kubernetes.io/tls               3      42s
logging              our-own-ca                                      kubernetes.io/tls               3      41s
monitoring           our-own-ca                                      kubernetes.io/tls               3      42s

If needed, I can even limit the replication to a certain set of namespaces, with adding the following notation to the certificate.

reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "monitoring,logging,support"

When I add the last line in my our-own-ca certificate resource, the secret will only be replicated to these namespaces.

Even though these replicated secrets are not in git, the configuration to get them are. So still GitOps right? Maybe not entirely, but it does help me solving the issue that I don’t want to spend any time doing manual actions. When the CA is renewed, it regenerates the secret. And reflector automatically syncing it again to all (or subset) namespaces.

What do you think? Would this work for you too or did I miss a valid 3rd option to use in my setup? Do let me know!