All posts

Kubernetes Secrets are not encrypted

They are base64. Everyone knows this and everyone forgets it the moment a Secret shows up in a code review.

kubernetessecurity

Base64 is an encoding. It takes bytes that might not survive a text channel and turns them into bytes that will. It is not a cipher. There is no key. Anyone holding the output holds the input.

Everyone knows this. It still goes wrong constantly, because a Kubernetes Secret looks encrypted:

yaml
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
data:
  password: c3VwZXItc2VjcmV0LXBhc3N3b3Jk

That is super-secret-password, and the only thing standing between a reader and it is a willingness to type:

bash
echo c3VwZXItc2VjcmV0LXBhc3N3b3Jk | base64 -d

The kind is called Secret. The value is unreadable at a glance. Both of those are true, and neither is protection — but together they are enough to make a Secret in a pull request diff feel handled in a way that a plaintext password never would.

What the encoding is actually for

It is there so the value can be arbitrary bytes.

A Secret has to hold TLS private keys, binary tokens, certificates with newlines in them — things that do not round-trip cleanly through YAML. Base64 guarantees the value survives being written down, read back, and shipped over an API that expects text. That is the whole job, and it does it well.

Nothing in the design ever claimed confidentiality. The API server stores Secrets in etcd, and before encryption-at-rest was configured it stored them exactly as you sent them. The Kubernetes documentation says so directly, in a warning box most people meet after the incident rather than before it.

Where it actually bites

The failure is rarely someone decoding a Secret from a running cluster — that requires access to the cluster, at which point there are easier paths. It is that base64 defeats the tools that would otherwise have caught the mistake.

  • Secret scanners miss it. A scanner looking for AKIA[0-9A-Z]{16} does not see QUtJQVFZ.... Most pre-commit hooks and CI scanners match plaintext patterns, so encoding a credential is enough to walk it straight past them.
  • Code review misses it. A reviewer skims data: and moves on. The value is opaque, it is in a file called secret.yaml, and the diff looks like every other Secret they have approved.
  • Git remembers. Once it is committed, rotating the credential is the easy part. Removing it from history means a force-push across every clone, and in practice it stays there.

The common thread is that base64 does not hide a credential from an attacker. It hides it from your own safety net.

What to do instead

Nothing here is novel, which is rather the point:

  1. Do not commit Secrets. Use External Secrets Operator, Sealed Secrets, or your cloud’s secret manager, and let the cluster fetch the value at runtime. The manifest holds a reference, and a reference is safe to review.
  2. Turn on encryption at rest for secrets in the API server config, so etcd holds ciphertext. This is about the storage layer, not the manifest — it does nothing for the copy in your repository.
  3. Scan for the encoded forms too. If a scanner knows AKIA, teach it QUtJQ. It is a small regex change and it closes the gap that made the encoding load-bearing in the first place.
  4. Use RBAC as though Secrets were plaintext, because to anyone with get on them, they are.

I wrote a decoder for this mostly because reading a multi-key Secret one base64 -d at a time is tedious, and tedium is what makes people skip the check. It runs entirely in the browser — pasting a production Secret into a website is its own version of this mistake.