Rotating a database password without a restart: how our Redis and Valkey Kubernetes operator works

Our free plan has no disk, so Valkey runs with appendonly no and save "", so when the pod restarts, the customer's data is gone. That one fact shaped most of our Kubernetes operator. If rotating a password restarts the pod, a routine security task wipes someone's cache. So, as expected, we split configuration into two classes: changes that need a restart, and changes that must never cause one. This post walks through how the operator works and why it's built this way, including the parts that aren't finished. Disclosure: this is the operator behind KVNest, the managed Redis and Valkey service we run. Versions and numbers are as of November 2026. TL;DR One cluster-scoped custom resource becomes up to nine child objects, nearly all in a dedicated namespace. Children are created in dependency order, so a pod never exists without its quota and NetworkPolicy. Server settings are hashed into a pod annotation and trigger a rollout. ACL users are left out of that hash and are applied live with ACL SETUSER. Each instance has a private internal user for probes, metrics and the operator, so customer password rotation can't break them. It's deliberately single-node. No failover, no backups yet, no SLA. More on that at the end. The API is one object When we started, the existing Valkey operators didn't fit a paid, externally exposed product, so we wrote a thin one with kubebuilder. The controller package is under 1,400 lines of Go 💙, not counting tests. Our web API writes exactly one object per instance. The Kubernetes API is the database for instances, so there's no second copy to drift out of sync: apiVersion: cache.kvnest.com/v1alpha1 kind: ValkeyInstance metadata: name: inst-demo1 labels: kvnest.com/owner: user-123 spec: plan: starter engine: valkey # or redis version: "9.1" # 8.10 for redis, etc... allowedSources: - 203.0.113.0/24 users: - name: analytics access: readonly keys: ["events:*"] It's cluster-scoped for a reason. Kubernetes doesn't allow a namespaced owner to own cluster-scoped objects or objects in another namespace. A cluster-scoped CR can do both, so this one object can own its Namespace and a Secret in a separate credentials namespace. Validation lives in the CRD. There's no webhook to keep available, because CEL rules and schema patterns run in the API server: plan is immutable engine is immutable spec.users[0].name: Invalid value: "kvnest-x": name is reserved spec.allowedSources[0]: Invalid value: "nope": must be a CIDR such as 203.0.113.0/24 The engine is immutable because Redis and Valkey data files aren't interchangeable. Key patterns must match ^[^\s"'\\]{1,64}$, which stops a pattern from smuggling extra tokens into a generated user line. One CR, nine children The order is deliberate. The quota and NetworkPolicy exist before anything can be scheduled, and the public route comes last. The allowlist middleware is created before the route references it, and it's deleted only after the route stops referencing it. Deletion is one line of logic. If the CR has a deletion timestamp, Reconcile returns and garbage collection does the rest. There's no finalizer, because nothing lives outside the cluster. The StatefulSet's PVC retention policy is WhenDeleted: Delete, so the volume goes too. Traefik's CRDs are handled as unstructured objects, so the operator doesn't depend on Traefik's Go module. The informer cache is label-filtered. The operator only caches objects carrying its managed-by label, so it never holds every Secret and Service in the cluster in memory. The reconcile loop A few details here took more thought than the diagram suggests. A ready replica doesn't mean the rollout is done. During a rolling restart, the old pod still counts as ready while it shuts down. If we trusted readyReplicas alone, we'd report Ready too early and send ACL commands to a pod on its way out. So there's a second check: func rolledOut(sts *appsv1.StatefulSet) bool { st := sts.Status if st.ObservedGeneration != sts.Generation { return false } if st.UpdateRevision != "" && st.CurrentRevision != "" && st.UpdateRevision != st.CurrentRevision { return false } replicas := int32(1) if sts.Spec.Replicas != nil { replicas = *sts.Spec.Replicas } return st.UpdatedReplicas == replicas } Failed isn't a terminal state. An instance is marked Failed after 10 minutes without becoming Ready, but the loop keeps running and it recovers on its own if the pod comes up. The timer counts from when the Ready condition last turned False, so a restart of a six-month-old instance doesn't read as six months late. StatefulSet updates touch only what can drift. Most fields are immutable or defaulted by the API server, so overwriting the whole spec would fight it. After creation, we update replicas, the config hash annotation, scheduling fields, and each container's image, resources, env and readiness probe. Missing containers get added whole. That's how the metrics sidecar reached instances created before it existed. It costs one restart per instance, which on a diskless plan means losing the cache, so that kind of change gets weighed carefully. Two classes of configuration This is the core of the design. valkey.conf is one file with two parts, but only one part counts. Condensed from the controller: conf := ValkeyConf(plan, vi.Spec) // The Secret holds both parts... config.Data = map[string][]byte{ "valkey.conf": []byte(conf + ACLConf(users)), internalPasswordKey: []byte(internal), } // ...but users are left out of the hash: they change live, without a restart. configHash := hash(conf) tpl.Annotations[configHashAnno] = configHash Change the eviction policy or turn on modules, and the hash changes, the pod template changes, and the StatefulSet rolls. Add a user or rotate a password, and the hash stays the same, so no pod restarts. The operator applies the change over the network instead. Users still go in the file because a new pod has to start with the right ones. Here's a real rendered line for a read-only user during a rotation, with the hashes shortened: user analytics on resetpass #97b056…f88191 #6da063…f9de0d resetkeys ~events:* resetchannels &* nocommands -@all +@read +@connection +info -@admin Two things make this work: Every user definition starts from a clean slate. resetpass, resetkeys, resetchannels and nocommands come first, so applying the rules again fully replaces the user. The same rules work in the file and in ACL SETUSER, and applying them twice is harmless. Passwords are stored as SHA-256 hashes. Plaintext never reaches valkey.conf. The sync itself is short. It opens a connection with a 5-second budget, runs ACL SETUSER for each expected user, lists what the server has with ACL USERS, and deletes anything extra. Redefining a user keeps their open connections, while deleting one closes them. The 10-minute resync heals a real race. Secret volumes update eventually, so a pod that restarts right after a user change can load an older file. The next reconcile corrects it. A failed sync doesn't take the instance down. The server is still serving traffic, so it stays Ready. What doesn't advance is status.observedAccessGeneration. The API compares that to metadata.generation and tells the customer their access change is still pending. The readiness of the database and the convergence of its config are separate signals, and we report them separately. Password rotation is declarative Rotation is a spec change, like everything else: spec: passwordRotation: generation: 3 previousValidUntil: "2026-11-06T12:00:00Z" Raising generation above the value stored in the credentials Secret issues a new password. Because that comparison uses stored state, a retried reconcile can't generate a second one. During the grace period, the server accepts both passwords. That's what the two hashes in the line above are. The API caps the grace period at 7 days. Revocation needs no cron job. The requeue is set for just after the earliest expiry: requeue := aclResync // 10 minutes if !ch.nextExpiry.IsZero() { requeue = min(requeue, time.Until(ch.nextExpiry)+time.Second) } There's one behavior worth knowing as an operator: revoking a password stops new authentications, but it doesn't close connections that are already authenticated. Those last until the client disconnects or hits the 300-second idle timeout. Why there's a separate internal user The readiness probe, the exporter and the operator all authenticate as kvnest, an ACL user whose password never leaves the instance namespace and never rotates. Without it, rotating a customer password could make the pod unready. Protecting that user meant carving subcommands out of the customer's +@all: +@all -acl|setuser -acl|deluser -acl|load -acl|save -acl|list -acl|getuser -acl|dryrun -client|kill -client|pause -client|unpause CLIENT KILL is the interesting one. ACL rules can't scope it to the caller's own connections, so allowing it would let a customer drop the probe's connection and make their own pod look unhealthy. No common client library sends it, so removing it costs little. CLIENT PAUSE goes for the same reason. CONFIG, DEBUG, MODULE, SHUTDOWN, REPLICAOF and the replication commands are disabled for everyone with rename-command. That includes the internal user, so the exporter runs with --config-command=-, and modules are loaded from the config file at startup. Moving to this design did have a cost. Existing instances restarted once when the probes switched users, and Starter instances lost their cached data at that point. That's the same trade-off as the sidecar, and it's why we think hard before changing a pod template. Isolation and the network path Routing. Clients connect to inst-.eu.kvnest.com:6380 over TLS. Traefik terminates TLS with a wildcard certificate and routes by SNI to the instance's ClusterIP Service. The load balancer sends PROXY protocol headers, so Traefik's ipAllowList sees the client's real address. NetworkPolicy. There are exactly three ingress rules: the Traefik namespace on 6379, the Prometheus namespace on 9121, and the operator's pod on 6379, matched by both namespace and pod labels. The policy declares Egress with no rules, so all outbound traffic is denied. Valkey doesn't need to call anyone. ResourceQuota. One pod, one PVC if the plan has one, no LoadBalancer or NodePort Services, and CPU and memory equal to the plan plus the sidecar's 50m and 32Mi. The sidecar's share is added on top, so metrics never come out of the memory a customer pays for. Pod spec. The namespace enforces the restricted Pod Security Standard. Requests equal limits on both containers for Guaranteed QoS. The pod runs as uid 999 with a read-only root filesystem, all capabilities dropped, the RuntimeDefault seccomp profile, no ServiceAccount token, and enableServiceLinks: false. Sizing around failure modes Plan Memory limit maxmemory CPU Storage maxclients Starter 256Mi 192 MB 100m emptyDir, no persistence 100 Casual 1Gi 768 MB 250m 10Gi PVC, AOF 500 Pro 4Gi 3000 MB 1000m 20Gi PVC, AOF 2000 maxmemory sits at about 70% of the container limit. An AOF rewrite forks the process, and copy-on-write can touch a lot of pages under write load. Set maxmemory close to the limit, and the rewrite is what gets you OOM-killed. A unit test fails if any plan goes above 80%, or if its storage is under 5x maxmemory. Readiness and liveness ask different questions. Readiness runs PING through the CLI and requires PONG. Liveness is only a TCP check. While a large AOF loads, the server replies LOADING, so the pod is correctly unready. If liveness used the same check, Kubernetes would kill the pod during a long load, and the next attempt would start over and die the same way. The probe authenticates through REDISCLI_AUTH, so the internal password never appears in the process arguments. Measuring time-to-ready without lying The operator exposes kvnest_instance_time_to_ready_seconds{engine, plan}, a histogram from CR creation to the first Ready condition. The hard part is observing each instance only once across operator restarts and later pod restarts, since metrics live in memory and reconciles repeat. So the state lives on the object, in an annotation: The first reconcile sets kvnest.com/time-to-ready: pending. Once the stored status says Ready, the operator replaces it with the measured duration, and then records the sample. Instances that predate the metric never get pending, so they're never observed with a made-up value. The ordering is intentional. Annotating first means a failed write retries without a duplicate sample, and the worst case is losing one sample. For a number we'd quote publicly, undercounting beats double-counting. The duration comes from the persisted lastTransitionTime, and a merge patch keeps the annotation from bumping generation. # p95 (use 0.5 for p50) over the last 30 days histogram_quantile(0.95, sum by (le) (rate(kvnest_instance_time_to_ready_seconds_bucket[30d]))) Per-instance metrics come from the redis_exporter sidecar through one PodMonitor that spans all namespaces and relabels each series with the instance name. The exporter only reads server stats, never keys or values. How we test it Pure builders get unit tests. The functions that render config, quotas, policies and routes take a plan and a spec and return data, so tests can assert invariants like "the quota fits the pod." The reconciler runs against envtest, a real API server and etcd. Eighteen specs cover things like restoring a deleted child, moving old pods to new settings, and rejecting a plan change. Envtest has no kubelet, so pods never run. Those specs mark readiness by hand and use a fake ACL syncer, which leaves a gap. A separate test runs the real syncer against a live server, including checks that the default user gets NOPERM on ACL SETUSER and that read-only users can't write. It only runs when you point it at a server, so it isn't part of the default CI run. What it doesn't do If you're evaluating this as an SRE, here's what we'd want to know: Single node. Each instance is one replica with no failover. A node failure or pod restart is an outage for that instance, and on Starter it's also data loss. No backups during the preview, and no SLA yet. Deleting the CR deletes the data, volume included. Nothing is retained. No plan resizing. spec.plan is immutable for now. TLS ends at Traefik. Traffic from Traefik to the pod is plain TCP inside the cluster. Isolation is at the namespace and container level. There's no sandboxed runtime, and tenants don't get dedicated nodes. The operator supports a tenant node pool, but it isn't enabled yet. The operator is part of security enforcement. If it's down, running instances keep serving, but a password in its grace period won't be revoked until the operator is back. The server only drops it when told to. Those are real limits, and some workloads should go elsewhere because of them. They also explain why the operator is small: most of the complexity in a database operator comes from replication and failover, and we don't have those yet. What we'd reuse elsewhere If you're building an operator for something stateful, these are the ideas we'd take with us: Decide which changes are allowed to restart the workload, then enforce it in what you hash. Don't equate a ready replica with a finished rollout. Report "serving" and "converged" as separate signals. Give your own tooling its own identity, so customer actions can't break your probes. Keep exactly-once state on the object, not in the process. You can try it on the free Starter plan, with no card required. Connect with valkey-cli --tls and see for yourself. How do you handle credential rotation for stateful workloads on Kubernetes? We'd especially like to hear from anyone who has run a Redis or Valkey operator with replication in production, since that's the part we haven't built yet.