- Shipped
- September 4, 2026 at 7:50 PM UTC
- Author
- Kamo
- Commit
- aefbb92
Deploys replaced the only pod of each service with nothing to catch the requests in flight. Three settings, applied across the fleet: - preStop sleeps 10s before the process sees SIGTERM. Kubernetes removes the pod from its EndpointSlice and signals it at the same moment, and Traefik only learns of the removal by watch — so for a moment it keeps dispatching new requests into a pod that has already begun refusing them. That gap is where the 502s on an otherwise clean rollout came from. - terminationGracePeriodSeconds raised above the preStop sleep, so the hook is not itself SIGKILLed, and in-flight work has room to finish. It is a ceiling, not a wait: an idle pod still exits in about a second. - minReadySeconds 15, so a pod that passes readiness once and then falls over cannot retire the healthy pod it replaced after CI has already called the rollout good. topologySpreadConstraints are added ready for a second replica; they are inert at one. Audited from the live cluster: 63 of 65 deployments in `kamo` ran a single replica, 1 of 65 had a preStop hook, and none had minReadySeconds. MediaService shipped with `strategy: Recreate` at one replica with a 5s grace period, so every deploy deleted the pod serving chat, attachments, the chat and notification WebSockets, presence and push, and only then started the replacement — whose own logs put its boot at 59 seconds. That is a ~65s window in which the service did not exist, and an upload on the wire was severed at the 5s mark. A member lost a document upload to exactly this. There is no PersistentVolumeClaim here and never was — the volumes are a ConfigMap, two Secrets and two emptyDirs — so nothing ever required Recreate. It was left over. RollingUpdate with maxUnavailable 0 / maxSurge 1; grace 660s to match kamo-internal, which carries the same 3 GiB attachments; **************** 5s -> 600s, which was the limit actually cutting uploads off; CI's rollout timeout raised to 900s to sit above the grace. replicas stays at 1: MediaService cannot serve two pods correctly yet. Its STOMP broker is in-heap and its per-conversation JetStream consumer is an exclusive durable named after the session GUID, so a second pod would silently receive nothing for conversations the first pod bound. That is fixed separately, before the replica count moves.