- Shipped
- September 5, 2026 at 3:18 PM UTC
- Author
- Kamo
- Commit
- 4d5ccc4
Correcting the previous commit, which was half right and caused a short outage. The memory diagnosis was correct: four workers, each loading its own ~3GB copy of the argos model set, do not fit a 12Gi limit, and the kernel reclaiming mmapped model pages is what made every cold language pair take seconds. Dropping to two workers was the wrong remedy. /languages is served by these same gunicorn workers, so with both of them inside a long translation the readiness probe cannot be answered — and on 2026-09-05 both replicas left the Service endpoints at once and translation stopped entirely. The health endpoint needs a worker free to answer it. Four workers with the memory to actually hold their models satisfies both constraints. That memory does not exist on k1m1, which carries the control plane, the database and the CI builders at ~95% of its memory requests — asking for it there is what made the second replica fail to schedule mid-rollout. So the pod moves to k3m1, at ~63% and already holding the same model set at this hostPath from an earlier period. Both replicas were always pinned to a single node; this changes which node, not the redundancy story. Verified after rollout: both replicas Ready on k3m1, both in the Service endpoints, en->ru returning correct output.