Five reasons the golden image had never actually been built
The build had never been run. Running it found one bug per attempt, and the first three would each have produced a plausible-looking image with something missin...
A way to actually build the golden image
build-golden-image.sh said "Run by CI (see ../../.forgejo/workflows/)". No such workflow was ever written. It needs libguestfs, qemu-img and virt-customize, non...
The three templates that never got committed
service-template.yaml, cloud-init.userdata.yaml and cloud-init.networkdata.yaml. They were written into a second worktree and committed from neither — the earli...
Apply the shared infrastructure on deploy
The namespace, the single-replica StorageClass, the clusterwide seal, computeservice's RBAC and the tenant gateway. Guarded on the hosted-computers/ directory c...
Per-org VM islands, suspend-to-disk, and a white-labeled golden image
A member's own Linux computer, running as a KubeVirt VM in its organization's namespace and opened from the browser. Separate from desktop/ in every way that ma...
Encode the kamo14 reel
The marketing reel was recut (new closing logo shot) and uploaded as public/kamo14.mp4. It gets a new name instead of replacing kamo13 in place because the ladd...
A reply to a sending domain is refused as the recipient, in words that fit NoReply
Campaigns now send Reply-To: NoReply@<their From's domain> by default, and those replies reach postfix-k3m1-inbound. The refusal said "Sender address rejected: ...
universe backup: pull mc from quay.io, docker.io/minio/mc is gone [skip ci]
MinIO withdrew its Docker Hub images, so the backup's upload container sat in ImagePullBackOff from 2026-09-12. With concurrencyPolicy Forbid that one stuck job...
node-config: registry prune must not garbage-collect with --delete-untagged [skip ci]
The nightly prune on k1m1 lived only on the host. Its garbage-collect ran with --delete-untagged, which deletes the per-platform manifests an OCI image index re...
auto-cert: stop provisioning www.stack.loans
The ratestack estate has been scaled to 0 since 2026-08-04, so nothing answers HTTP-01 for stack.loans. Its Certificates expired on 2026-08-02 and their orders ...
Disable node-exporter's xfs collector [skip ci]
It cannot parse this kernel's /proc/fs/xfs/stat ("xpc") and logged an error on every scrape, ~240/h per node. Filesystem usage is unaffected (filesystem collect...
Kamo_app owns the ocr_* tables [skip ci]
OCRService (web and worker) alters its own ocr_* tables at startup and refuses to start without ownership; after the switch to kamo_app the worker crash-looped ...
Non-superuser kamo_app role for all application workloads [skip ci]
Services connected as the superuser kamo. Add an idempotent script that creates kamo_app (no superuser, not a member of kamo so it cannot SET ROLE back) with gr...
Bounces for organizations' sending domains reach EmailService through postfix-k3m1-inbound
A campaign sent through the shared relay has its From at the organization's sending domain as its envelope sender, and that domain is a CNAME to contact.kamocrm...
Scrape YugabyteDB tserver/YSQL and SecurityService pool metrics [skip ci]
Connection sizing had no data behind it: nothing scraped the database or the Hikari pools. Add a curated server-level tserver job (the full endpoint is ~3.25M l...
Postmaster@ and abuse@contact.kamocrm.com reach KamoCRM — the shared relay's name receives its checks
contact.kamocrm.com is the shared org relay's name (47.181.8.86's PTR, MX contact.kamocrm.com), and its MX is 47.181.8.86 → k3m1, where postfix-k3m1-inbound ans...
Postfix-orgs, the shared bulk relay every organization sends through as 47.181.8.86
KamoCRM's own campaign mail keeps leaving through postfix-bulk (47.181.8.87) and regular mail through postfix (47.181.8.84). Every other organization's campaign...
node-config(k3m1-ext): eno1 on DHCP as k3m1-ext, used only by traffic from its own address
k3m1's third cable is the port the shared bulk relay (postfix-orgs, 47.181.8.86) sends through. DHCP gives it an address and nothing else, so the main table is ...
Prune unused images on k3m1 too, and stop bulk removals timing out
k3m1 had no prune job, and kubelet image GC waits for 85% disk, so every deploy's SHA-tagged image piled up: 3,368 images (539G), 86 in use. Add a daily k3m1-im...
Sign Feedback-ID in DKIM signatures
Gmail counts a campaign message toward its Feedback Loop only when the Feedback-ID header is covered by the sending domain's DKIM signature. local.d/dkim.conf k...
Receive postmaster@/abuse@stack.loans on k3m1 for the bulk IP's checks
47.181.8.87's reverse DNS is mail.stack.loans and the router forwards its port 25 to k3m1, but nothing there accepted mail, so a provider verifying who runs the...
Route fbl@/abuse@hello.kamocrm.com complaint reports to EmailService
Spam-complaint feedback-loop reports (ARF, RFC 5965) and abuse reports sent to fbl@ and abuse@hello.kamocrm.com are rewritten to complaints-mail.invalid and han...
Mail to unsubscribe@hello.kamocrm.com reaches EmailService
Every bulk and shared-article message offers an unsubscribe by email (List-Unsubscribe mailto), but mail to that address went to postmaster and was never acted ...
Postfix-bulk accepts any sender from EmailService [skip ci]
The relay is now a platform outbound host that every organization's campaign mail may go through, as whatever From that organization sends as. Only the null sen...
This relay only sends as kamocrm.com
Anything in mynetworks (every pod, and the LAN) could relay through 47.181.8.84 as any domain, which receivers treat as spoofing and charge to this IP; that is ...
The delivery-events sidecar drops a batch EmailService rejects as invalid
A 400/413/422 means the batch itself will never be accepted; retrying it forever would hold back every result queued behind it. It is logged and dropped. 401/40...
Deploy postfix-bulk with the rest of the mail server [skip ci]
The bulk relay's four manifests were applied by hand because the mail step applies its files by name. It now applies them too, refuses to continue if the sideca...
Sign hello.kamocrm.com with its own customer key, by the From domain itself
Campaign mail leaves as hello.kamocrm.com through postfix-bulk. It gets its own DKIM key (selector customer, published as customer._domainkey.hello), and use_es...
Postfix-bulk, a second outbound relay for campaign mail, with delivery results fed back
Campaign mail now leaves through its own relay on k3m1 (egress 47.181.8.87) as hello.kamocrm.com, so complaints about bulk mail never touch the reputation of ma...
Keep hello.kamocrm.com off this relay, and stop three errors on every start
- A @hello.kamocrm.com sender is refused relay here with a 4xx, so bulk mail can only leave through postfix-bulk; delivery to our own domains is unaffected....
Strip identity headers on myloan.*/api/ before the portal BFF relays them
KamoMLOS's BFF routes under /api/ (the /api/proxy/* passthrough, /api/session/extend, /api/chat/routing, /api/chat/subject-state) copy every client header excep...
Dial 911 and 933 from every extension, with an MLTS email notice
Extensions could not reach 911: no outbound route matched 911 or 933, so from-internal fell through to bad-number and played "cannot be completed as dialed" wit...
Delete identity headers at the edge before header-trusting backends
Add a strip-identity-headers Middleware. A customRequestHeaders value of "" deletes the header, so X-Org-Id, X-Member-Id, X-User-Id, X-God-Mode-Active, X-Member...
Report each account's coder allowance to the console
The Linux Desktop menu is getting two meters — how much of the member's numbers. transcripts do not carry them, and neither OTel metric is about the subscripti...
Four workers on a node that can hold them, not two on one that cannot
Correcting the previous commit, which was half right and caused a short outage. The memory diagnosis was correct: four workers, each loading its own ~3GB copy ...
Give the translation models room to stay resident, and refuse a scanner
libretranslate ran 4 gunicorn workers against a 12Gi limit, and each worker loads its own ~3GB copy of the argos model set — 11.8GB resident, 98% of the limit, ...
The terminal had no scrollback to scroll, so turn tmux's mouse on
for anything else. I built it against a bare xterm.js writing its own output and never against the real pipeline, which is the whole of the mistake. tmux is a ...
Narrow the window a DDL leaves behind for stale backends
Applying DDL bumps the catalog version, and backends still holding the old one fail their next statement with SQLState 40001 until they are recycled. The last o...
Say whether a terminal is working or waiting for its member
Every terminal in the listing now carries `state`: busy, or waiting. It is the one question somebody asks of a window they cannot see, and it is what a minimize...
Let a coder session run at the model and effort the member picked
for a new session and for a resumed one alike, since `--model` and `--effort` are per-session flags either way. They are the ONLY part of the command a caller ...
Send To AI must not open a tab in sage's editor
A Send-to-AI hand-off is hosted in a Remote Terminal the operator is already watching in the console. On top of that, the agent also handed the session to VS Co...
Restarting the agent must not kill every member's shells
mid-work today, and would have taken every other member's terminals with it. systemd's default is KillMode=control-group: on stop or restart it SIGKILLs every ...
A coder terminal wears the name coder gave the conversation
running had long since been titled "Startup fee implementation and Harmony account adjustment". Two separate things were swallowing that name. **The listing.**...
A budget must not be able to deadlock a node drain
Switches every PodDisruptionBudget from minAvailable: 1 to maxUnavailable: 1. On a two-replica Deployment the two are identical — one pod evictable at a time. ...
Budget the last four services to gain a second replica
BillingService, EmailService, KamoLOS and SecurityService now run two pods, each having put its shared-effect scheduled sweeps behind a distributed lease first....
Budget the six services that just gained a second replica
AIService, KBService, ESigService, VOIPService, DocsService and TranslateService now run two pods, so a drain of either node must evict them one at a time rathe...
Measure whether a deploy actually costs anyone a request
Every fix in this programme is a claim about what happens during a rollout. Until now those claims were untested: MediaService shipped with strategy Recreate at...
coder sessions on the dev machine, listed and resumable
is here, because the machine is the only thing that knows any of it. ## The session index home, exactly as their terminals are whatever tmux is running. No ta...
Budget every drain, and fail a deploy that regresses rollout safety
PodDisruptionBudgets for the 25 Deployments that now genuinely run two replicas. minAvailable: 1 lets a drain evict one pod and wait for its replacement instead...
Run the deploy-safety audit on every deploy
Ungated on purpose. Every other step here is scoped to its own changed directory, but this one checks state that arrives from 48 other repositories, so gating i...
Like what you see shipping?
All of it arrives in your workspace on its own. Start on the free plan and read this page again in a month.