Stop shipping the shared credential in this repo
The mail MySQL password sat in plaintext across fourteen files here — in ConfigMaps that kubectl will hand to anyone who can read them, in two Jobs that CI appl...
Enforce storage quota on inbound delivery, and stop shipping the DB password
Object storage already refuses content that would take an organization past its ceiling, which covers every domain going through the object store. Inbound mail ...
Enforce storage quota on inbound delivery, and stop shipping the DB password
Object storage already refuses content that would take an organization past its ceiling, which covers every domain going through the object store. Inbound mail ...
The Dovecot quota change, written out and not applied
Object storage now refuses content that would take an organization past its ceiling, which covers every domain that goes through the object store. Inbound mail ...
The Dovecot quota change, written out and not applied
Object storage now refuses content that would take an organization past its ceiling, which covers every domain that goes through the object store. Inbound mail ...
Give YugabyteDB real resources and pin its memory
It was running on LESS than the CockroachDB it replaced -- 20Gi/6cpu against CockroachDB's 32Gi/8cpu -- while needing more, because YSQL runs a separate Postgre...
Give YugabyteDB real resources and pin its memory
It was running on LESS than the CockroachDB it replaced -- 20Gi/6cpu against CockroachDB's 32Gi/8cpu -- while needing more, because YSQL runs a separate Postgre...
Allow YugabyteDB port 5433 in kamowssecurity Cilium egress
A CiliumNetworkPolicy selecting app=kamowssecurity allowed host/remote-node egress on 26257, 4222 and 9000 only. After the cutover the database moved to 5433 an...
Allow YugabyteDB port 5433 in kamowssecurity Cilium egress
A CiliumNetworkPolicy selecting app=kamowssecurity allowed host/remote-node egress on 26257, 4222 and 9000 only. After the cutover the database moved to 5433 an...
Load with explicit column lists -- positional COPY corrupts silently
COPY t FROM STDIN with no column list maps CSV fields to the table's PHYSICAL column order. The CSVs come from COPY (SELECT *) on CockroachDB, and Hibernate cre...
Load with explicit column lists -- positional COPY corrupts silently
COPY t FROM STDIN with no column list maps CSV fields to the table's PHYSICAL column order. The CSVs come from COPY (SELECT *) on CockroachDB, and Hibernate cre...
Schema parity with CockroachDB -- 714 tables, 8706 columns
The drift diff earned its place. KamoInitializerService builds the schema from JPA entities and produced 599 tables; production carries 714. The 115 difference ...
Schema parity with CockroachDB -- 714 tables, 8706 columns
The drift diff earned its place. KamoInitializerService builds the schema from JPA entities and produced 599 tables; production carries 714. The 115 difference ...
Full CockroachDB->YSQL translator for the universe migrations
All 15 migrations now apply cleanly to YugabyteDB: 111 tables, matching CockroachDB exactly. Eight distinct CockroachDB-only constructs had to be handled, found...
Full CockroachDB->YSQL translator for the universe migrations
All 15 migrations now apply cleanly to YugabyteDB: 111 tables, matching CockroachDB exactly. Eight distinct CockroachDB-only constructs had to be handled, found...
Correct the rationale on pre-created schemas
KamoInitializerService does set **************** true, so it creates these schemas itself. The previous comment claimed it did not. The file stays as explicit b...
Correct the rationale on pre-created schemas
KamoInitializerService does set **************** true, so it creates these schemas itself. The previous comment claimed it did not. The file stays as explicit b...
Unique_rowid() shim and non-public schemas in YugabyteDB
The shim is a sequence seeded at the production maximum plus a margin. The seed carries two requirements, not one: it must exceed every existing id or the first...
Unique_rowid() shim and non-public schemas in YugabyteDB
The shim is a sequence seeded at the production maximum plus a margin. The seed carries two requirements, not one: it must exceed every existing id or the first...
Deploy single-node YugabyteDB 2026.1.0.1 alongside CockroachDB
Both engines now run side by side on k1m1; CockroachDB is untouched and remains the rollback path. Three things that were not obvious: - The Helm chart versio...
Deploy single-node YugabyteDB 2026.1.0.1 alongside CockroachDB
Both engines now run side by side on k1m1; CockroachDB is untouched and remains the rollback path. Three things that were not obvious: - The Helm chart versio...
CSV export via COPY TO STDOUT, plus a CSV-aware verifier
Transport is COPY ... TO STDOUT WITH (FORMAT CSV), not `cockroach sql --format=csv`. The latter renders SQL NULL as the literal string "NULL", indistinguishable...
CSV export via COPY TO STDOUT, plus a CSV-aware verifier
Transport is COPY ... TO STDOUT WITH (FORMAT CSV), not `cockroach sql --format=csv`. The latter renders SQL NULL as the literal string "NULL", indistinguishable...
Make every sweep schema-aware; kamo has 714 tables, not 680
Cross-checking SHOW BACKUP (714 objects) against a table_schema='public' inventory (680) exposed 34 tables living in five other schemas: email_metadata, email_p...
Make every sweep schema-aware; kamo has 714 tables, not 680
Cross-checking SHOW BACKUP (714 objects) against a table_schema='public' inventory (680) exposed 34 tables living in five other schemas: email_metadata, email_p...
Baseline capture for CockroachDB to YugabyteDB parity checking
Two passes: cheap batched count(*), then an order-independent checksum. The checksum sums per-row md5 hashes as integers rather than the obvious md5(string_agg...
Baseline capture for CockroachDB to YugabyteDB parity checking
Two passes: cheap batched count(*), then an order-independent checksum. The checksum sums per-row md5 hashes as integers rather than the obvious md5(string_agg...
Implementation plan for the CockroachDB to YugabyteDB migration
15 tasks. The two gates sit before any downtime: Task 5 proves the backups actually restore, and Task 10 diffs the KamoInitializer-built schema against the live...
Implementation plan for the CockroachDB to YugabyteDB migration
15 tasks. The two gates sit before any downtime: Task 5 proves the backups actually restore, and Task 10 diffs the KamoInitializer-built schema against the live...
Design for the CockroachDB to YugabyteDB migration
Records the measured current state (3.4 GB, 791 tables, 671 FKs, ~30 consumers), the five design decisions, and the cutover sequence. The two findings that sha...
Stop the 60s entrypoint read timeout capping uploads
respondingTimeouts.readTimeout is the deadline for reading a WHOLE request, body included, and Traefik v3 defaults it to 60s where v2 had no limit. That silentl...
Pin the remaining hostPath workloads to the node holding their data
Same fault as embedding-model, found by auditing every hostPath volume in the repo. A hostPath is node-local and nothing replicates it, so an unpinned pod can b...
Pin to the node its model weights are actually on
The model is a hostPath under /var/lib/embedding-model on k1m1, and nothing replicates it. When k3m1 joined, the new ReplicaSet's pod was scheduled there and ne...
Stop the passdb from splitting a mailbox across two maildirs
password_query returned `email as user`, which overwrote the %Lu-lowercased login name with the row's stored casing. user_query then interpolated %n from the re...
STUN answered on the wrong node, so kamocrm.com:3478 was dead
coturn is hostNetwork with no nodeSelector, so it was scheduled on k3m1 (public 47.181.8.87). Its own --external-ip, and every client's `stun:kamocrm.com:3478`,...
Bridge advertised only private ICE candidates — no meet had media for 35 days
The bridge never completed a single ICE connection in the 35 days this pod has been up (0 successes, 18 terminations). Every conference joined over XMPP fine an...
Liveness probe could never pass — cause of 920 restarts in 35 days
Root cause: the liveness probe was `tcpSocket: port 10000`, but Scylla's REST API binds to api_address 127.0.0.1 (its default) while a tcpSocket probe dials the...
Manage ScyllaDB through deploy-services instead of by hand
scylladb/ held complete manifests but had no deploy step — the comment said it was "deployed MANUALLY after the 16TB drive swap" because the data dir /mnt/backu...
Resync drifted manifests with live before enabling full reconciles
Auditing what a full reconcile would apply turned up two manifests that had drifted far enough from the cluster that re-applying them would have caused an outag...
Rename cluster ratestack-scylla -> kamo-scylla, correct ownership docs
This instance is kamo's: it backs the `kamouniverse` keyspace (read/written by **************** through the scylladb-direct ExternalName) and the `audit` keyspa...
Scale ratestack MySQL to 0 and suspend its backup job
The ratestack estate has been torn down to free cluster resources for kamocrm. The cluster-services MySQL instance serves ratestack only (kamo and mail each run...
Stop the avatar-resolver policy blocking every login
This policy took the whole platform's login offline for hours. Cilium enforces egress against the RESOLVED BACKEND, after service DNAT, so the port it matches ...
Egress containment for the avatar resolver, as a CiliumNetworkPolicy
Reinstates what 8f6c672 reverted, in the only form that actually works here. The earlier attempt failed because a plain NetworkPolicy cannot express this clust...
Drop the avatar-resolver egress policy
Reverts the policy added in 06db7db and patched in 24f76ae. It took SecurityService down for roughly six minutes in production (pod 0/1, Hikari connections dyin...
Contain egress from the avatar resolver; note the Redis eviction pool
SecurityService now runs a fetcher whose target URL is chosen by an untrusted party — a BIMI record is DNS content published by whoever owns a sending domain, a...
Master Machine uses VNC (x11vnc), not RDP
KRdp/RDP is incompatible with Guacamole: guacd has no RDP Graphics Pipeline (EGFX) and KRdp is H.264-only, so the connection authenticated but died with GRAPHIC...
Add "Master Machine" (kamo-dev-1) RDP device to Guacamole
Local Kubuntu workstation shares its live Plasma/Wayland session over RDP via KRdp. Adds an idempotent bootstrap SQL connection (192.168.4.46:3389, sage/shared ...
Pin CockroachDB to v24.2.4 to avoid v24.3 no-license txn throttle
v24.3 (and 24.2.5+) enforce the new CockroachDB Software License; with no license key the cluster throttles to 5 concurrent open transactions cluster-wide. A v2...
Pin RDP color-depth + reap xrdp sessions so god reuses one desktop
xrdp 0.9.24 keys sessions on <user,bpp> (Policy=Default) and never reaps disconnected ones (KillDisconnected=false, DisconnectedTimeLimit=0). The RDP user is al...
Stop expired/orphaned secrets shadowing live per-host certs
update_tls_store() rebuilt the TLSStore from a plain Secret listing, bucketed only on name prefix and type, with no check that a Certificate still owned the sec...
Like what you see shipping?
All of it arrives in your workspace on its own. Start on the free plan and read this page again in a month.