Live Change Log

Transport is COPY ... TO STDOUT WITH (FORMAT CSV), not `cockroach sql --format=csv`. The latter renders SQL NULL as the literal string "NULL", indistinguishable...

Cross-checking SHOW BACKUP (714 objects) against a table_schema='public' inventory (680) exposed 34 tables living in five other schemas: email_metadata, email_p...

Cross-checking SHOW BACKUP (714 objects) against a table_schema='public' inventory (680) exposed 34 tables living in five other schemas: email_metadata, email_p...

Two passes: cheap batched count(*), then an order-independent checksum. The checksum sums per-row md5 hashes as integers rather than the obvious md5(string_agg...

Two passes: cheap batched count(*), then an order-independent checksum. The checksum sums per-row md5 hashes as integers rather than the obvious md5(string_agg...

15 tasks. The two gates sit before any downtime: Task 5 proves the backups actually restore, and Task 10 diffs the KamoInitializer-built schema against the live...

15 tasks. The two gates sit before any downtime: Task 5 proves the backups actually restore, and Task 10 diffs the KamoInitializer-built schema against the live...

Records the measured current state (3.4 GB, 791 tables, 671 FKs, ~30 consumers), the five design decisions, and the cutover sequence. The two findings that sha...

respondingTimeouts.readTimeout is the deadline for reading a WHOLE request, body included, and Traefik v3 defaults it to 60s where v2 had no limit. That silentl...

Same fault as embedding-model, found by auditing every hostPath volume in the repo. A hostPath is node-local and nothing replicates it, so an unpinned pod can b...

The model is a hostPath under /var/lib/embedding-model on k1m1, and nothing replicates it. When k3m1 joined, the new ReplicaSet's pod was scheduled there and ne...

password_query returned `email as user`, which overwrote the %Lu-lowercased login name with the row's stored casing. user_query then interpolated %n from the re...

coturn is hostNetwork with no nodeSelector, so it was scheduled on k3m1 (public 47.181.8.87). Its own --external-ip, and every client's `stun:kamocrm.com:3478`,...

The bridge never completed a single ICE connection in the 35 days this pod has been up (0 successes, 18 terminations). Every conference joined over XMPP fine an...

Root cause: the liveness probe was `tcpSocket: port 10000`, but Scylla's REST API binds to api_address 127.0.0.1 (its default) while a tcpSocket probe dials the...

scylladb/ held complete manifests but had no deploy step — the comment said it was "deployed MANUALLY after the 16TB drive swap" because the data dir /mnt/backu...

Auditing what a full reconcile would apply turned up two manifests that had drifted far enough from the cluster that re-applying them would have caused an outag...

This instance is kamo's: it backs the `kamouniverse` keyspace (read/written by **************** through the scylladb-direct ExternalName) and the `audit` keyspa...

The ratestack estate has been torn down to free cluster resources for kamocrm. The cluster-services MySQL instance serves ratestack only (kamo and mail each run...

This policy took the whole platform's login offline for hours. Cilium enforces egress against the RESOLVED BACKEND, after service DNAT, so the port it matches ...

Reinstates what 8f6c672 reverted, in the only form that actually works here. The earlier attempt failed because a plain NetworkPolicy cannot express this clust...

Reverts the policy added in 06db7db and patched in 24f76ae. It took SecurityService down for roughly six minutes in production (pod 0/1, Hikari connections dyin...

SecurityService now runs a fetcher whose target URL is chosen by an untrusted party — a BIMI record is DNS content published by whoever owns a sending domain, a...

KRdp/RDP is incompatible with Guacamole: guacd has no RDP Graphics Pipeline (EGFX) and KRdp is H.264-only, so the connection authenticated but died with GRAPHIC...

Local Kubuntu workstation shares its live Plasma/Wayland session over RDP via KRdp. Adds an idempotent bootstrap SQL connection (192.168.4.46:3389, sage/shared ...

v24.3 (and 24.2.5+) enforce the new CockroachDB Software License; with no license key the cluster throttles to 5 concurrent open transactions cluster-wide. A v2...

xrdp 0.9.24 keys sessions on <user,bpp> (Policy=Default) and never reaps disconnected ones (KillDisconnected=false, DisconnectedTimeLimit=0). The RDP user is al...

update_tls_store() rebuilt the TLSStore from a plain Secret listing, bucketed only on name prefix and type, with no check that a Certificate still owned the sec...

Traefik ranks by rule-string length; the general /public/ theme route is longer, so the expo-updates header injection never ran. priority:100000 forces the win.

expo-protocol-version:1 + expo-sfv-version:0 response headers that the expo-updates client requires and static MinIO cannot emit. Longer path prefix wins over t...

IngressRoute (priority 100, websecure, ws-upgrade middleware sans compress) routes wss://api.<domain>/ws/ai-chat to kamowsai-service:80 — same pattern as the me...

The Tomcat webapp idles near ~800Mi, so a 1Gi ceiling left almost no headroom. Opening an RDP session pushed RSS over 1Gi and the container was OOMKilled (exit ...

deploy-services.yml re-applied every service AND reinstalled cluster operators (cert-manager, KubeVirt, CDI — with 600s waits) on every push to main, so a one-l...

rspamd's /var/lib/rspamd was hostPath-backed by the rclone FUSE mount of the MinIO shared-files bucket. Every hyperscan recompile / map refresh rewrote files th...

Jibri's 10-config init aborts with '/dev/shm must be at least 2GB' when shm is 512Mi, crash-looping the recorder. Bump the tmpfs to 2Gi and the memory limit to ...

Lets jibri finalize.sh authenticate its recording upload to MediaService; matches the cluster *** value MediaService validates against.

Remove the single-node pin and switch to a per-pod model cache so pods can schedule on any node, and add an HPA (min 1 / max 4, CPU 75%, deliberate scale-up giv...

Override **************** so the 'Connecting to Guacamole...' screen (and related status) reads KamoDesktop.

Manifest used guacManifestVersion:1.0; the bundled extensions use guacamoleVersion. The wrong field made Guacamole reject the branding extension ('Unable to rea...

- Meet-prosody: version-control the custom-plugins ConfigMap (captures the existing token_verification no-op override that was created out-of-band) and add ...

- JSON_SECRET_KEY (optional) on guacamole from the shared desktop-sso secret enables the bundled guacamole-auth-json extension the god-SSO links validate agai...

Two god-only nav icons (Linux/Windows) in kamo-internal that open Guacamole in a SecurityService (god-eligible gate), deep-linked straight to each instance. Reb...

guacd on hostNetwork (k1m1 eno49 MTU 1500) vs guacamole pod on cilium overlay (MTU 1420): RDP connected but large frames were dropped across the boundary -> 'Gu...

desktop.kamocrm.com IngressRoute lives in the desktop ns, but Traefik's CRD provider only watched **************** — so the route was ignored and requests fell ...

CDI's block importer crashlooped: 'blockdev: cannot open /dev/cdi-block-volume: Permission denied' — it can't access the raw Longhorn block device. Filesystem v...

- cloudInitNoCloud userdata secret field is 'secretRef' (not userDataSecretRef); KubeVirt v1.8.4 pruned the wrong field -> 'must have at least one userdatasou...

Adds a full remote-desktop stack, GitOps-applied via the deploy workflow: - KubeVirt + CDI installed at deploy time (pinned-stable release URLs) - desktop-1: K...

KubeVirt Kubuntu VM (16Gi/500Gi Longhorn) behind Apache Guacamole HTML5, exposed at desktop.kamocrm.com via Traefik (TLS + rate-limit + IP-ban). Adds LAN Window...

The log-scan discovery mechanism can never bootstrap the `media` alias: it is only ever contacted by the browser over a WebSocket/XHR subresource (SockJS /ws/in...