Explicit priority so the OTA updates route wins over the general theme route
Traefik ranks by rule-string length; the general /public/ theme route is longer, so the expo-updates header injection never ran. priority:100000 forces the win.
Self-hosted expo-updates headers for Mobile Copilot OTA
expo-protocol-version:1 + expo-sfv-version:0 response headers that the expo-updates client requires and static MinIO cannot emit. Longer path prefix wins over t...
Expose AIService /ws/ai-chat via api.* hosts for mobile clients
IngressRoute (priority 100, websecure, ws-upgrade middleware sans compress) routes wss://api.<domain>/ws/ai-chat to kamowsai-service:80 — same pattern as the me...
Raise guacamole memory 1Gi→2Gi to stop OOM mid-RDP-session
The Tomcat webapp idles near ~800Mi, so a 1Gi ceiling left almost no headroom. Opening an RDP session pushed RSS over 1Gi and the container was OOMKilled (exit ...
Only deploy components whose config changed; never needlessly reinstall
deploy-services.yml re-applied every service AND reinstalled cluster operators (cert-manager, KubeVirt, CDI — with 600s waits) on every push to main, so a one-l...
Move data dir off MinIO FUSE mount to local disk
rspamd's /var/lib/rspamd was hostPath-backed by the rclone FUSE mount of the MinIO shared-files bucket. Every hyperscan recompile / map refresh rewrote files th...
/dev/shm 2Gi (headless Chrome needs >=2GB) + 6Gi mem limit
Jibri's 10-config init aborts with '/dev/shm must be at least 2GB' when shm is 512Mi, crash-looping the recorder. Bump the tmpfs to 2Gi and the memory limit to ...
Meet-secrets **************** for jibri recording ingest
Lets jibri finalize.sh authenticate its recording upload to MediaService; matches the cluster *** value MediaService validates against.
Make whisper-asr independently scalable (HPA, multi-node)
Remove the single-node pin and switch to a per-pod model cache so pods can schedule on any node, and add an HPA (min 1 / max 4, CPU 75%, deliberate scale-up giv...
Rebrand Guacamole client connection status text to KamoDesktop
Override **************** so the 'Connecting to Guacamole...' screen (and related status) reads KamoDesktop.
Correct Guacamole branding manifest version field
Manifest used guacManifestVersion:1.0; the bundled extensions use guacamoleVersion. The wrong field made Guacamole reject the branding extension ('Unable to rea...
Prosody moderator module, whisper-asr, jibri recording upload
- Meet-prosody: version-control the custom-plugins ConfigMap (captures the existing token_verification no-op override that was created out-of-band) and add ...
Enable Guacamole json-auth SSO + KamoDesktop rebrand
- JSON_SECRET_KEY (optional) on guacamole from the shared desktop-sso secret enables the bundled guacamole-auth-json extension the god-SSO links validate agai...
Design spec for KamoDesktop god SSO icons + Guacamole rebrand
Two god-only nav icons (Linux/Windows) in kamo-internal that open Guacamole in a SecurityService (god-eligible gate), deep-linked straight to each instance. Reb...
Move guacd to pod network (MTU fix) + unicast WoL
guacd on hostNetwork (k1m1 eno49 MTU 1500) vs guacamole pod on cilium overlay (MTU 1420): RDP connected but large frames were dropped across the boundary -> 'Gu...
Watch the desktop namespace for IngressRoutes
desktop.kamocrm.com IngressRoute lives in the desktop ns, but Traefik's CRD provider only watched **************** — so the route was ignored and requests fell ...
Import VM disk as Filesystem, not Block
CDI's block importer crashlooped: 'blockdev: cannot open /dev/cdi-block-volume: Permission denied' — it can't access the raw Longhorn block device. Filesystem v...
Correct KubeVirt VM cloud-init field + modernize spec
- cloudInitNoCloud userdata secret field is 'secretRef' (not userDataSecretRef); KubeVirt v1.8.4 pruned the wrong field -> 'must have at least one userdatasou...
Hardened KubeVirt remote desktop + Guacamole gateway
Adds a full remote-desktop stack, GitOps-applied via the deploy workflow: - KubeVirt + CDI installed at deploy time (pinned-stable release URLs) - desktop-1: K...
Design spec for hardened KubeVirt remote desktop + Guacamole gateway
KubeVirt Kubuntu VM (16Gi/500Gi Longhorn) behind Apache Guacamole HTML5, exposed at desktop.kamocrm.com via Traefik (TLS + rate-limit + IP-ban). Adds LAN Window...
Proactively provision browser-only aliases (media.*)
The log-scan discovery mechanism can never bootstrap the `media` alias: it is only ever contacted by the browser over a WebSocket/XHR subresource (SockJS /ws/in...
Provision SOCIAL_MESSAGES JetStream stream (social.inbound.>, social.lead.>) R1 for social-media inbound pipeline
Provision R1 MEDIA JetStream stream via idempotent bootstrap Job
Media/lead-feed events (media.feed.*, media.post.*) had no JetStream stream, so publishes returned '503 No Responders Available For Request' (chat.> is covered ...
node-config(k1m1-runner): record data.forgejo.org IPv4 pin + capacity=6
Documents the live on-node change: data.forgejo.org is IPv6-only and k1m1 has no IPv6 route, so job containers hung ~133s and actions/checkout failed. Pin its I...
Correct FPBXCONF heredoc + repoint universe Scylla
- freepbx: the FPBXCONF here-document closing delimiter was indented (8 spaces); bash requires <<WORD delimiters at column 0, so startup.sh died with 'unexpec...
Note ratestack Forgejo :3000 node registry config requirement
ratestack apps pull from the Forgejo registry 192.168.4.22:3000 (plain HTTP, auth required), which must be added to each node's registries.yaml (mirror -> http ...
Pin kamo Traefik to k1m1; make auto-cert single-cluster
SSL errors: kamo Traefik is a LoadBalancer with externalIPs 192.168.4.22 (k1m1) + externalTrafficPolicy=Local, so it MUST run on k1m1 — the pod rebalance moved ...
Consolidate klusterservices onto single k1m1+k3m1 cluster
- NATS: standalone single pod pinned k1m1 (drop k2m1 + generic cluster variants) - CockroachDB: pin k1m1, single-node --join - MySQL: single instance moved in f...
Bootstrap universe-minio secret + ugc bucket note (§14)
Add step 6 to bootstrap-universe.sh: creates the universe-minio secret (ACCESS_KEY/SECRET_KEY from universe-backup-s3 root creds) and notes that the universe-ug...
Upgrade k1m1 node image v24.2.4 -> v24.3.33
Rolling major upgrade of the k1m1 CRDB StatefulSet node. Note: cluster will not finalize to 24.3 until the k3m1 node is also upgraded (tracked separately).
Drop the decommissioned k2m1 cluster section
k2m1 (10.8.2.1:6443) is decommissioned and unreachable; its steps lived in the same deploy_services job and failed it (missing kubeconfig / no route) even when ...
Indent heredoc bodies so the startup.sh configmap is valid YAML
The startup.sh block scalar embedded 5 shell heredocs (rtp.conf, http.conf, an SQL block, WebRTC endpoints, freepbx.conf) whose bodies sat at column 0 — less in...
Don't let a missing kubectl cache dir kill the whole job
The kubectl-install step runs 'set -eo pipefail' then find **************** -name kubectl ... | head -1 When that cache dir is absent (cleared between runner ...
Upgrade v3.0 -> v3.3 (fix intermittent WebSocket disconnect)
v3.0.0 intermittently completes the WS 101 upgrade then closes the socket before any frame flows — the recurring 'disconnect on load' in the universe game clien...
Move Dovecot maildirs to a local-path PVC (off rclone/MinIO FUSE)
Maildirs were on hostPath /mnt/shared-files/email/mail — the rclone FUSE mount backed by MinIO object storage, which can't safely host a maildir (no atomic rena...
Put MySQL datadir on a local-path PVC instead of rclone/MinIO FUSE
The mail MySQL datadir (and maildirs) were on hostPath /mnt/shared-files, an rclone FUSE mount backed by MinIO object storage, which cannot durably or consisten...
Backup verified working — DB-scoped backup, MinIO region us-west-ca-s-1, EXTERNALIOIMPLICITACCESS grant in bootstrap
Drop WITH revision_history from backup (enterprise-only); core BACKUP is OSS
node-config: document k1m1 forgejo-runner config (--add-host for submodule DNS)
Job containers on k1m1 fall back to public DNS, so the fleet-swept http://k1m1:3000 .gitmodules URLs fail to clone there. Pin k1m1 into job containers via conta...
node-config: document k1m1 docker daemon insecure-registries
The k1m1 forgejo-runner's builds push to the HTTP Forgejo registry; the daemon.json shipped with only the :5000 entries, so every build job landing on k1m1 fail...
K1m1 image registry replacing retired k0m1:5000 (same credentials)
Loki 6.x SingleBinary needs zeroed scalable targets; crdb exec via localhost (cert SANs)
Kamo-universe namespace+quota+valkey, CockroachDB backups, observability stack
- universe/: namespace + ResourceQuota, valkey-universe (AOF, hostPath k1m1), nightly CockroachDB backup CronJob, operator bootstrap script - monitoring/: pin...
Join shared two-node cluster with k3m1 over LAN; account isolation
Retire WireGuard routes (10.8.x.x) for LAN routes (192.168.4.22 <-> 192.168.4.36 :6222) with creds embedded in the route URLs. Rename cluster to kamo-ratestack-...
Add k3m1 node + migrate cluster peering to LAN IPs
- statefulset: advertise/join over 192.168.4.x (k1m1=.22, k3m1=.36); WireGuard dropped (k3m1 wg0 10.8.0.1 collided with k0m1's advertise IP). - init-job + CI ...
Refresh TLSStore every loop so newly-issued certs get served
update_tls_store() only ran at startup and on shared-storage imports, so a cert freshly issued by cert-manager (HTTP-01) was Ready with a tls-* secret but never...
Give redirect-only route a backend so Traefik v3 enables it
Traefik v3 disables an IngressRoute router with services: [] when allowEmptyServices is off (as it is here), so the domain-agnostic HTTP->HTTPS redirect router ...
Like what you see shipping?
All of it arrives in your workspace on its own. Start on the free plan and read this page again in a month.