Allowlist Telnyx in both fail2ban jails
Adds the Telnyx US signaling addresses (192.76.120.10, 64.16.250.10 — sip.telnyx.com) to ignoreip now that the carrier trunk is provisioned. Also overrides ign...
Make fail2ban actually run — enable Asterisk's security log
fail2ban has never started on this PBX. Its asterisk-security jail watches /var/log/asterisk/security, a file Asterisk never created: FreePBX generates logger_l...
Bounce a bare visit to the Kamo login, and end the session on tab close
Closing the KamoDesktop tab left a live session sitting on the machine. Guacamole keeps its auth token in localStorage (GUAC_AUTH), which survives closing the t...
The auto-reconnect reload-looped every page into a blank screen
I shipped this and it took the desktop down: a blank white page and a tab loading forever. Disabled in-cluster immediately; this is the real fix. `ng-switch` s...
The auto-reconnect was watching for a class that never appears
It keyed on **************** That class exists in Guacamole's stylesheet and in none of its templates, so the selector matched nothing: the script loaded, ran, ...
Cap Guacamole's heap, and unbreak the deploy that was blocking it
Two faults, one of them mine. MINE FIRST: the `node --test` gate added with the auto-reconnect script pointed at a DIRECTORY, and the runner's node resolves a ...
Stop the canary crying wolf after every legitimate restart
session_durable treated a REJECTED token as a durability failure. That is not the outage it guards: a gateway restart, or Guacamole's own inactivity timeout, in...
Stop every deploy from throwing everyone off their desktop
"An error has occurred and this action cannot be completed" after leaving a desktop open was not an idle timeout and not a browser problem. It was CI. deploy-s...
An access canary that actually logs in, and alerts that reach a person
Two gaps, both proven against the live cluster today. FIRST: every existing KamoDesktop alert measures whether a COMPONENT is alive. All four were green during...
Run a dispatched session in the workspace's common parent
sage's VS Code is a multi-root workspace holding all 63 repositories, so taking the first workspace folder rooted a bug about the login site in aiservice — the ...
Say what a dead KamoDesktop connection actually means
Guacamole's stock CLIENT.ERROR_TUNNEL_204 is "The requested connection does not exist. Please check the connection name and try again." There is no connection n...
Finish removing Roundcube webmail
The Roundcube deployment/service and the mail.kamocrm.com IngressRoute were already dropped from the CI deploy list, but the MySQL bootstrap still created and g...
Submit the bug report without simulating a keystroke
vscode:// URI handler and sent a synthetic Return. Measured on desktop-1, that cannot work: KDE's screen locker holds an exclusive X input grab whenever the scr...
Dispatch a bug report into sage's open VS Code
URI handler that creates a conversation in a RUNNING window and pre-fills the input, which is the only way to reach the editor sage already has open instead of ...
auto-cert: let the junk-cert cleanup actually see the junk it was written for
cleanup_orphan_certificates() listed with '-l auto-cert.kamo/fqdn'. That label is newer than the scanner certs the function exists to remove, so the selector ex...
auto-cert: issue an org's certificates as a batch, not one every 90 seconds
A new organization needs a certificate for all eleven of its hosts, and each one is a separate cert-manager Certificate. The loop waited for each host to become...
Prefer brotli on the theme host, and correct the previous rationale
Two things, one of them a correction to the commit before this. The previous commit justified `encodings: [br, gzip]` with a probe against api.kamocrm.com. Tha...
Prefer brotli, stop compressing what is already compressed
The shared `compress` middleware was `compress: {}`, which is three separate defaults and none of them is the one we want. Traefik picks gzip whenever the clie...
Cache CORS preflights for two hours, not 100 seconds
`accessControlMaxAge: 100` meant browsers re-ran the preflight roughly once per visit, so every page load on the marketing site paid two extra OPTIONS round-tri...
Stop the edge rate limit from 429ing every cold KamoDesktop load
Opening an assigned Linux instance always landed on Guacamole's generic "An error has occurred and this action cannot be completed" modal, and always worked aft...
Provision MARKETING_EVENTS for visitor conversion signals
Marketing tracking codes credit a partner for the signups, organizations and paid accounts their link produced, and those three moments happen in services that ...
Make the HLS playlists immutable, and say what that costs
The playlists were the last entry in Lighthouse's cache-lifetime audit — 300s originally, then an hour, still reported. Everything else in the ladder was alread...
Compress the theme host, which was serving every text asset raw
MinIO returns object bytes exactly as stored — it does not negotiate Content-Encoding — and the two theme IngressRoutes carried `strip-hsts` and `cors` but no c...
Swap the hero reel to kamo13
kamo13.mp4 uploaded to MinIO at public/kamo13.mp4 (30 MB, sha256 174aaec6…, verified byte-identical to the local source after transfer — this link has a history...
Pin and verify the kubectl download instead of falling back to a broken wget
The deploy died at "Install kubectl". Three separate faults lined up: 1. curl aborted mid-stream with (56) "decryption failed or bad record mac" — the corru...
Correct the README to the routes that actually shipped
The WebRTC endpoint moved from /webrtc/<ch>/whep to /live/<ch>/whep when channels were renamed live/<ch> to drop StripPrefix, and the README still described the...
Name channels live/<name>, use Recreate, feed the loop a uniform GOP
Three defects found by exercising the deployed service. 1. The /live/ route 404'd through Traefik. MediaMTX answers the first HLS request with a 302 to `?co...
Move the encode job's decision logic out of the mc container
The plan stage died on `sed: command not found`. The minio/mc image is minimal to the point of having no sed, awk, grep, wget or curl — it can move objects and ...
Upload segments with an explicit glob, not --recursive
mc cp --recursive takes a directory; the segments were being passed as a glob, which is the wrong shape for that flag. Segments and the playlist beside them nee...
Deploy a streaming server and an ABR ladder for the hero reel
MediaMTX serves live channels over HLS, WebRTC, RTSP and RTMP, and loops the marketing reel as an always-available channel via alwaysAvailableFile — no ffmpeg s...
Preserve the client IP so the KAMOTEAM export ACL matches
Mounts failed with 'access denied by server'. A packet capture inside the guest showed the connection arriving from 10.42.0.151 — a pod-network address — not th...
Make the workstation fstab rewrite safe
Backs up /etc/fstab, normalises a missing trailing newline (a plain append would otherwise splice onto the last entry), and refuses to install a file that lost ...
Workstation-side mount script for the KAMOTEAM share
NFSv4 over the NodePort, mounted soft with a systemd automount so an unreachable cluster returns an error instead of hanging every process that touches the fold...
KAMOTEAM shared folder on every desktop
desktop-1 hosts /srv/kamoteam and exports it over NFSv4; every VM user gets ~/Desktop/KAMOTEAM as a symlink (via /etc/skel for new accounts and the agent for ex...
Stop xrdp restarts orphaning sessions and causing black screens
systemctl restart xrdp makes xrdp-sesman forget its in-memory session registry while the Xorg processes keep running. The next login cannot find the session so ...
Alert on OOMKills, and on the occupancy that precedes them
Nothing was watching memory. MediaService was OOMKilled every few hours for most of a day and EmailService six times, and the way anyone found out was a chat wi...
Block outbound SSH except sage, resolve k1m1/k3m1, bundle the agent dir
Outbound tcp/22 is rejected for every uid except root and sage, keyed on socket owner so future accounts are covered without re-running anything. Reapplied at b...
Standard toolchain, taskbar pins and passwordless installs for all users
Adds krita and maven **************** were already present), pins Chrome, VS Code, Kate, Spectacle and Krita to the taskbar for every user, and lets dev users i...
Allow SecurityService egress to the dev machine agent, and apply policies from CI
Cilium egress is default-deny for every endpoint it selects, and toEndpoints:[{}] is scoped to the policy's OWN namespace — so SecurityService could not reach d...
Agent secrets in CI, guacamole liveness, availability alerts
Alert expressions verified against the live Prometheus. Uses kube-state-metrics rather than kubevirt_* — the KubeVirt metrics endpoint is not scraped here, so r...
Idempotent agent installer, shared by cloud-init and live installs
Staged under /opt/kamo-desktop-agent so install-agent.sh has a distinct source and destination. Covers netplan-by-driver, cloud-init network disable, xrdp sessi...
Add the provisioning agent and its unit tests
Python 3 stdlib only. Every mutation runs subprocess with an argv list, never a shell string; passwords go to chpasswd on stdin so they stay out of /proc.
Pin the VM MAC and add RDP readiness/liveness probes
KubeVirt generates a random MAC per VMI when macAddress is absent. The guest's netplan matched on the first-boot MAC, so every VMI recreate silently orphaned th...
Implementation plan for dev machine provisioning
16 tasks across KlusterServices, kamo-shared-library, KamoInitializerService, SecurityService and kamo-internal.
Design for per-member dev machine provisioning
Covers the root cause of the 23-day KamoDesktop outage (netplan matching a first-boot MAC that KubeVirt regenerates on every VMI recreate), the live fix already...
Repair the kamouniverse backup, broken since the Yugabyte cutover
Renaming the dead CockroachBackupMissing alert in the previous commit made it fire immediately: no universe backup had succeeded since the 2026-08-09 migration....
Retire CockroachDB now that YugabyteDB owns the data
CockroachDB v24.2.4 was pinned to the last release under the old free BSL terms and was replaced by YugabyteDB on 2026-08-09. Nothing has connected to it since;...
Raise CPU limit off the 2-core ceiling
Both replicas sat pinned at 1997m/1999m against a 2-core limit for ~47h. That starved /languages enough that the liveness and readiness probes timed out (443 an...
Stop shipping the shared credential in this repo
The mail MySQL password sat in plaintext across fourteen files here — in ConfigMaps that kubectl will hand to anyone who can read them, in two Jobs that CI appl...
Like what you see shipping?
All of it arrives in your workspace on its own. Start on the free plan and read this page again in a month.