Resolve JVB crash loop — wrong MUC JID + port 8080 conflict
Two issues causing JVB to crash in a loop for 43+ hours: 1. JVB_BREWERY_MUC was **************** (full JID) but JVB appends XMPP_INTERNAL_MUC_DOMAIN automat...
JVB_BREWERY_MUC should be room name only, not full JID
Was **************** which caused JVB to construct a double-domain JID **************** with an @ in the domain part — crashing JVB with XmppStringprepException...
Commit pending infrastructure config changes across auto-cert, cockroachdb, coturn, janus, mail, nats, and traefik
Add dnsPolicy ClusterFirstWithHostNet to JVB StatefulSets so hostNetwork pods can resolve cluster DNS
Configure Prosody, Jicofo, and secrets for Jibri recording support
Enable recording in Prosody, add Jibri XMPP passwords to secrets, and configure Jicofo brewery MUC for Jibri instance discovery.
Add Jibri deployment manifests for Meet meeting recording
Kubernetes deployment, service, and configmap for Jibri (Meet Broadcasting Infrastructure) to enable meeting recording with MinIO storage integration.
Reference *** instead of non-existent turn-secret in Prosody deployment
Add Meet service deployment steps to CI/CD pipeline for k1m1 and k2m1
Add distributed JVB pool with per-node StatefulSets on k1m1 and k2m1
Add Jicofo conference focus with JVB brewery and lobby enforcement
Add Prosody XMPP server deployment with JWT auth and lobby support
Enable auto-truncate on embedding model to prevent 413 errors
Add --auto-truncate flag so inputs exceeding the 256 token limit are silently truncated instead of rejected with a validation error.
Load embedding model from local hostPath to prevent HuggingFace download failures
The embedding-model pod was in CrashLoopBackOff because it couldn't download the model from HuggingFace on startup (network/DNS issue). Mount the model from a h...
Auto-cert service not renewing expired SSL certificates
The auto-cert service was importing expired certs from shared storage and skipping Certificate CR creation, so cert-manager couldn't manage renewals. Also fixed...
Add Qdrant vector DB and MiniLM embedding model K8s manifests
- Qdrant v1.12.1 deployment + ClusterIP service (ports 6333/6334) - HuggingFace TEI with all-MiniLM-L6-v2 deployment + service (port 8080) - Updated deploy-serv...
Fix kubectl not found in pod, add docs alias, proactive SUBDOMAINS
- Mount /usr/local/bin/kubectl from host via hostPath instead of failing apt-get install (kubectl not in default Debian repos) - Add "docs" to SUBDOMAINS list...
Add wopi-*** for DocsService
Adds WOPI JWT signing secret used by DocsService for Docs access tokens. Deployed to both K1M1 and K2M1 clusters.
Remove forced WebSocket headers from media middleware
Traefik v3 handles WebSocket Upgrade/Connection headers natively. Remove forced headers that broke non-WebSocket API requests.
Add system account to all NATS configs
Add $SYS account with admin credentials to k0m1, k1m1, and k2m1 NATS configs. Enables nats CLI cluster management operations (peer removal, step-down, etc.) for...
Resilient probes so cluster stays up when one node is offline
- Liveness: tcpSocket on 4222 (do not depend on JetStream meta leader) - Readiness: initialDelay 90s, period 15s, failureThreshold 10 so k0m1+k1m1 can form qu...
Update Traefik routes: themes.* assets now only load from k1m1 minio instead of balancing across k0m1, k1m1, and k2m1
Use VPN addresses for k0m1/k2m1 minio-public backends
- Endpoints: k0m1 10.8.0.1, k2m1 10.8.2.1 (VPN); k1m1 stays local - IngressRoute host match: 100.64.x.x -> 10.8.0.1, 10.8.1.1, 10.8.2.1
Skip CockroachDB init job if any pods exist - cluster already initialized
Improve CockroachDB init check to wait for pods and verify cluster state
Prevent CockroachDB init job from running when cluster is already initialized
Update CockroachDB probes, HTTP redirect, and MinIO service endpoints
Add CORS middleware to theme routes to allow cross-origin access from all websites
Add HTTP to HTTPS redirect middleware and IngressRoute
- Create redirect-to-https middleware for permanent redirects - Add catch-all IngressRoute on web entrypoint to redirect all kamocrm.com domains to HTTPS - Fixe...
Remove hostPort from Traefik deployment
- Port 443 is already bound by ingress-nginx via CNI hostPort - Keep standard containerPort configuration
Change Traefik service to LoadBalancer with externalIPs for direct access
- Use LoadBalancer type with externalIPs instead of NodePort - Allows router to forward port 443 directly to service IP - This should bypass ingress-nginx hostP...
Add hostPort to Traefik deployment for direct port access
- Use hostPort 443 for websecure entrypoint to bypass NodePort - Allows router to forward port 443 directly to Traefik - Also set hostPort for web (30080) and a...
Set External Traffic Policy to Local for Traefik NodePort service
- Allows proper routing of external traffic from router - Preserves source IP addresses for better logging
Update CockroachDB StatefulSet to use dynamic VPN IP based on node hostname
- Fix advertise-addr to use VPN IP (10.8.1.1 for k1m1, 10.8.2.1 for k2m1) - Update join parameter to use VPN IPs instead of hostnames - Update node certificate ...
Add node certificates to CockroachDB secret creation
- Add node.crt and node.key to cockroachdb directory - Update CI/CD workflow to include node certificates when creating *** secret - Fixes CockroachDB CrashLoop...
Like what you see shipping?
All of it arrives in your workspace on its own. Start on the free plan and read this page again in a month.