Agent Guide: Mathbox Server

This server runs Docker, Certbot, and HAProxy. All services are exposed under *.mathbox.90.cz via HAProxy.

Scope of this document:

  • production host operations only
  • deployment and HAProxy/certificate procedures
  • no application-level feature behavior (see module README files)

Quick Rules (Must Follow)

  • Do not install system packages or new services on the host.
  • Deploy applications only inside Docker containers.
  • Each project must be controlled via a Docker Compose file in its project directory.
  • Read /etc/haproxy/haproxy.cfg before making changes to learn existing domains and ports.
  • Always acquire the HAProxy lock before changes and release it after.
  • Always configure a proper Let's Encrypt certificate with Certbot.

Key Paths

  • HAProxy config: /etc/haproxy/haproxy.cfg
  • Docker deployments: /home/agent/docker_deployments/$PROJECT
  • HAProxy helpers: /usr/local/sbin/haproxy-apply, /usr/local/sbin/haproxy-le

Core Rules

  • Only add or adjust projects; do not remove or break existing ones.
  • Project directories must be owned by the agent user.
  • Use domains in the form $PROJECT.mathbox.90.cz.

Server Lock Procedure

  • Before any HAProxy change, acquire the lock with haproxy-apply --lock.
  • If the lock is already held, do not change configuration. Inform the user.
  • After your changes are complete, release the lock with haproxy-apply --unlock.

Deploying a New Project (Docker)

  1. Create the project directory: /home/agent/docker_deployments/$PROJECT.
  2. Place the Docker config there (compose file, env files, etc.).
  3. Start the containers for the project.
  4. Choose a project port exposed on localhost.
  5. Update HAProxy to route a subdomain (e.g. $PROJECT.mathbox.90.cz) to that port.
  6. Issue a Let's Encrypt certificate for the new domain(s).

Standard Docker Compose Template

Use a docker-compose.yml in each project directory:

version: "3.8"
services:
  app:
    build: .
    ports:
      - "5000:5000"
    restart: unless-stopped

Deployment Checklist

  1. Confirm the project directory exists and is owned by agent.
  2. Ensure docker-compose.yml exists and describes the service.
  3. Start the service with docker compose up -d.
  4. Verify the local port responds on 127.0.0.1:<port>. U reply-pilot-db muze byt PostgreSQL vedome publikovana i mimo loopback, pokud je to explicitne nastavene v modulovem env pres HOST_DB_BIND.
  5. Lock HAProxy, update config, and apply with haproxy-apply.
  6. Issue/renew certificates with haproxy-le.
  7. Validate HTTP and HTTPS from the public domain.
  8. Unlock HAProxy.

Pro schema zmeny plati expand/contract poradi: standardni DB deploy nejdriv aplikuje jen zpetne kompatibilni expand migrace, potom se nasadi konzumenti. Destruktivni contract migrace nejsou soucasti standardniho deploye a mohou probehnout az v pozdejsim samostatnem deployi po overeni vsech konzumentu. Backend deploy se povazuje za hotovy az po uspesnem Docker healthchecku. Pro firemni email import policy (0086) nasad nejdrive DB expand migraci, potom Google modul s internim cache-thread endpointem, BE, worker a web app. Cache replay pouziva stavajici interni sit a bind mounty; zadny novy port, OAuth grant ani presun mailbox dat neni potreba.

Reply Pilot Module Layout

Exception logs in Grafana

reply-pilot-app and reply-pilot-be write JSON Lines to stdout. Each exception and its causes are stored in stack_trace in the same entry as level and message, so Loki's |= "ERROR" filter retains the trace. In Explore, expand the entry and inspect stack_trace, or render it with:

{server="mathbox.90.cz", container=~"reply-pilot-(app|be)"} | json | level="ERROR" | line_format "{{.message}}\n{{.stack_trace}}"

For older plain-text logs, remove the ERROR line filter or use Show context on the error entry: their subsequent traceback lines may be separate entries. This format change does not reprocess historical logs or alter the host collector.

Module directories

Reply Pilot je multi-service repozitar. Na serveru proto nepokladej vse do jednoho adresare data/ a conf/ pro cely projekt. Kazdy modul ma mit vlastni podadresar:

  • /home/agent/docker_deployments/reply-pilot/reply-pilot-app/
  • budoucne napriklad /home/agent/docker_deployments/reply-pilot/<module_name>/

V kazdem modulovem adresari ma byt:

  • docker-compose.yml
  • .env.server pro serverovy deploy daneho modulu; na deploy klientu se ma standardne vyrenderovat z secrets/prod/<module>.env pres sops
  • .env.local pro lokalni start workflow daneho modulu; pokud repo pouziva sops, ma se pri startu materializovat z secrets/local/<module>.env
  • pripadne dalsi plaintext runtime artefakty materializovane ze sops, typicky service account JSON z secrets/<environment>/<module>-<purpose>.json
  • u reply-pilot-google Gmail OAuth token materializovany ze secrets/<environment>/reply-pilot-google-token.json do data/gmail_token.json
  • u reply-pilot-google take per-user Calendar OAuth client konfigurace a sifrovaci klic v modulovem secrets/<environment>/reply-pilot-google.env; tyto hodnoty nepatri do reply-pilot-be env
  • data/, conf/
  • scripts/ s operacnimi skripty pojmenovanymi podle kratkeho nazvu modulu bez prefixu reply-pilot-, napriklad app-start.sh
  • modulovy README.md

Remote deploy pred rozbalenim nove verze odstrani predchozi modulovy kod, aby na serveru nezustavaly smazane nebo prejmenovane soubory. Server-owned adresare data/ a conf/ se pri tom zachovavaji a deploy je neprepise.

Aktualni moduly:

  • /home/agent/docker_deployments/reply-pilot/reply-pilot-app/
  • /home/agent/docker_deployments/reply-pilot/reply-pilot-be/
  • /home/agent/docker_deployments/reply-pilot/reply-pilot-google/
  • /home/agent/docker_deployments/reply-pilot/reply-pilot-worker/
  • /home/agent/docker_deployments/reply-pilot/reply-pilot-search/
  • /home/agent/docker_deployments/reply-pilot/reply-pilot-docs/
  • /home/agent/docker_deployments/reply-pilot/reply-pilot-db/
  • /home/agent/docker_deployments/reply-pilot/reply-pilot-db-backup/
  • /home/agent/docker_deployments/reply-pilot/reply-pilot-jira-reports/

Port, domena a HAProxy backend se resi per modul. reply-pilot-search standardne nema verejnou HAProxy routu, ale muze publikovat lokalni loopback port pro operatorni GET /api/search a Solr admin UI. Solr admin UI nema byt verejne vystaveny bez dalsi autentizace nebo jine ochranne vrstvy. Search modul se nasazuje atomicky jako dva kontejnery: Spring Boot aplikace reply-pilot-search a oficialni vendor kontejner reply-pilot-search-solr. Oba bezi pod serverovym HOST_UID:HOST_GID; Solr home zustava v server-owned reply-pilot-search/data/solr a nesmi se pri deployi mazat. Compose u Solru spousti primo solr-foreground, protoze vendor entrypoint pred startem vyzaduje zapis do /var/solr pod image UID, ktery se neshoduje se serverovym deploy UID. reply-pilot-docs standardne publikuje loopback port 127.0.0.1:9095 a verejnou domenu reply-pilot-docs.mathbox.90.cz pres HAProxy. reply-pilot-worker muze mit interni HTTP status API na Docker siti pro ostatni moduly, ale standardne nema mit host port ani HAProxy routu. Jeho image pouziva Java 21 a /app/reply-pilot-worker.jar se Spring ThreadPoolTaskScheduler; na server se neinstaluje Java ani Python. Compose spousti jednu instanci pod HOST_UID:HOST_GID a kontroluje ji pres Java --healthcheck. Pri nahrazeni Python verze standardni deploy odstrani stary modulovy kod a zachova server-owned data/ a conf/; existujici env klice, interni /healthz a /statusz zustavaji stejne. Dva schedulery nesmi bezet soucasne, aby nespoustely stejne joby dvakrat. reply-pilot-db-backup take nema host port ani HAProxy routu. Zapisuje logical backupy do /home/agent/docker_deployments/reply-pilot/reply-pilot-db-backup/data/backups; adresar i soubory musi vlastnit agent:agent, nikdy root. reply-pilot-jira-reports standardne publikuje staticke reporting artefakty z vlastniho data/public/ na loopback portu 127.0.0.1:9096. Verejna HAProxy routa je reply-pilot-jira-reports.mathbox.90.cz a smeruje na backend 127.0.0.1:9096; tato routa vedome zverejnuje staticke Jira reporting artefakty. Pro emailove grafy reply-pilot-jira-reports vola reply-pilot-google pres interni Docker sit reply-pilot-internal, standardne na http://reply-pilot-google:5000/api/reports/email-weekly-counts; Gmail modul pocita agregaci ze sve vlastni account-scoped emailove cache/runtime dat pod /app/data/accounts/<GMAIL_CACHE_ACCOUNT_ID>/emails. Jedna Gmail cache obsahuje inbox i sent-only vlakna pro danou Gmail schranku a Gmail watch defaultne sleduje labely INBOX,SENT. Emailova data se mezi moduly nesdili pres bind mount.

Jira search a changelogy vola reply-pilot-jira-reports pres http://reply-pilot-be:5000/api/internal/jira-reports/*. Reports SOPS env drzi jen interni JIRA_REPORTS_BACKEND_API_TOKEN; odpovidajici JIRA_REPORTS_API_TOKEN a vsechny Jira Cloud credentials vlastni BE SOPS env.

reply-pilot-be vola stejny Google modul pres GOOGLE_API_BASE_URL take pro per-user Google Calendar a osobni Gmail schranky. Backend drzi pouze aplikacni autorizaci a opaque sifrovany credential envelope v DB; OAuth exchange, refresh/revoke, sifrovani a vsechna volani Google Calendar/Gmail API probiha v reply-pilot-google. Osobni Gmail cache je oddelena pod /app/data/accounts/app-user-<id>/emails a je pristupna pouze po overeni sync_inbox v backendu. Per-user interni endpointy vyzaduji bearer GOOGLE_API_TOKEN, ktery musi mit BE a Google modul v danem prostredi nastaveny na stejnou hodnotu. Sdilena schranka pouziva puvodni GMAIL_WATCH_* Pub/Sub zdroje; osobni schranky pouzivaji GMAIL_PERSONAL_WATCH_TOPIC_NAME a samostatnou environment-specific GMAIL_PERSONAL_WATCH_SUBSCRIPTION_NAME, bez fallbacku na sdilenou subscription.

Osobni DB import omezuje PERSONAL_GMAIL_IMPORT_DAYS v BE SOPS env (local i prod default 15, 0 = bez limitu), podle data nejnovejsi zpravy celeho vlakna v APP_TIMEZONE=Europe/Prague, vcetne hranicniho dne. Zmena vyzaduje redeploy BE. Google cache pod /home/agent/docker_deployments/reply-pilot/reply-pilot-google/data/accounts/app-user-<id>/emails stale obsahuje celou stazenou historii INBOX/SENT. Novy bearer-protected endpoint POST /api/user-gmail/cache/threads umoznuje workerem triggerovane cyklicke prehodnoceni cache bez nove Gmail udalosti nebo OAuth grantu; sync_inbox zustava povinny. Pri nasazeni RP-5012 nasad nejprve Google modul s timto endpointem, potom BE. Existujici worker job email_import_backfill zajistuje replay; neni potreba nova DB migrace, mount ani port. Dosavadni historie a prilohy zustavaji zachovane. Podrobnosti a davkovani jsou v Gmail.

Od dokonceni produkcni stabilizace 2026-08-18 je jedinym podporovanym Google runtime modulem, DNS nazvem a host path reply-pilot-google, respektive /home/agent/docker_deployments/reply-pilot/reply-pilot-google/. Predchozi reply-pilot-gmail alias, kontejner, image a host path byly po overeni Gmail, Calendar a reporting konzumentu odstraneny.

HAProxy Changes

MCP se nasazuje jako soucast reply-pilot-be, bez vlastniho kontejneru nebo serveroveho adresare. Osobni bearer tokeny autentizuji Pippu i externi klienty na jedinem endpointu /mcp; service bearer a delegacni hlavicky uz nejsou podporovane. Pro verejny pristup routuj pouze presnou cestu /mcp na domene reply-pilot.mathbox.90.cz na stavajici BE host port (default 9091). Backend /api/*, vcetne spravy tokenu, musi zustat privatni za Flask aplikaci. Verejny MCP endpoint pouziva HTTPS bez presmerovani na interaktivni login; preda Authorization backendu a nesmi tuto hlavicku logovat. APP routy si ponechavaji dosavadni prihlasovani. Over HAProxy konfiguraci pomoci helperu nize. Nezmenenou verejnou routu nelze povazovat za overenou pouhou zmenou BE kodu. Nejprve aplikuj migraci 0092, pak BE a APP. BE SOPS env obsahuje samostatny MCP_TOKEN_ENCRYPTION_KEY a MCP_ALLOWED_HOSTS vcetne verejne domeny. Pippa vola /mcp pres loopback HTTP s osobnim tokenem typu Pippa; pri otevreni Pippy se opravnenemu uzivateli chybejici platny token vytvori automaticky. Deploy zachova server-owned conf/; volitelny conf/pippa.yaml prebije verzovany default-conf/pippa.yaml z image. Po uprave configu restartuj BE. Podrobnosti jsou v MCP.

Use the HAProxy apply helper to update config safely: - sudo /usr/local/sbin/haproxy-apply --lock - sudo /usr/local/sbin/haproxy-apply /var/lib/haproxy-agent/incoming/new.cfg - The script validates the config, backs up the current version, applies, and reloads. - On reload failure, it automatically rolls back. - sudo /usr/local/sbin/haproxy-apply --unlock

Let's Encrypt Certificates

HAProxy is configured to route /.well-known/acme-challenge/ to a local Certbot listener on 127.0.0.1:8899.

Use the helper to issue or renew certificates: - Issue: - sudo /usr/local/sbin/haproxy-le --email ops@mathbox.90.cz -d $PROJECT.mathbox.90.cz -d www.$PROJECT.mathbox.90.cz - Renew all: - sudo /usr/local/sbin/haproxy-le --renew

If a cert already exists for any requested domain, the helper auto-expands it to include all requested domains. The helper creates fullchain.pem.key symlinks to privkey.pem for HAProxy. Configure HAProxy to use fullchain.pem and rely on the .key sidecar file. HAProxy is reloaded automatically after successful issuance or renewal.

Allowed Sudo Commands

  • sudo /usr/local/sbin/haproxy-apply ...
  • sudo /usr/local/sbin/haproxy-le ...

Rollback

  • To revert HAProxy to the last working config: sudo /usr/local/sbin/haproxy-apply --revert

Troubleshooting

  • If a service is down, check the local port first: curl -sS http://127.0.0.1:<port>.
  • If HAProxy reload fails, check /var/log/haproxy.log or journalctl -u haproxy.
  • If cert issuance fails, re-run with --staging and check /var/log/letsencrypt/letsencrypt.log.

Documentation Requirement

For each project you deploy or modify, create a README in the project directory that includes: - Ports used - Domain(s) - How to start/stop the service - Special operational notes

Runtime User and File Ownership (mandatory)

  • Podrobny projektovy runtime kontrakt je v docs/container-runtime-contract.md.
  • Aplikacni proces v kontejneru MUSI bezet jako non-root uzivatel.
  • Pro tento projekt se runtime uzivatel predava pres HOST_UID a HOST_GID do Compose, pokud vendor image nevyzaduje vlastni startup model.
  • Pri remote deployi se HOST_UID/HOST_GID maji standardne odvodit z uzivatele, pod kterym bezi deploy na serveru (id -u, id -g).
  • Relativni host path v modulovych .env souborech se berou vuci modulu, ne vuci cwd shellu.
  • Bind mount adresar data/ na hostu musi zustat zapisovatelny timto uzivatelem.
  • U Reply Pilotu se to posuzuje per modul, napriklad reply-pilot-app/data.
  • Pokud se objevi root-owned soubory v data/, je potreba pred dalsim startem srovnat ownership na runtime UID:GID.
  • reply-pilot-db je zdokumentovana vendor-backed vyjimka: oficialni PostgreSQL image si ridi init flow sama, ale stale musi mit oddelene host path pro data/ a conf/.

Logovani

Jedinou povolenou variantou je zapis sluzeb na stdout/stderr. Na serveru ani lokalne se nevytvareji aplikacni log soubory, modulove logs/ adresare ani log bind mounty. Logy se ctou pres docker compose logs; persistenci, retenci a rotaci zajistuje serverovy container log collector.