Corp/ops/deploy/README.md

CORP blue-green production deployment

Production is deployed from the current master commit in Gitea repository Robotail/Corp. The Gitea runner only opens a restricted SSH connection. Build, backup, migrations, health checks, and traffic switching run on the CORP server.

Two native slots are used instead of containers:

• blue: backend 9001, frontend 3001;
• green: backend 9002, frontend 3002;
• releases: /srv/eusrr-releases/<commit-sha>;
• slot links: /srv/eusrr-slots/{blue,green};
• active release link: /srv/eusrr-current.

The inactive release is built and checked while the active release continues to serve users. Nginx then reloads an atomic upstream change. Existing WebSocket connections may reconnect when the old backend is drained, but HTTP traffic does not require a maintenance window. Old and new Next.js static assets are retained in /srv/eusrr-shared/next-static so already-open pages keep working.

Database migrations run before traffic switches and therefore must be backward compatible with the currently active application. Do not combine destructive schema removal with the release that stops using the schema. PostgreSQL snapshots are stored in /var/backups/eusrr/deploy for 14 days, but they are not restored automatically while the old release is accepting writes.

Bootstrap

1. Generate a dedicated Ed25519 runner key. Never commit its private part.

2. Copy the public key to CORP and run:

   sudo /srv/eusrr/ops/deploy/install-corp-runner-access /tmp/eusrr-gitea-runner.pub
   

3. Store the private key as repository Actions secret CORP_DEPLOY_SSH_KEY in Gitea.

4. To receive deployment reports in Telegram, add repository Actions secrets TELEGRAM_BOT_TOKEN and TELEGRAM_CHAT_ID. The chat ID may identify a user, group, or channel that the bot is allowed to message.

5. Run Deploy production manually from master and verify the first blue deployment before enabling automatic master deployments.

The SSH key is forced through /usr/local/sbin/eusrr-deploy-ssh. The deploy command rejects every revision except the current Gitea origin/master SHA. Sudo permits exactly one lowercase 40-character SHA argument, not administrative flags. Slot links are held in a root-owned directory and validated before any cleanup. Application units and builds run with NoNewPrivileges so repository code cannot gain privileges using setuid executables or sudo. Fetching uses the literal approved Gitea SSH URL, ignores global Git/SSH config, disables Git hooks, rejects URL rewrites, and verifies the pinned Gitea host key from root-owned /etc/eusrr/gitea_known_hosts. A different origin fails closed. Logs are appended to /var/log/eusrr-deploy.log; the active slot and successful SHA are recorded under /var/lib/eusrr-deploy.

Failure handling and disk usage

The active and previous successful releases are retained. Before each attempt, unused Git worktrees are removed under the deployment lock. Failed builds are removed on error, SIGTERM, SIGINT or SIGHUP, after their processes and slot services have stopped. Symlink releases, mounted directories and referenced releases are never removed. A failed cleanup aborts the operation instead of deleting broadly.

Node.js must be 22.13+ on the 22.x LTS line, or 24.x LTS; this is checked before creating a candidate. Frontend npm ci uses strict engine validation and the committed lockfile. Network-only failures may retry up to five installation attempts under one 30-minute timeout and the same disk watchdog. Completed downloads are preferred from the candidate-local cache between attempts. Dependency, engine and postinstall failures are not retried. The per-candidate npm cache is removed after installation and with any failed candidate.

Each Python candidate bootstraps the pinned, SHA-256-verified pip wheel in pip-bootstrap.txt. Package downloads use bounded retries/resume for truncated network transfers. TLS and package hash checks remain enabled; a real integrity mismatch still fails the deployment. The system pip and active venv are not changed.

The deployment requires at least 4 GiB and 50,000 free inodes before creating a release. Build and database-dump process groups have a 30-minute timeout and are monitored every two seconds; they are stopped if free space falls below 512 MiB or free inodes below 50,000. This reserve is a guard, not a filesystem quota: other services can also consume space. Pip uses no persistent cache, and the npm download cache is confined to the candidate release and removed after install. Database snapshots are validated before receiving the final .dump filename; partial dumps are removed on failure. Existing complete dumps retain their 14-day retention. Deployment and native Celery logs rotate through logrotate. Legacy Celery log names are included too. Persistent journald storage is limited to 512 MiB with a 2 GiB free-space reserve and 14-day retention after the journal service next restarts; installing tooling does not vacuum existing evidence.

Before building the frontend, pip dependency checks and a real isolated Uvicorn WebSocket upgrade/frame probe verify the Python environment. Before switching traffic the probe runs again and verifies the candidate rejects anonymous /ws/ connections. It never creates a user, login ticket, task or database row. HTTP health checks send the trusted HTTPS-forwarding header over loopback, allowing production SSL redirect to remain enabled. These checks complement, not replace, business-operation regression tests. The host must also provide ImageMagick (magick or convert) for the existing bounded oversized-product-image conversion path; the tooling self-test and candidate checks fail early if this system dependency is missing.

A pending-deployment journal survives a power failure or SIGKILL. The next run cleans up an interrupted candidate only when the current symlink, active slot, successful revision and Nginx upstreams still agree. If traffic/state changes were interrupted, deployment stops for manual recovery and preserves the releases and upstream backup. Database changes are never automatically reversed.

Legacy Celery services (including their umbrella unit) are stopped and disabled once native workers are live. Systemd conditions prevent dependency-driven restarts of legacy workers. The old slot must stop successfully before the new Celery scheduler starts. Gunicorn receives graceful SIGTERM on the master only, with a 60-second systemd deadline beyond its 30-second graceful timeout. Normal Next.js SIGTERM exit code 143 is not considered a crash.

To update tooling without deploying application code:

sudo /path/to/ops/deploy/install-deploy-hardening

The installer stages protected copies, validates them, and backs up every changed script, unit and configuration. A validation failure restores the previous files. It reloads systemd definitions without restarting the active application. It does not automatically perform cleanup; maintenance remains an explicit separate step.

For subsequent maintenance use sudo /usr/local/sbin/eusrr-deploy --maintenance. This acquires the same lock, validates live state, reconciles legacy workers and cleans orphan releases. It neither builds a release nor runs migrations. Linux integration tests run with:

python3 -m unittest discover -s ops/deploy/tests -p test_deploy_lifecycle.py -v

The final workflow step reports successful, failed, cancelled, and unknown deployment outcomes with the repository, branch, revision, actor, current production state, and Gitea run link. Reporting is skipped when either Telegram secret is absent. The runner tries curl, wget, and Python's standard HTTP library in that order. Missing tools, network failures, Telegram API errors, and reporting timeouts are logged but never change the deployment result.