Corp/ops/deploy/README.md

# CORP blue-green production deployment

Production is deployed from the current `master` commit in Gitea repository
`Robotail/Corp`. The Gitea runner only opens a restricted SSH connection. Build,
backup, migrations, health checks, and traffic switching run on the CORP server.

Two native slots are used instead of containers:

- `blue`: backend `9001`, frontend `3001`;
- `green`: backend `9002`, frontend `3002`;
- releases: `/srv/eusrr-releases/<commit-sha>`;
- slot links: `/srv/eusrr-slots/{blue,green}`;
- active release link: `/srv/eusrr-current`.

The inactive release is built and checked while the active release continues to
serve users. Nginx then reloads an atomic upstream change. Existing WebSocket
connections may reconnect when the old backend is drained, but HTTP traffic does
not require a maintenance window. Old and new Next.js static assets are retained
in `/srv/eusrr-shared/next-static` so already-open pages keep working.

Database migrations run before traffic switches and therefore must be backward
compatible with the currently active application. Do not combine destructive
schema removal with the release that stops using the schema. PostgreSQL snapshots
are stored in `/var/backups/eusrr/deploy` for 14 days, but they are not restored
automatically while the old release is accepting writes.

## Bootstrap

1. Generate a dedicated Ed25519 runner key. Never commit its private part.
2. Copy the public key to CORP and run:

   ```sh
   sudo /srv/eusrr/ops/deploy/install-corp-runner-access /tmp/eusrr-gitea-runner.pub
   ```

3. Store the private key as repository Actions secret
   `CORP_DEPLOY_SSH_KEY` in Gitea.
4. To receive deployment reports in Telegram, add repository Actions secrets
   `TELEGRAM_BOT_TOKEN` and `TELEGRAM_CHAT_ID`. The chat ID may identify a user,
   group, or channel that the bot is allowed to message.
5. Run **Deploy production** manually from `master` and verify the first blue
   deployment before enabling automatic master deployments.

The SSH key is forced through `/usr/local/sbin/eusrr-deploy-ssh`. The deploy
command rejects every revision except the current Gitea `origin/master` SHA.
Sudo permits exactly one lowercase 40-character SHA argument, not administrative
flags. Slot links are held in a root-owned directory and validated before any
cleanup. Application units and builds run with `NoNewPrivileges` so repository
code cannot gain privileges using setuid executables or sudo.
Fetching uses the literal approved Gitea SSH URL, ignores global Git/SSH config,
disables Git hooks, rejects URL rewrites, and verifies the pinned Gitea host key
from root-owned `/etc/eusrr/gitea_known_hosts`. A different origin fails closed.
Logs are appended to `/var/log/eusrr-deploy.log`; the active slot and successful
SHA are recorded under `/var/lib/eusrr-deploy`.

## Failure handling and disk usage

The active and previous successful releases are retained. Before each attempt,
unused Git worktrees are removed under the deployment lock. Failed builds are
removed on error, SIGTERM, SIGINT or SIGHUP, after their processes and slot services
have stopped. Symlink releases, mounted directories and referenced releases are
never removed. A failed cleanup aborts the operation instead of deleting broadly.

Node.js must be 22.13+ on the 22.x LTS line, or 24.x LTS; this is checked before
creating a candidate. Frontend `npm ci` uses strict engine validation and the
committed lockfile. Network-only failures may retry up to five installation
attempts under one 30-minute timeout and the same disk watchdog. Completed
downloads are preferred from the candidate-local cache between attempts. Dependency,
engine and postinstall failures are not retried. The per-candidate npm cache is
removed after installation and with any failed candidate.

Each Python candidate bootstraps the pinned, SHA-256-verified pip wheel in
`pip-bootstrap.txt`. Package downloads use bounded retries/resume for truncated
network transfers. TLS and package hash checks remain enabled; a real integrity
mismatch still fails the deployment. The system pip and active venv are not changed.

The deployment requires at least 4 GiB and 50,000 free inodes before creating a
release. Build and database-dump process groups have a 30-minute timeout and are
monitored every two seconds; they are stopped if free space falls below 512 MiB
or free inodes below 50,000. This reserve is a guard, not a filesystem quota:
other services can also consume space. Pip uses no persistent cache, and the npm
download cache is confined to the candidate release and removed after install.
Database snapshots are validated before receiving the final `.dump` filename;
partial dumps are removed on failure. Existing complete dumps retain their
14-day retention. Deployment and native Celery logs rotate through logrotate.
Legacy Celery log names are included too. Persistent journald storage is limited
to 512 MiB with a 2 GiB free-space reserve and 14-day retention after the journal
service next restarts; installing tooling does not vacuum existing evidence.

Before building the frontend, pip dependency checks and a real isolated Uvicorn
WebSocket upgrade/frame probe verify the Python environment. Before switching
traffic the probe runs again and verifies the candidate rejects anonymous `/ws/`
connections. It never creates a user, login ticket, task or database row. HTTP
health checks send the trusted HTTPS-forwarding header over loopback, allowing
production SSL redirect to remain enabled. These checks complement, not replace,
business-operation regression tests.
The host must also provide ImageMagick (`magick` or `convert`) for the existing
bounded oversized-product-image conversion path; the tooling self-test and
candidate checks fail early if this system dependency is missing.

A pending-deployment journal survives a power failure or SIGKILL. The next run
cleans up an interrupted candidate only when the current symlink, active slot,
successful revision and Nginx upstreams still agree. If traffic/state changes were
interrupted, deployment stops for manual recovery and preserves the releases and
upstream backup. Database changes are never automatically reversed.

Legacy Celery services (including their umbrella unit) are stopped and disabled
once native workers are live. Systemd conditions prevent dependency-driven
restarts of legacy workers. The old slot must stop successfully before the new
Celery scheduler starts. Gunicorn receives graceful `SIGTERM` on the master only,
with a 60-second systemd deadline beyond its 30-second graceful timeout. Normal
Next.js SIGTERM exit code 143 is not considered a crash.

To update tooling without deploying application code:

```sh
sudo /path/to/ops/deploy/install-deploy-hardening
```

The installer stages protected copies, validates them, and backs up every changed
script, unit and configuration. A validation failure restores the previous files.
It reloads systemd definitions without restarting the active application. It does
not automatically perform cleanup; maintenance remains an explicit separate step.

For subsequent maintenance use `sudo /usr/local/sbin/eusrr-deploy --maintenance`.
This acquires the same lock, validates live state, reconciles legacy workers and
cleans orphan releases. It neither builds a release nor runs migrations.
Linux integration tests run with:

```sh
python3 -m unittest discover -s ops/deploy/tests -p test_deploy_lifecycle.py -v
```

The final workflow step reports successful, failed, cancelled, and unknown
deployment outcomes with the repository, branch, revision, actor, current
production state, and Gitea run link. Reporting is skipped when either Telegram
secret is absent. The runner tries `curl`, `wget`, and Python's standard HTTP
library in that order. Missing tools, network failures, Telegram API errors, and
reporting timeouts are logged but never change the deployment result.