wuyang
97 lines
3.4 KiB
Markdown
97 lines
3.4 KiB
Markdown
# Operations runbook
|
|
|
|
## Production layout
|
|
|
|
- Public URL: `https://agent.k1412.top`
|
|
- NAS project: `/boot/config/plugins/compose.manager/projects/k1412-agent`
|
|
- Public host port: `12004`
|
|
- Execution provider: `ssh-docker`
|
|
- Execution host Docker root: `/srv/k1412-data/docker`
|
|
- Runtime model API route: `http://172.31.0.1:3000` (NAS host gateway
|
|
from the `k1412-agent_egress` network, bypassing the public proxy)
|
|
- Execution host workspace volumes: Docker volumes named
|
|
`k1412-ws-data-<hashed-user-id>`
|
|
|
|
No credential belongs in this document or repository.
|
|
|
|
## Deployment
|
|
|
|
Build and push immutable `linux/amd64` images:
|
|
|
|
```bash
|
|
docker buildx build --platform linux/amd64 -f docker/web.Dockerfile \
|
|
-t docker.k1412.top/wuyang/k1412-agent-web:<release> --push .
|
|
docker buildx build --platform linux/amd64 -f docker/runtime.Dockerfile \
|
|
-t docker.k1412.top/wuyang/k1412-agent-runtime:<release> --push .
|
|
docker buildx build --platform linux/amd64 -f docker/gateway.Dockerfile \
|
|
-t docker.k1412.top/wuyang/k1412-agent-gateway:<release> --push .
|
|
docker buildx build --platform linux/amd64 -f docker/workspace.Dockerfile \
|
|
-t docker.k1412.top/wuyang/k1412-agent-workspace:<release> --push .
|
|
```
|
|
|
|
Copy `deploy/docker-compose.yml`, `deploy/docker-compose.override.yml`, and
|
|
`deploy/docker-compose.ssh.yml` to the NAS project. Update only immutable image
|
|
tags and non-secret settings in the protected `.env`, validate with
|
|
`docker compose config`, pull, and recreate.
|
|
|
|
On the NAS deployment, set `MODEL_API_BASE_URL` to the internal model API route
|
|
listed above. `https://api.k1412.top` remains the external API entrypoint, but
|
|
Runtime must not send long-running Agent inference through the public reverse
|
|
proxy.
|
|
|
|
The SSH override removes `/var/run/docker.sock` from Gateway and mounts the
|
|
dedicated SSH directory read-only at `/root/.ssh`.
|
|
|
|
## Workspace migration
|
|
|
|
Before changing execution hosts:
|
|
|
|
1. stop Gateway so no new workspace mutation can begin;
|
|
2. enumerate containers and volumes with the
|
|
`app.k1412.component=user-workspace` label/name prefix;
|
|
3. stop each workspace container;
|
|
4. stream a tar archive of each named volume to an identically named volume on
|
|
the new Docker host;
|
|
5. pull the exact workspace image on the new host;
|
|
6. start Gateway with the new SSH target;
|
|
7. verify file listing and one read for every migrated volume;
|
|
8. retain the old volumes until the verification window ends.
|
|
|
|
Volume names contain only a user-ID hash, not email or display name.
|
|
|
|
## Verification
|
|
|
|
For every release:
|
|
|
|
```bash
|
|
.venv/bin/pytest
|
|
RUN_DOCKER_INTEGRATION=1 .venv/bin/pytest tests/test_docker_workspace.py
|
|
./scripts/verify-e2e.sh
|
|
./scripts/audit-images.sh
|
|
```
|
|
|
|
Production smoke checks must cover:
|
|
|
|
- anonymous requests are redirected/rejected;
|
|
- an approved user can use Chat;
|
|
- Chat can upgrade to Work but cannot downgrade;
|
|
- Work creates and verifies a real file;
|
|
- the file browser lists and downloads that file;
|
|
- a second user cannot see it;
|
|
- Gateway health reports `ssh-docker`;
|
|
- the workspace container exists on the physical execution host, not the NAS.
|
|
|
|
## Backup priorities
|
|
|
|
Back up:
|
|
|
|
1. PostgreSQL;
|
|
2. the Open WebUI data volume;
|
|
3. Runtime state in PostgreSQL;
|
|
4. per-user workspace volumes on the execution host;
|
|
5. the protected deployment `.env` and SSH directory through the private
|
|
infrastructure backup process.
|
|
|
|
Redis is reconstructible coordination state and is lower priority than the
|
|
database and workspace volumes.
|