wuyang
125 lines
4.6 KiB
Markdown
125 lines
4.6 KiB
Markdown
# Operations runbook
|
|
|
|
## Production layout
|
|
|
|
- Public URL: `https://agent.k1412.top`
|
|
- NAS project: `/boot/config/plugins/compose.manager/projects/k1412-agent`
|
|
- Public host port: `12004`
|
|
- Execution provider: `ssh-docker`
|
|
- Execution host Docker root: `/srv/k1412-data/docker`
|
|
- Runtime model API route: `http://172.31.0.1:3000` (NAS host gateway
|
|
from the `k1412-agent_egress` network, bypassing the public proxy)
|
|
- Execution host workspace volumes: Docker volumes named
|
|
`k1412-ws-data-<hashed-user-id>`
|
|
|
|
No credential belongs in this document or repository.
|
|
|
|
## Deployment
|
|
|
|
Build and push immutable `linux/amd64` images:
|
|
|
|
```bash
|
|
docker buildx build --platform linux/amd64 -f docker/web.Dockerfile \
|
|
-t docker.k1412.top/wuyang/k1412-agent-web:<release> --push .
|
|
docker buildx build --platform linux/amd64 -f docker/runtime.Dockerfile \
|
|
-t docker.k1412.top/wuyang/k1412-agent-runtime:<release> --push .
|
|
docker buildx build --platform linux/amd64 -f docker/gateway.Dockerfile \
|
|
-t docker.k1412.top/wuyang/k1412-agent-gateway:<release> --push .
|
|
docker buildx build --platform linux/amd64 -f docker/workspace.Dockerfile \
|
|
-t docker.k1412.top/wuyang/k1412-agent-workspace:<release> --push .
|
|
```
|
|
|
|
Copy `deploy/docker-compose.yml`, `deploy/docker-compose.override.yml`, and
|
|
`deploy/docker-compose.ssh.yml` to the NAS project. Update only immutable image
|
|
tags and non-secret settings in the protected `.env`, validate with
|
|
`docker compose config`, pull, and recreate.
|
|
|
|
On the NAS deployment, set `MODEL_API_BASE_URL` to the internal model API route
|
|
listed above. `https://api.k1412.top` remains the external API entrypoint, but
|
|
Runtime must not send long-running Agent inference through the public reverse
|
|
proxy.
|
|
|
|
Set `DEEPSEEK_API_BASE_URL=https://api.deepseek.com` and place
|
|
`DEEPSEEK_API_KEY` only in the protected deployment `.env`. The key is used
|
|
only by Runtime for DeepSeek V4 Pro and must never be added to a browser,
|
|
Compose file, image, repository, or log.
|
|
|
|
The SSH override removes `/var/run/docker.sock` from Gateway and mounts the
|
|
dedicated SSH directory read-only at `/root/.ssh`.
|
|
|
|
## Ollama model residency
|
|
|
|
The NAS Ollama service keeps Luna, Terra, and Sol resident with:
|
|
|
|
- `OLLAMA_KEEP_ALIVE=-1`;
|
|
- `OLLAMA_MAX_LOADED_MODELS=3`;
|
|
- `OLLAMA_NUM_PARALLEL=1`;
|
|
- `OLLAMA_GPU_OVERHEAD=2147483648`;
|
|
- Flash Attention and `q8_0` KV cache enabled.
|
|
|
|
The three 32K-context runners were measured at approximately 2.88 GB, 4.12 GB,
|
|
and 18.44 GB of GPU allocation respectively. Their combined allocation is
|
|
about 25.44 GB on the 32 GB GPU. Do not increase context size or per-model
|
|
parallelism without repeating the simultaneous-load test and leaving capacity
|
|
for inference overhead.
|
|
|
|
The persistent Unraid template is
|
|
`/boot/config/plugins/dockerMan/templates-user/my-ollama.xml`. A locked,
|
|
idempotent five-minute check in
|
|
`/boot/config/plugins/dynamix/k1412-ollama.cron` prewarms only missing models
|
|
after a host or container restart. Confirm `/api/ps` lists all three with an
|
|
infinite expiry after recovery.
|
|
|
|
## Workspace migration
|
|
|
|
Before changing execution hosts:
|
|
|
|
1. stop Gateway so no new workspace mutation can begin;
|
|
2. enumerate containers and volumes with the
|
|
`app.k1412.component=user-workspace` label/name prefix;
|
|
3. stop each workspace container;
|
|
4. stream a tar archive of each named volume to an identically named volume on
|
|
the new Docker host;
|
|
5. pull the exact workspace image on the new host;
|
|
6. start Gateway with the new SSH target;
|
|
7. verify file listing and one read for every migrated volume;
|
|
8. retain the old volumes until the verification window ends.
|
|
|
|
Volume names contain only a user-ID hash, not email or display name.
|
|
|
|
## Verification
|
|
|
|
For every release:
|
|
|
|
```bash
|
|
.venv/bin/pytest
|
|
RUN_DOCKER_INTEGRATION=1 .venv/bin/pytest tests/test_docker_workspace.py
|
|
./scripts/verify-e2e.sh
|
|
./scripts/audit-images.sh
|
|
```
|
|
|
|
Production smoke checks must cover:
|
|
|
|
- anonymous requests are redirected/rejected;
|
|
- an approved user sees exactly four Agent models;
|
|
- each model routes through the custom Agent loop;
|
|
- the Agent creates and verifies a real file;
|
|
- the file browser lists and downloads that file;
|
|
- a second user cannot see it;
|
|
- Gateway health reports `ssh-docker`;
|
|
- the workspace container exists on the physical execution host, not the NAS.
|
|
|
|
## Backup priorities
|
|
|
|
Back up:
|
|
|
|
1. PostgreSQL;
|
|
2. the Open WebUI data volume;
|
|
3. Runtime state in PostgreSQL;
|
|
4. per-user workspace volumes on the execution host;
|
|
5. the protected deployment `.env` and SSH directory through the private
|
|
infrastructure backup process.
|
|
|
|
Redis is reconstructible coordination state and is lower priority than the
|
|
database and workspace volumes.
|