zk-data-agent/docs/operations.md

Operations runbook

Production layout

• Public URL: https://agent.k1412.top
• NAS project: /boot/config/plugins/compose.manager/projects/k1412-agent
• Public host port: 12004
• Execution provider: ssh-docker
• Execution host Docker root: /srv/k1412-data/docker
• Runtime model API route: http://172.31.0.1:3000 (NAS host gateway from the k1412-agent_egress network, bypassing the public proxy)
• Execution host workspace volumes: Docker volumes named k1412-ws-data-<hashed-user-id>

No credential belongs in this document or repository.

Deployment

Build and push immutable linux/amd64 images:

docker buildx build --platform linux/amd64 -f docker/web.Dockerfile \
  -t docker.k1412.top/wuyang/k1412-agent-web:<release> --push .
docker buildx build --platform linux/amd64 -f docker/runtime.Dockerfile \
  -t docker.k1412.top/wuyang/k1412-agent-runtime:<release> --push .
docker buildx build --platform linux/amd64 -f docker/gateway.Dockerfile \
  -t docker.k1412.top/wuyang/k1412-agent-gateway:<release> --push .
docker buildx build --platform linux/amd64 -f docker/workspace.Dockerfile \
  -t docker.k1412.top/wuyang/k1412-agent-workspace:<release> --push .

Copy deploy/docker-compose.yml, deploy/docker-compose.override.yml, and deploy/docker-compose.ssh.yml to the NAS project. Update only immutable image tags and non-secret settings in the protected .env, validate with docker compose config, pull, and recreate.

On the NAS deployment, set MODEL_API_BASE_URL to the internal model API route listed above. https://api.k1412.top remains the external API entrypoint, but Runtime must not send long-running Agent inference through the public reverse proxy.

Set DEEPSEEK_API_BASE_URL=https://api.deepseek.com and place DEEPSEEK_API_KEY only in the protected deployment .env. The key is used only by Runtime for DeepSeek V4 Pro and must never be added to a browser, Compose file, image, repository, or log.

The SSH override removes /var/run/docker.sock from Gateway and mounts the dedicated SSH directory read-only at /root/.ssh.

Ollama model residency

The NAS Ollama service keeps Luna, Terra, and Sol resident with:

• OLLAMA_KEEP_ALIVE=-1;
• OLLAMA_MAX_LOADED_MODELS=3;
• OLLAMA_NUM_PARALLEL=1;
• OLLAMA_GPU_OVERHEAD=2147483648;
• Flash Attention and q8_0 KV cache enabled.

The three 32K-context runners were measured at approximately 2.88 GB, 4.12 GB, and 18.44 GB of GPU allocation respectively. Their combined allocation is about 25.44 GB on the 32 GB GPU. Do not increase context size or per-model parallelism without repeating the simultaneous-load test and leaving capacity for inference overhead.

The persistent Unraid template is /boot/config/plugins/dockerMan/templates-user/my-ollama.xml. A locked, idempotent five-minute check in /boot/config/plugins/dynamix/k1412-ollama.cron prewarms only missing models after a host or container restart. Confirm /api/ps lists all three with an infinite expiry after recovery.

Workspace migration

Before changing execution hosts:

1. stop Gateway so no new workspace mutation can begin;
2. enumerate containers and volumes with the app.k1412.component=user-workspace label/name prefix;
3. stop each workspace container;
4. stream a tar archive of each named volume to an identically named volume on the new Docker host;
5. pull the exact workspace image on the new host;
6. start Gateway with the new SSH target;
7. verify file listing and one read for every migrated volume;
8. retain the old volumes until the verification window ends.

Volume names contain only a user-ID hash, not email or display name.

Verification

For every release:

.venv/bin/pytest
RUN_DOCKER_INTEGRATION=1 .venv/bin/pytest tests/test_docker_workspace.py
./scripts/verify-e2e.sh
./scripts/audit-images.sh

Production smoke checks must cover:

• anonymous requests are redirected/rejected;
• an approved user sees exactly four Agent models;
• each model routes through the custom Agent loop;
• the Agent creates and verifies a real file;
• the file browser lists and downloads that file;
• a second user cannot see it;
• Gateway health reports ssh-docker;
• the workspace container exists on the physical execution host, not the NAS.

Backup priorities

Back up:

1. PostgreSQL;
2. the Open WebUI data volume;
3. Runtime state in PostgreSQL;
4. per-user workspace volumes on the execution host;
5. the protected deployment .env and SSH directory through the private infrastructure backup process.

Redis is reconstructible coordination state and is lower priority than the database and workspace volumes.