zk-data-agent/docs/experiments.md

Agent experiment system

中文 · English

Goal

The project exists to make Agent-loop iteration cheap, observable, and reversible. An experiment should change one behavioral hypothesis while holding authentication, workspace isolation, and delivery infrastructure constant.

The public portal at https://agent.k1412.top/doc/#experiments contains a curated experiment board. This Markdown document defines the durable process; the portal catalog is a presentation layer.

Experiment unit

Every experiment should define:

1. Question — the behavior we want to improve.
2. Hypothesis — the concrete loop change and expected direction.
3. Variant identifier — a stable string recorded in run.created.
4. Task set — fixed prompts, starting workspace fixtures, and model tier.
5. Metrics — success, quality, latency, model/tool usage, and regressions.
6. Safety constraints — behavior that must not worsen.
7. Decision rule — promote, iterate, or reject.
8. Artifact — code commit, configuration, raw event export, and report.

Do not compare variants from different task inputs or silently change the model, prompt, context budget, or workspace fixture.

Recommended metrics

Primary:

• task success under explicit evidence checks;
• verified-deliverable rate;
• unresolved tool-failure rate;
• human or rubric quality score.

Efficiency:

• end-to-end latency;
• model calls and iterations;
• prompt/completion tokens when available;
• tool-call count;
• duplicated or policy-blocked calls;
• child-Agent count and concurrency;
• estimated provider and electricity cost.

Reliability and safety:

• incomplete checkpoints correctly rejected;
• false-success rate;
• workspace escape or cross-user access failures;
• non-zero command exits incorrectly recorded as success;
• identity/authentication failures;
• Runtime or workspace restart recovery.

Evaluation layers

Unit and policy tests

Fast deterministic tests cover parsing, tool normalization, evidence gates, authentication, context selection, event persistence, and provider response normalization.

Real Docker integration

The integration test creates two disposable users and proves:

• separate containers, networks, and volumes;
• workspace path confinement;
• file browse/download/archive behavior;
• read-only root and dropped capabilities;
• persistent Python virtual environments;
• foreground and background pipefail exit codes.

Disposable full-stack E2E

The E2E stack uses a deterministic fake model provider and exercises:

• registration and admin approval;
• the exact four-model public catalog;
• the custom Agent loop;
• evidence-backed file generation;
• file delivery;
• per-user isolation.

Live-model evaluation

scripts/eval-live-work.py runs a real model against a unique disposable workspace and records latency, token usage, events, tool counts, and output files. Live evaluations complement deterministic tests; they do not replace them.

Production smoke test

After deployment, use a low-impact authenticated task to confirm public Web, Runtime, Gateway, the remote workspace host, and file retrieval. Remove only the smoke artifact after verification.

Current baselines

ID	Area	Status	Result
loop-v3-evidence	Completion policy	Production baseline	Artifacts require mutation followed by verification; reports and benchmarks require stronger evidence.
safe-parallel-v1	Scheduler	Production baseline	Consecutive read-only calls and read-only delegates run concurrently; mutations remain serialized.
recent-visible-v1	Context	Production baseline	Keep recent visible messages under a per-model character budget and attach up to eight memories.
workspace-python-v1	Execution	Validated	Persistent .venv, persistent user caches, 900-second tool ceiling, and true pipeline exits.
fresh-gateway-identity-v1	Authentication	Validated	Runtime re-signs verified identity for each tool request so long runs survive the original token expiry.

Initial backlog

1. Semantic or hybrid memory retrieval with conflict and expiry policy.
2. Structured old-context summarization with replay comparison.
3. Dependency-aware tool DAG scheduling rather than consecutive safe groups.
4. Child-Agent result contracts and budget allocation.
5. Experiment assignment stored per run and an aggregate comparison API.
6. Cost accounting across local electricity and paid provider tokens.
7. Workspace quotas, egress policy, abuse monitoring, and backup drills.

Adding an experiment

1. Add the hypothesis and metrics to this document or a dedicated docs/experiments/<id>.md file.
2. Add the experiment to docs/site/experiments.json for the public board.
3. Add or update the relevant strategy/scheduler/context identifier in run.created.
4. Add deterministic tests before a live run.
5. Run the standard verification and capture a report.
6. Promote with an immutable image tag; retain the previous tag for rollback.

Never put provider keys, private event payloads, user identifiers, prompts from real users, or internal credentials in an experiment artifact.