wuyang
134 lines
5.1 KiB
Markdown
134 lines
5.1 KiB
Markdown
# Agent experiment system
|
|
|
|
## Goal
|
|
|
|
The project exists to make Agent-loop iteration cheap, observable, and
|
|
reversible. An experiment should change one behavioral hypothesis while
|
|
holding authentication, workspace isolation, and delivery infrastructure
|
|
constant.
|
|
|
|
The public portal at <https://agent.k1412.top/doc/#experiments> contains a
|
|
curated experiment board. This Markdown document defines the durable process;
|
|
the portal catalog is a presentation layer.
|
|
|
|
## Experiment unit
|
|
|
|
Every experiment should define:
|
|
|
|
1. **Question** — the behavior we want to improve.
|
|
2. **Hypothesis** — the concrete loop change and expected direction.
|
|
3. **Variant identifier** — a stable string recorded in `run.created`.
|
|
4. **Task set** — fixed prompts, starting workspace fixtures, and model tier.
|
|
5. **Metrics** — success, quality, latency, model/tool usage, and regressions.
|
|
6. **Safety constraints** — behavior that must not worsen.
|
|
7. **Decision rule** — promote, iterate, or reject.
|
|
8. **Artifact** — code commit, configuration, raw event export, and report.
|
|
|
|
Do not compare variants from different task inputs or silently change the
|
|
model, prompt, context budget, or workspace fixture.
|
|
|
|
## Recommended metrics
|
|
|
|
Primary:
|
|
|
|
- task success under explicit evidence checks;
|
|
- verified-deliverable rate;
|
|
- unresolved tool-failure rate;
|
|
- human or rubric quality score.
|
|
|
|
Efficiency:
|
|
|
|
- end-to-end latency;
|
|
- model calls and iterations;
|
|
- prompt/completion tokens when available;
|
|
- tool-call count;
|
|
- duplicated or policy-blocked calls;
|
|
- child-Agent count and concurrency;
|
|
- estimated provider and electricity cost.
|
|
|
|
Reliability and safety:
|
|
|
|
- incomplete checkpoints correctly rejected;
|
|
- false-success rate;
|
|
- workspace escape or cross-user access failures;
|
|
- non-zero command exits incorrectly recorded as success;
|
|
- identity/authentication failures;
|
|
- Runtime or workspace restart recovery.
|
|
|
|
## Evaluation layers
|
|
|
|
### Unit and policy tests
|
|
|
|
Fast deterministic tests cover parsing, tool normalization, evidence gates,
|
|
authentication, context selection, event persistence, and provider response
|
|
normalization.
|
|
|
|
### Real Docker integration
|
|
|
|
The integration test creates two disposable users and proves:
|
|
|
|
- separate containers, networks, and volumes;
|
|
- workspace path confinement;
|
|
- file browse/download/archive behavior;
|
|
- read-only root and dropped capabilities;
|
|
- persistent Python virtual environments;
|
|
- foreground and background `pipefail` exit codes.
|
|
|
|
### Disposable full-stack E2E
|
|
|
|
The E2E stack uses a deterministic fake model provider and exercises:
|
|
|
|
- registration and admin approval;
|
|
- the exact four-model public catalog;
|
|
- the custom Agent loop;
|
|
- evidence-backed file generation;
|
|
- file delivery;
|
|
- per-user isolation.
|
|
|
|
### Live-model evaluation
|
|
|
|
`scripts/eval-live-work.py` runs a real model against a unique disposable
|
|
workspace and records latency, token usage, events, tool counts, and output
|
|
files. Live evaluations complement deterministic tests; they do not replace
|
|
them.
|
|
|
|
### Production smoke test
|
|
|
|
After deployment, use a low-impact authenticated task to confirm public Web,
|
|
Runtime, Gateway, the remote workspace host, and file retrieval. Remove only
|
|
the smoke artifact after verification.
|
|
|
|
## Current baselines
|
|
|
|
| ID | Area | Status | Result |
|
|
| --- | --- | --- | --- |
|
|
| `loop-v3-evidence` | Completion policy | Production baseline | Artifacts require mutation followed by verification; reports and benchmarks require stronger evidence. |
|
|
| `safe-parallel-v1` | Scheduler | Production baseline | Consecutive read-only calls and read-only delegates run concurrently; mutations remain serialized. |
|
|
| `recent-visible-v1` | Context | Production baseline | Keep recent visible messages under a per-model character budget and attach up to eight memories. |
|
|
| `workspace-python-v1` | Execution | Validated | Persistent `.venv`, persistent user caches, 900-second tool ceiling, and true pipeline exits. |
|
|
| `fresh-gateway-identity-v1` | Authentication | Validated | Runtime re-signs verified identity for each tool request so long runs survive the original token expiry. |
|
|
|
|
## Initial backlog
|
|
|
|
1. Semantic or hybrid memory retrieval with conflict and expiry policy.
|
|
2. Structured old-context summarization with replay comparison.
|
|
3. Dependency-aware tool DAG scheduling rather than consecutive safe groups.
|
|
4. Child-Agent result contracts and budget allocation.
|
|
5. Experiment assignment stored per run and an aggregate comparison API.
|
|
6. Cost accounting across local electricity and paid provider tokens.
|
|
7. Workspace quotas, egress policy, abuse monitoring, and backup drills.
|
|
|
|
## Adding an experiment
|
|
|
|
1. Add the hypothesis and metrics to this document or a dedicated
|
|
`docs/experiments/<id>.md` file.
|
|
2. Add the experiment to `docs/site/experiments.json` for the public board.
|
|
3. Add or update the relevant strategy/scheduler/context identifier in
|
|
`run.created`.
|
|
4. Add deterministic tests before a live run.
|
|
5. Run the standard verification and capture a report.
|
|
6. Promote with an immutable image tag; retain the previous tag for rollback.
|
|
|
|
Never put provider keys, private event payloads, user identifiers, prompts from
|
|
real users, or internal credentials in an experiment artifact.
|