docs: publish architecture and experiment portal

docs/experiments.md

@@ -0,0 +1,133 @@
+# Agent experiment system
+
+## Goal
+
+The project exists to make Agent-loop iteration cheap, observable, and
+reversible. An experiment should change one behavioral hypothesis while
+holding authentication, workspace isolation, and delivery infrastructure
+constant.
+
+The public portal at <https://agent.k1412.top/doc/#experiments> contains a
+curated experiment board. This Markdown document defines the durable process;
+the portal catalog is a presentation layer.
+
+## Experiment unit
+
+Every experiment should define:
+
+1. **Question** — the behavior we want to improve.
+2. **Hypothesis** — the concrete loop change and expected direction.
+3. **Variant identifier** — a stable string recorded in `run.created`.
+4. **Task set** — fixed prompts, starting workspace fixtures, and model tier.
+5. **Metrics** — success, quality, latency, model/tool usage, and regressions.
+6. **Safety constraints** — behavior that must not worsen.
+7. **Decision rule** — promote, iterate, or reject.
+8. **Artifact** — code commit, configuration, raw event export, and report.
+
+Do not compare variants from different task inputs or silently change the
+model, prompt, context budget, or workspace fixture.
+
+## Recommended metrics
+
+Primary:
+
+- task success under explicit evidence checks;
+- verified-deliverable rate;
+- unresolved tool-failure rate;
+- human or rubric quality score.
+
+Efficiency:
+
+- end-to-end latency;
+- model calls and iterations;
+- prompt/completion tokens when available;
+- tool-call count;
+- duplicated or policy-blocked calls;
+- child-Agent count and concurrency;
+- estimated provider and electricity cost.
+
+Reliability and safety:
+
+- incomplete checkpoints correctly rejected;
+- false-success rate;
+- workspace escape or cross-user access failures;
+- non-zero command exits incorrectly recorded as success;
+- identity/authentication failures;
+- Runtime or workspace restart recovery.
+
+## Evaluation layers
+
+### Unit and policy tests
+
+Fast deterministic tests cover parsing, tool normalization, evidence gates,
+authentication, context selection, event persistence, and provider response
+normalization.
+
+### Real Docker integration
+
+The integration test creates two disposable users and proves:
+
+- separate containers, networks, and volumes;
+- workspace path confinement;
+- file browse/download/archive behavior;
+- read-only root and dropped capabilities;
+- persistent Python virtual environments;
+- foreground and background `pipefail` exit codes.
+
+### Disposable full-stack E2E
+
+The E2E stack uses a deterministic fake model provider and exercises:
+
+- registration and admin approval;
+- the exact four-model public catalog;
+- the custom Agent loop;
+- evidence-backed file generation;
+- file delivery;
+- per-user isolation.
+
+### Live-model evaluation
+
+`scripts/eval-live-work.py` runs a real model against a unique disposable
+workspace and records latency, token usage, events, tool counts, and output
+files. Live evaluations complement deterministic tests; they do not replace
+them.
+
+### Production smoke test
+
+After deployment, use a low-impact authenticated task to confirm public Web,
+Runtime, Gateway, the remote workspace host, and file retrieval. Remove only
+the smoke artifact after verification.
+
+## Current baselines
+
+| ID | Area | Status | Result |
+| --- | --- | --- | --- |
+| `loop-v3-evidence` | Completion policy | Production baseline | Artifacts require mutation followed by verification; reports and benchmarks require stronger evidence. |
+| `safe-parallel-v1` | Scheduler | Production baseline | Consecutive read-only calls and read-only delegates run concurrently; mutations remain serialized. |
+| `recent-visible-v1` | Context | Production baseline | Keep recent visible messages under a per-model character budget and attach up to eight memories. |
+| `workspace-python-v1` | Execution | Validated | Persistent `.venv`, persistent user caches, 900-second tool ceiling, and true pipeline exits. |
+| `fresh-gateway-identity-v1` | Authentication | Validated | Runtime re-signs verified identity for each tool request so long runs survive the original token expiry. |
+
+## Initial backlog
+
+1. Semantic or hybrid memory retrieval with conflict and expiry policy.
+2. Structured old-context summarization with replay comparison.
+3. Dependency-aware tool DAG scheduling rather than consecutive safe groups.
+4. Child-Agent result contracts and budget allocation.
+5. Experiment assignment stored per run and an aggregate comparison API.
+6. Cost accounting across local electricity and paid provider tokens.
+7. Workspace quotas, egress policy, abuse monitoring, and backup drills.
+
+## Adding an experiment
+
+1. Add the hypothesis and metrics to this document or a dedicated
+   `docs/experiments/<id>.md` file.
+2. Add the experiment to `docs/site/experiments.json` for the public board.
+3. Add or update the relevant strategy/scheduler/context identifier in
+   `run.created`.
+4. Add deterministic tests before a live run.
+5. Run the standard verification and capture a report.
+6. Promote with an immutable image tag; retain the previous tag for rollback.
+
+Never put provider keys, private event payloads, user identifiers, prompts from
+real users, or internal credentials in an experiment artifact.