Audit Agent completion and recovery research

2026-07-28 00:06:57 +08:00
parent 5373d564ec
commit 3b55bfaad2
12 changed files with 840 additions and 21 deletions
@@ -260,3 +260,41 @@ The report was deployed as immutable image
 `docker.k1412.top/wuyang/agent-eval:20260727T084425Z-cd1eacb` through Unraid Compose Manager on NAS port
 12005. NPM route and Let's Encrypt HTTPS were enabled for `https://agent-eval.k1412.top/`, and the site
 was registered in the k1412 homepage `projects` category.
+
+## 13. Stop Implementing, Research Completion First
+
+After reviewing the evaluation report, the user rejected the work as useful research. The failure was
+not that the site lacked another chart. The design had moved from code inspection to task contracts,
+scoring, and presentation without first establishing what current Agent research can actually prove
+about completion and recovery.
+
+The user explicitly paused experiments and requested general conclusions before any new web form.
+The project therefore audited a problem-driven 40-paper full-text pool spanning false success, state
+and procedure verification, evaluator validity, test adequacy, tool failure, root-cause attribution,
+recovery routing, and stopping. Thirty-seven papers entered an evidence ledger; three remained
+historical context.
+
+The main corrections are:
+
+- task completion is an evidence contract, not an Agent statement or one benchmark bit;
+- concrete capabilities include state change, side-effect control, procedure compliance, coordination,
+  failure detection, localization, recovery routing, restoration, re-verification, and repeated
+  reliability;
+- deterministic state checks, trace rules, learned judges, and humans have different responsibilities;
+- verification only creates an explicit failure; it does not provide diagnosis or recovery;
+- fixed retry, reflection, replanning, and repair loops all have documented regressions;
+- the benchmark protocol and verifier must themselves be audited for weak tests, leakage, drift,
+  model changes, and reward hacking.
+
+The cross-paper synthesis proposes a completion certificate, structured failure object, and
+state-dependent recovery router. These are marked as system inferences, not experimental results.
+
+Decision retained: do not implement the Agent evaluation runner or redesign its report until the user
+accepts the research decomposition. If implementation resumes, rewrite the task contracts around
+outcome, invariant, procedure, semantic residual, uncertainty, and explicit recovery stages.
+
+Current source of truth:
+
+- `research/completion-verification/findings.md`
+- `research/completion-verification/evidence-ledger.md`
+- `experiments/knowledge-compilation/2026-07-27-agent-completion-research-audit.md`