wuyang/agent
2
2
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Audit Agent completion and recovery research

Browse Source
This commit is contained in:
wuyang
2026-07-28 00:06:57 +08:00
parent 5373d564ec
commit 3b55bfaad2
12 changed files with 840 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/08-next-work.md
+49 -14
View File
@@ -2,7 +2,29 @@
last_reviewed: 2026-07-27
## P0: Prove the Paper Compiler
## P0: Challenge the Completion Research
The immediate work is not an Agent experiment and not a new evaluation page. Review
`research/completion-verification/findings.md` with the user and determine whether its problem
decomposition is actually useful.
The review should test whether the research can answer, without falling back to paper titles:
1. what concrete capabilities an Agent evaluation is measuring;
2. what evidence is sufficient to claim completion in different task types;
3. where deterministic checks, trace rules, learned judges, and humans fail;
4. why failure detection, diagnosis, routing, restoration, and re-verification differ;
5. what has genuinely improved since reflection/state-only evaluation, and what remains unsolved.
For every challenged conclusion, return to the full-text evidence ledger, add a counterexample or
narrow the scope. Do not defend the synthesis by adding prose. The acceptance condition is that the
user can use the model to diagnose an unfamiliar Agent design and identify what its claimed result
does not prove.
Only after this review should the candidate completion certificate, failure object, and recovery router
be translated into task contracts. They are currently system inferences, not validated implementations.
## P1: Prove the Paper Compiler
The main project goal is now a high-efficiency knowledge funnel, not a paper browser and not a fixed
course. It should let one person process a corpus on the scale of 1,000 papers while retaining the
@@ -95,12 +117,20 @@ The first vertical slice is successful only if:
The frontend is optional. JSON/Markdown plus a rigorous review session is enough for the first proof.
## P1: Implement the Agent Evaluation Runner
## P2: Implement the Agent Evaluation Runner
The design contract is complete at
The first design contract exists at
`research/evaluation/k1412-agent-evaluation-v1.md`. Implementation belongs in
`https://git.k1412.top/wuyang/zk-data-agent.git`.
Implementation is paused. Before resuming, revise the contract using the completion research:
- separate outcome, forbidden-effect invariant, required procedure, and semantic residual;
- add `ambiguous` and `unverifiable` result states;
- model mutating actions, idempotency, rollback and compensation;
- record detection, localization, routing, restoration and re-verification separately;
- document validator coverage and known false-positive/false-negative boundaries.
Implement in this order:
1. extract a reusable runner from `scripts/eval-live-work.py`;
@@ -131,7 +161,7 @@ The report at `https://agent-eval.k1412.top/` explains the contract and simulate
It is not evidence that any Agent variant passed. Future result views must consume the same validator
JSONL instead of embedding numbers in frontend code.
## P1: Continue Collection Through Questions
## P2: Continue Collection Through Questions
The corpus now contains 1,125 papers, 1,110 of them queued. More collection is useful only when paired
with processing:
@@ -163,6 +193,11 @@ workflow:
Avoid another graph or dashboard redesign until either the paper compiler or evaluation runner produces
something the user repeatedly uses.
For the completion research, a later web presentation should begin with a concrete claim or failure,
then progressively reveal the completion contract, evidence, conflicting evaluator results, recovery
branches and source papers. It should not lead with a benchmark hierarchy, architecture diagram,
leaderboard or aggregate score.
## P2: Operations
Still missing:
@@ -187,16 +222,16 @@ Still missing:
For the knowledge-base repository:
1. choose one bounded topic and enumerate the complete relevant corpus;
2. define the fingerprint schema and evidence locators;
3. compile and audit a representative batch;
4. test whether research-move compression preserves important distinctions;
5. run the result with the user before building UI.
1. read `research/completion-verification/findings.md` with the user;
2. collect objections, ambiguous terms and missing counterexamples;
3. tighten or reject conclusions against the evidence ledger;
4. only then convert the accepted model into a minimal task-contract schema;
5. design the progressive web presentation after the knowledge shape is stable.
For `zk-data-agent`:
1. port the two currently runnable task contracts;
2. extract the live runner and add one deterministic validator;
3. export valid result JSONL;
4. run a smoke comparison only after deterministic layers pass;
5. record the decision and failures, not just the score.
1. do not port or run tasks yet;
2. wait for the completion-research review;
3. revise the v1 contract before extracting a runner;
4. retain the existing code inspection and scorer as historical implementation inputs;
5. never present the current evaluation report as benchmark evidence.