Audit Agent completion and recovery research
This commit is contained in:
wuyang
@@ -1,7 +1,7 @@
|
||||
# Project Status
|
||||
|
||||
last_reviewed: 2026-07-27
|
||||
baseline_before_update: `8e4ff37`
|
||||
baseline_before_update: `5373d56`
|
||||
branch: `main`
|
||||
remote: `https://git.k1412.top/wuyang/agent.git`
|
||||
|
||||
@@ -47,6 +47,21 @@ The memory research snapshot contains:
|
||||
|
||||
The strongest current conclusion is that recent progress is concentrated in better diagnosis/evaluation and task-specific execution-state memory, not a universal memory architecture. Read `research/memory/findings.md` before making memory design claims.
|
||||
|
||||
The Agent completion/verification research snapshot contains:
|
||||
|
||||
| Stage | Count |
|
||||
| --- | ---: |
|
||||
| full-text problem-driven pool | 40 |
|
||||
| papers promoted into the evidence ledger | 37 |
|
||||
| historical context only | 3 |
|
||||
|
||||
Its strongest conclusion is that Agent completion is an evidence contract over intended effects,
|
||||
forbidden effects, required procedure, and remaining uncertainty. State checks, trace rules, learned
|
||||
judges, and humans have different jobs; none is a universal oracle. Verification and recovery are
|
||||
separate capabilities, and recovery must distinguish detection, localization, routing, restoration,
|
||||
and re-verification. Read `research/completion-verification/findings.md` before extending the Agent
|
||||
evaluation contract.
|
||||
|
||||
## Artifact Map
|
||||
|
||||
| Area | Source of truth | What it contains |
|
||||
@@ -62,13 +77,13 @@ The strongest current conclusion is that recent progress is concentrated in bett
|
||||
| industry | `industry/` | official reports, blogs, products, benchmarks |
|
||||
| collection runs | `collection-runs/` | queries, scope, counts, method notes |
|
||||
| collection tools | `tools/collection/` | item creation, arXiv collection, promotion, indexing, URL checks |
|
||||
| research | `research/memory/` | field findings and evidence ledger |
|
||||
| research | `research/memory/`, `research/completion-verification/` | field findings and evidence ledgers |
|
||||
| Agent evaluation | `research/evaluation/` | executable design contract for task suites, validators, paired comparisons, and release gates |
|
||||
| evaluation data/tools | `data/evaluation/`, `tools/evaluation/` | versioned task contracts and JSONL result scorer |
|
||||
| evaluation report | `evaluation-site/` | Chinese interactive explanation of the contract, decision gates, tasks, and artifacts |
|
||||
| research data | `data/research/` | committed corpus, screening, landscape and CSV ledger |
|
||||
| research tools | `tools/research/` | paginated search, abstract fetch, screening, classification, landscape build |
|
||||
| experiments | `experiments/knowledge-compilation/` | failed pilot and completed field audit |
|
||||
| experiments | `experiments/knowledge-compilation/` | failed pilot and completed full-text research audits |
|
||||
| learning prototype | `learning/agent-memory/` | preserved failed 12-paper learning module |
|
||||
| web application | `web/` | Atlas, search agent, paper reader/chat, Ollama actions |
|
||||
| project integrity | `tools/check_project.py` | offline structural and data checks |
|
||||
@@ -99,6 +114,20 @@ paginated arXiv queries + local memory-tagged papers
|
||||
|
||||
Do not merge these two lines silently. The first is a broad browsing corpus; the second is an evidence audit with explicit boundaries.
|
||||
|
||||
The Agent completion audit follows the same evidence discipline without a model-assisted screening
|
||||
stage:
|
||||
|
||||
```text
|
||||
problem-driven arXiv candidates
|
||||
-> 40 downloaded and text-located full papers
|
||||
-> 37 audited experiment/ablation/limitation records
|
||||
-> research/completion-verification/evidence-ledger.md
|
||||
-> research/completion-verification/findings.md
|
||||
```
|
||||
|
||||
The PDF and extracted-text working set was temporary. The committed evidence ledger preserves source
|
||||
IDs, key numbers, claim boundaries, and exclusions so another researcher can refetch the sources.
|
||||
|
||||
The Agent evaluation design follows a third line:
|
||||
|
||||
```text
|
||||
@@ -111,7 +140,10 @@ zk-data-agent runtime/events + Zero decision-loop constraints
|
||||
```
|
||||
|
||||
Only the task contracts and scorer exist in this repository. The reusable runner, fixtures, validators,
|
||||
and real-model comparison still need to be implemented in `zk-data-agent`.
|
||||
and real-model comparison still need to be implemented in `zk-data-agent`. That implementation is
|
||||
intentionally paused while the research findings are reviewed. Before implementation, task contracts
|
||||
need separate outcome/invariant/procedure/semantic-residual layers, non-binary
|
||||
`ambiguous/unverifiable` states, and explicit recovery-stage fields.
|
||||
|
||||
`evaluation-site/` visualizes this contract but does not add evidence. Its adjustable decision values are
|
||||
explicitly labeled as a method simulation, not a real benchmark.
|
||||
|
||||
@@ -260,3 +260,41 @@ The report was deployed as immutable image
|
||||
`docker.k1412.top/wuyang/agent-eval:20260727T084425Z-cd1eacb` through Unraid Compose Manager on NAS port
|
||||
12005. NPM route and Let's Encrypt HTTPS were enabled for `https://agent-eval.k1412.top/`, and the site
|
||||
was registered in the k1412 homepage `projects` category.
|
||||
|
||||
## 13. Stop Implementing, Research Completion First
|
||||
|
||||
After reviewing the evaluation report, the user rejected the work as useful research. The failure was
|
||||
not that the site lacked another chart. The design had moved from code inspection to task contracts,
|
||||
scoring, and presentation without first establishing what current Agent research can actually prove
|
||||
about completion and recovery.
|
||||
|
||||
The user explicitly paused experiments and requested general conclusions before any new web form.
|
||||
The project therefore audited a problem-driven 40-paper full-text pool spanning false success, state
|
||||
and procedure verification, evaluator validity, test adequacy, tool failure, root-cause attribution,
|
||||
recovery routing, and stopping. Thirty-seven papers entered an evidence ledger; three remained
|
||||
historical context.
|
||||
|
||||
The main corrections are:
|
||||
|
||||
- task completion is an evidence contract, not an Agent statement or one benchmark bit;
|
||||
- concrete capabilities include state change, side-effect control, procedure compliance, coordination,
|
||||
failure detection, localization, recovery routing, restoration, re-verification, and repeated
|
||||
reliability;
|
||||
- deterministic state checks, trace rules, learned judges, and humans have different responsibilities;
|
||||
- verification only creates an explicit failure; it does not provide diagnosis or recovery;
|
||||
- fixed retry, reflection, replanning, and repair loops all have documented regressions;
|
||||
- the benchmark protocol and verifier must themselves be audited for weak tests, leakage, drift,
|
||||
model changes, and reward hacking.
|
||||
|
||||
The cross-paper synthesis proposes a completion certificate, structured failure object, and
|
||||
state-dependent recovery router. These are marked as system inferences, not experimental results.
|
||||
|
||||
Decision retained: do not implement the Agent evaluation runner or redesign its report until the user
|
||||
accepts the research decomposition. If implementation resumes, rewrite the task contracts around
|
||||
outcome, invariant, procedure, semantic residual, uncertainty, and explicit recovery stages.
|
||||
|
||||
Current source of truth:
|
||||
|
||||
- `research/completion-verification/findings.md`
|
||||
- `research/completion-verification/evidence-ledger.md`
|
||||
- `experiments/knowledge-compilation/2026-07-27-agent-completion-research-audit.md`
|
||||
|
||||
+49
-14
@@ -2,7 +2,29 @@
|
||||
|
||||
last_reviewed: 2026-07-27
|
||||
|
||||
## P0: Prove the Paper Compiler
|
||||
## P0: Challenge the Completion Research
|
||||
|
||||
The immediate work is not an Agent experiment and not a new evaluation page. Review
|
||||
`research/completion-verification/findings.md` with the user and determine whether its problem
|
||||
decomposition is actually useful.
|
||||
|
||||
The review should test whether the research can answer, without falling back to paper titles:
|
||||
|
||||
1. what concrete capabilities an Agent evaluation is measuring;
|
||||
2. what evidence is sufficient to claim completion in different task types;
|
||||
3. where deterministic checks, trace rules, learned judges, and humans fail;
|
||||
4. why failure detection, diagnosis, routing, restoration, and re-verification differ;
|
||||
5. what has genuinely improved since reflection/state-only evaluation, and what remains unsolved.
|
||||
|
||||
For every challenged conclusion, return to the full-text evidence ledger, add a counterexample or
|
||||
narrow the scope. Do not defend the synthesis by adding prose. The acceptance condition is that the
|
||||
user can use the model to diagnose an unfamiliar Agent design and identify what its claimed result
|
||||
does not prove.
|
||||
|
||||
Only after this review should the candidate completion certificate, failure object, and recovery router
|
||||
be translated into task contracts. They are currently system inferences, not validated implementations.
|
||||
|
||||
## P1: Prove the Paper Compiler
|
||||
|
||||
The main project goal is now a high-efficiency knowledge funnel, not a paper browser and not a fixed
|
||||
course. It should let one person process a corpus on the scale of 1,000 papers while retaining the
|
||||
@@ -95,12 +117,20 @@ The first vertical slice is successful only if:
|
||||
|
||||
The frontend is optional. JSON/Markdown plus a rigorous review session is enough for the first proof.
|
||||
|
||||
## P1: Implement the Agent Evaluation Runner
|
||||
## P2: Implement the Agent Evaluation Runner
|
||||
|
||||
The design contract is complete at
|
||||
The first design contract exists at
|
||||
`research/evaluation/k1412-agent-evaluation-v1.md`. Implementation belongs in
|
||||
`https://git.k1412.top/wuyang/zk-data-agent.git`.
|
||||
|
||||
Implementation is paused. Before resuming, revise the contract using the completion research:
|
||||
|
||||
- separate outcome, forbidden-effect invariant, required procedure, and semantic residual;
|
||||
- add `ambiguous` and `unverifiable` result states;
|
||||
- model mutating actions, idempotency, rollback and compensation;
|
||||
- record detection, localization, routing, restoration and re-verification separately;
|
||||
- document validator coverage and known false-positive/false-negative boundaries.
|
||||
|
||||
Implement in this order:
|
||||
|
||||
1. extract a reusable runner from `scripts/eval-live-work.py`;
|
||||
@@ -131,7 +161,7 @@ The report at `https://agent-eval.k1412.top/` explains the contract and simulate
|
||||
It is not evidence that any Agent variant passed. Future result views must consume the same validator
|
||||
JSONL instead of embedding numbers in frontend code.
|
||||
|
||||
## P1: Continue Collection Through Questions
|
||||
## P2: Continue Collection Through Questions
|
||||
|
||||
The corpus now contains 1,125 papers, 1,110 of them queued. More collection is useful only when paired
|
||||
with processing:
|
||||
@@ -163,6 +193,11 @@ workflow:
|
||||
Avoid another graph or dashboard redesign until either the paper compiler or evaluation runner produces
|
||||
something the user repeatedly uses.
|
||||
|
||||
For the completion research, a later web presentation should begin with a concrete claim or failure,
|
||||
then progressively reveal the completion contract, evidence, conflicting evaluator results, recovery
|
||||
branches and source papers. It should not lead with a benchmark hierarchy, architecture diagram,
|
||||
leaderboard or aggregate score.
|
||||
|
||||
## P2: Operations
|
||||
|
||||
Still missing:
|
||||
@@ -187,16 +222,16 @@ Still missing:
|
||||
|
||||
For the knowledge-base repository:
|
||||
|
||||
1. choose one bounded topic and enumerate the complete relevant corpus;
|
||||
2. define the fingerprint schema and evidence locators;
|
||||
3. compile and audit a representative batch;
|
||||
4. test whether research-move compression preserves important distinctions;
|
||||
5. run the result with the user before building UI.
|
||||
1. read `research/completion-verification/findings.md` with the user;
|
||||
2. collect objections, ambiguous terms and missing counterexamples;
|
||||
3. tighten or reject conclusions against the evidence ledger;
|
||||
4. only then convert the accepted model into a minimal task-contract schema;
|
||||
5. design the progressive web presentation after the knowledge shape is stable.
|
||||
|
||||
For `zk-data-agent`:
|
||||
|
||||
1. port the two currently runnable task contracts;
|
||||
2. extract the live runner and add one deterministic validator;
|
||||
3. export valid result JSONL;
|
||||
4. run a smoke comparison only after deterministic layers pass;
|
||||
5. record the decision and failures, not just the score.
|
||||
1. do not port or run tasks yet;
|
||||
2. wait for the completion-research review;
|
||||
3. revise the v1 contract before extracting a runner;
|
||||
4. retain the existing code inspection and scorer as historical implementation inputs;
|
||||
5. never present the current evaluation report as benchmark evidence.
|
||||
|
||||
Reference in New Issue
Block a user