agent/docs/08-next-work.md

# Next Work

last_reviewed: 2026-07-27

## P0: Challenge the Completion Research

The immediate work is not an Agent experiment and not a new evaluation page. Review
`research/completion-verification/findings.md` with the user and determine whether its problem
decomposition is actually useful.

The review should test whether the research can answer, without falling back to paper titles:

1. what concrete capabilities an Agent evaluation is measuring;
2. what evidence is sufficient to claim completion in different task types;
3. where deterministic checks, trace rules, learned judges, and humans fail;
4. why failure detection, diagnosis, routing, restoration, and re-verification differ;
5. what has genuinely improved since reflection/state-only evaluation, and what remains unsolved.

For every challenged conclusion, return to the full-text evidence ledger, add a counterexample or
narrow the scope. Do not defend the synthesis by adding prose. The acceptance condition is that the
user can use the model to diagnose an unfamiliar Agent design and identify what its claimed result
does not prove.

Only after this review should the candidate completion certificate, failure object, and recovery router
be translated into task contracts. They are currently system inferences, not validated implementations.

## P1: Prove the Paper Compiler

The main project goal is now a high-efficiency knowledge funnel, not a paper browser and not a fixed
course. It should let one person process a corpus on the scale of 1,000 papers while retaining the
structure an expert normally acquires through repeated reading.

The first proof should use one bounded topic but process the whole relevant corpus, not a hand-picked
dozen papers.

### Stage A: Paper Fingerprints

Compile each relevant paper into a source-linked record:

```text
paper_id / version / source boundary
problem and why it matters
prior method or baseline
actual delta
mechanism
evidence and comparison conditions
failure boundary
closest related papers
candidate research move
confidence and review status
```

Use abstracts for high-recall triage only. Claims about mechanism, improvement, ablation, cost, or
limitations require full-text evidence with a page/table/section locator.

### Stage B: Research Moves and Lineages

Do not ask the user to read 1,000 isolated fingerprints. Collapse them into repeated moves such as:

```text
retrieve better / write selectively / preserve execution state
verify before completion / recover after failure / change the environment
trade generality for task structure / improve benchmark validity
```

For each move, preserve:

- the problem pressure that caused it;
- the strongest baseline;
- what variants actually change;
- comparable evidence and non-comparable claims;
- regressions and unsolved boundaries;
- chronological lineage and independent replications.

### Stage C: Field Grammar

Compile the moves into a small, challengeable model:

```text
problem pressures -> available moves -> evidence conditions
-> known trade-offs -> unresolved gaps -> design predictions
```

The field grammar must answer a new target without relying on paper titles:

1. what class of problem is this;
2. which moves have been tried;
3. why they work or fail;
4. what evidence would distinguish alternatives;
5. what appears missing and is worth testing.

### Stage D: User Compression

Only after the compiled model is useful, add persistent user state:

```text
accepted / rejected / uncertain claims
user rationale
counterevidence seen
application attempts
last reviewed / next review
```

Recall is not measured by page completion. The user should be able to reconstruct the field grammar,
diagnose an unseen design, and update a belief when contrary evidence appears.

### P0 Acceptance

The first vertical slice is successful only if:

- corpus coverage and exclusions are explicit;
- every consequential statement links to evidence or is marked as inference;
- repeated papers are compressed without erasing meaningful disagreement;
- the user can inspect a paper's position in seconds;
- the compiled model surfaces at least one non-obvious gap or testable prediction;
- a timed user session shows better diagnosis than raw title/abstract browsing.

The frontend is optional. JSON/Markdown plus a rigorous review session is enough for the first proof.

## P2: Implement the Agent Evaluation Runner

The first design contract exists at
`research/evaluation/k1412-agent-evaluation-v1.md`. Implementation belongs in
`https://git.k1412.top/wuyang/zk-data-agent.git`.

Implementation is paused. Before resuming, revise the contract using the completion research:

- separate outcome, forbidden-effect invariant, required procedure, and semantic residual;
- add `ambiguous` and `unverifiable` result states;
- model mutating actions, idempotency, rollback and compensation;
- record detection, localization, routing, restoration and re-verification separately;
- document validator coverage and known false-positive/false-negative boundaries.

Implement in this order:

1. extract a reusable runner from `scripts/eval-live-work.py`;
2. load versioned task JSON, workspace fixtures, budgets, and deterministic validators;
3. export complete run manifests, sanitized traces, and result JSONL;
4. run paired baseline/variant repetitions in randomized order;
5. add fault injection, multi-turn context, memory, and delegation task adapters;
6. aggregate with `tools/evaluation/score_agent_runs.py`;
7. inspect gains, regressions, both-fail, false completions, and safety events before deciding.

Start with the two task contracts marked `runnable-with-current-live-script`. Do not claim that the
remaining contracts run until their listed runner requirements exist.

### Evaluation Release Boundary

A candidate must not ship if it introduces:

- a safety or isolation violation;
- more false completions;
- an unrecorded model, prompt, tool, fixture, context, or budget change;
- a critical-family regression hidden by average success;
- gains that disappear under independent validation.

No live benchmark has been run yet. The latest evaluation papers are only abstract-level candidates and
need full-text audit before they can change the contract.

The report at `https://agent-eval.k1412.top/` explains the contract and simulates its decision rules.
It is not evidence that any Agent variant passed. Future result views must consume the same validator
JSONL instead of embedding numbers in frontend code.

## P2: Continue Collection Through Questions

The corpus now contains 1,125 papers, 1,110 of them queued. More collection is useful only when paired
with processing:

- run a narrow weekly increment from the last explicit date boundary;
- retain broad candidates and truthful status labels;
- prioritize papers that challenge a current research move or evaluation assumption;
- audit the latest evaluation cluster first;
- refresh Beijing jobs and official industry practice separately.

After item changes:

```bash
python3 tools/collection/build_index.py
python3 tools/check_project.py
```

## P2: Productize Only Validated Workflows

The Atlas/search/reader can remain as an orientation surface. New frontend work should follow a proven
workflow:

- a topic opens unresolved questions and compiled moves, not only a paper list;
- a paper shows the move it supports, challenges, or merely resembles;
- the primary view is the active research decision;
- candidate claims require evidence review before becoming durable knowledge;
- user judgments and review history remain inspectable and reversible.

Avoid another graph or dashboard redesign until either the paper compiler or evaluation runner produces
something the user repeatedly uses.

For the completion research, a later web presentation should begin with a concrete claim or failure,
then progressively reveal the completion contract, evidence, conflicting evaluator results, recovery
branches and source papers. It should not lead with a benchmark hierarchy, architecture diagram,
leaderboard or aggregate score.

## P2: Operations

Still missing:

- CI for `tools/check_project.py` and unit tests;
- authentication and authorization for any public multi-user state;
- automated backup for future personal learning data;
- repository-managed documentation of proxy/TLS deployment;
- web API and frontend flow tests.

## Explicit Non-Goals

- Do not present 1,110 queued papers as understood knowledge.
- Do not revive the 12-paper lecture as a field synthesis.
- Do not use per-paper summaries as the final knowledge unit.
- Do not build a leaderboard or single composite Agent score.
- Do not let an Agent or the same judge model certify its own success.
- Do not hide regressions behind average improvement.
- Do not make the graph more decorative as the next milestone.

## First Continuation Tasks

For the knowledge-base repository:

1. read `research/completion-verification/findings.md` with the user;
2. collect objections, ambiguous terms and missing counterexamples;
3. tighten or reject conclusions against the evidence ledger;
4. only then convert the accepted model into a minimal task-contract schema;
5. design the progressive web presentation after the knowledge shape is stable.

For `zk-data-agent`:

1. do not port or run tasks yet;
2. wait for the completion-research review;
3. revise the v1 contract before extracting a runner;
4. retain the existing code inspection and scorer as historical implementation inputs;
5. never present the current evaluation report as benchmark evidence.