Refresh papers and define Agent evaluation

docs/04-evaluation-observability.md

@@ -74,3 +74,21 @@ follow_ups:
 - 是否错过了更便宜或更稳定的路径？
 - 是否产生了不可恢复的副作用？
 - 失败时，日志是否足够定位原因？
+
+## Versioned Evaluation Contract
+
+真实 Agent 评测还必须固定 task/fixture、模型、系统 prompt、工具 schema、工作区镜像、
+上下文和预算。最终成功由独立 validator 判断，而不是由 Agent 的完成声明判断。
+
+对 baseline 和 variant 做配对比较时，至少分开报告：
+
+- baseline fail -> variant pass: gain；
+- baseline pass -> variant fail: regression；
+- both pass；
+- both fail；
+- provider/infrastructure error。
+
+平均成功率不能掩盖 regressions，安全违规和 false completion 也不能被成本收益抵消。
+
+当前可执行设计见
+[K1412 Agent Evaluation v1](../research/evaluation/k1412-agent-evaluation-v1.md)。

docs/06-project-status.md

@@ -1,7 +1,7 @@
 # Project Status
 
-last_reviewed: 2026-07-12
-baseline_before_handoff: `ab758e7`
+last_reviewed: 2026-07-27
+baseline_before_update: `8e4ff37`
 branch: `main`
 remote: `https://git.k1412.top/wuyang/agent.git`
 
@@ -16,19 +16,23 @@ The long-term product is not “a place to browse papers.” It should help the
 
 ## Current Snapshot
 
-The deterministic index currently contains 989 items:
+The deterministic index currently contains 1,140 items:
 
 | Collection | Count | State |
 | --- | ---: | --- |
-| papers | 974 | 959 queued, 15 skimmed |
+| papers | 1,125 | 1,110 queued, 15 skimmed |
 | jobs | 6 | 4 active, 2 unknown |
 | industry | 9 | 8 analyzed, 1 queued |
 
-The expanded arXiv run saw 1,506 candidates, promoted 970 high-recall candidates, and retained 536 reserve candidates. These are collection counts, not quality or reading-completion counts.
+The 2026-07-08 expanded arXiv run saw 1,506 candidates, promoted 970 high-recall candidates,
+and retained 536 reserve candidates. The 2026-07-27 incremental refresh covered July 9-27, saw
+249 unique candidates, selected 157 high-relevance records, skipped 6 existing papers, and added
+151 paper items. The newest submission returned was 2026-07-24. These are collection counts, not
+quality or reading-completion counts.
 
 The memory topic has two intentionally different counts:
 
-- 296 indexed items have the broad `memory` topic, including non-paper streams;
+- 336 indexed items now have the broad `memory` topic, including non-paper streams;
 - 293 local paper items were resolved for the dedicated memory audit.
 
 The memory research snapshot contains:
@@ -59,6 +63,8 @@ The strongest current conclusion is that recent progress is concentrated in bett
 | collection runs | `collection-runs/` | queries, scope, counts, method notes |
 | collection tools | `tools/collection/` | item creation, arXiv collection, promotion, indexing, URL checks |
 | research | `research/memory/` | field findings and evidence ledger |
+| Agent evaluation | `research/evaluation/` | executable design contract for task suites, validators, paired comparisons, and release gates |
+| evaluation data/tools | `data/evaluation/`, `tools/evaluation/` | versioned task contracts and JSONL result scorer |
 | research data | `data/research/` | committed corpus, screening, landscape and CSV ledger |
 | research tools | `tools/research/` | paginated search, abstract fetch, screening, classification, landscape build |
 | experiments | `experiments/knowledge-compilation/` | failed pilot and completed field audit |
@@ -92,6 +98,20 @@ paginated arXiv queries + local memory-tagged papers
 
 Do not merge these two lines silently. The first is a broad browsing corpus; the second is an evidence audit with explicit boundaries.
 
+The Agent evaluation design follows a third line:
+
+```text
+zk-data-agent runtime/events + Zero decision-loop constraints
+  -> versioned task contracts
+  -> isolated repeated baseline/variant runs
+  -> independent validators + trace rules
+  -> JSONL result records
+  -> gain/regression/safety decision
+```
+
+Only the task contracts and scorer exist in this repository. The reusable runner, fixtures, validators,
+and real-model comparison still need to be implemented in `zk-data-agent`.
+
 ## Web Application
 
 Capabilities currently implemented:
@@ -121,7 +141,11 @@ public mapping:  https://lab.k1412.top/
 health:          /api/health
 ```
 
-At the beginning of the 2026-07-12 handoff audit, no process was listening on 18080 and the public domain returned 502. The handoff added and enabled a user-level systemd service through `web/manage.sh`; local and public health then returned 200 with 974 papers and `ollama_ready: true`. Runtime status is still external state and must always be checked live.
+At the beginning of the 2026-07-12 handoff audit, no process was listening on 18080 and the public domain returned 502. The handoff added and enabled a user-level systemd service through `web/manage.sh`;
+local and public health then returned 200. Runtime status is external state and must always be checked
+live. After a corpus update, restart the service because the index is loaded at process startup.
+On 2026-07-27 both the Tailscale endpoint and `https://lab.k1412.top/api/health` returned the refreshed
+1,125-paper count with `ollama_ready: true`.
 
 The live Ollama inventory checked on 2026-07-12 included `ChatGPT-5.6:Luna`, `Terra`, `Sol`, `auto`, `embed`, and `gpt-4o:latest`. The web defaults were updated from stale `light/fast/large` tags to configurable Luna/Terra/Sol tiers.
 
@@ -144,6 +168,7 @@ Offline validation:
 
 ```bash
 python3 tools/check_project.py
+python3 -m unittest tools.evaluation.test_score_agent_runs
 git diff --check
 ```
 

docs/07-exploration-history.md

@@ -1,6 +1,6 @@
 # Exploration History
 
-last_reviewed: 2026-07-12
+last_reviewed: 2026-07-27
 
 This is the decision history, including failed routes. Preserve it so future work does not rediscover the same lessons.
 
@@ -152,8 +152,74 @@ This audit is the current research baseline. Read `research/memory/findings.md`
 
 Relevant commit: `ab758e7`.
 
-## 9. Current Position
+## 9. Position After the Memory Audit
 
 The repository now has a strong collection substrate, a functioning exploration prototype, and one evidence-led field audit. It does not yet have the stateful personal learning/internalization layer that motivated the project.
 
-The next meaningful experiment is not another graph redesign or another bulk paper sweep. It is one end-to-end learning path that records a user's claims, evidence judgments, contradictions, applications, and later recall, using the memory audit as its first evidence base.
+At this point, the proposed next experiment was an end-to-end learning path that records a user's claims,
+evidence judgments, contradictions, applications, and later recall.
+
+## 10. From Learning Page to Knowledge Funnel
+
+The user then made the throughput problem more precise: the target is not a nicer way to read papers one
+by one. It is to preserve as much of an expert's understanding as possible while reducing the time needed
+to process roughly 1,000 papers, ideally to one concentrated day.
+
+The important expert behavior is:
+
+```text
+paper -> what problem / why now / prior method / actual change
+-> evidence / failure boundary / place in the field
+-> compare with neighboring work
+-> notice missing methods and generate new hypotheses
+```
+
+This reframing changes the primary artifact from a course or browser to a paper compiler:
+
+1. extract compact, source-linked paper fingerprints;
+2. collapse repeated work into research moves and lineages;
+3. compare evidence and regressions inside each move;
+4. compile a field grammar that the user can inspect, challenge, and apply;
+5. persist the user's judgments and delayed review only after the compiled knowledge is useful.
+
+Decision retained: the frontend is optional at this stage. The first proof should be an inspectable data
+pipeline and a high-density research artifact, not another navigation surface.
+
+## 11. Incremental Refresh and Agent Evaluation Design
+
+On 2026-07-27 the paper boundary was refreshed from July 9 through July 27:
+
+- 249 unique arXiv candidates;
+- 157 high-relevance selected records;
+- 151 new queued paper items;
+- 1,125 total paper items after indexing.
+
+The arXiv export API timed out and returned 503, so the collector gained transient-error retries,
+configurable timeouts, progress output, and an official search-page fallback. The successful run used
+that fallback. This is an operational improvement, not evidence that the selected papers were read.
+
+The same work inspected `zk-data-agent` at commit `e2e7a7b` and the Zero product flow. The requested
+output was clarified as an Agent evaluation scheme, not a new Agent implementation.
+
+What the target already had:
+
+- isolated real workspaces and layered tests;
+- versioned run events and complete tool lifecycle traces;
+- evidence-gated completion;
+- one real-model live evaluation script.
+
+What it lacked:
+
+- a versioned task suite with fixtures and deterministic validators;
+- repeated paired baseline/variant runs;
+- complete reproducibility manifests;
+- unified failure labels and explicit false-completion checks;
+- separate gain and regression accounting.
+
+The resulting v1 contract lives under `research/evaluation/`, with ten task contracts and a JSONL scorer.
+No real-model comparison was run, and most tasks correctly remain marked as requiring target-repository
+runner work.
+
+Decision retained: evaluation exists to decide whether one controlled Agent change should ship. Do not
+compress safety, false completion, gains, regressions, latency, and cost into one decorative score, and
+do not build a dashboard before the runner and validators exist.

docs/08-next-work.md

@@ -1,167 +1,198 @@
 # Next Work
 
-last_reviewed: 2026-07-12
+last_reviewed: 2026-07-27
 
-## P0: Build One Stateful Learning Path
+## P0: Prove the Paper Compiler
 
-Use Agent Memory as the first vertical slice, but do not reuse the failed 45-minute lecture format.
+The main project goal is now a high-efficiency knowledge funnel, not a paper browser and not a fixed
+course. It should let one person process a corpus on the scale of 1,000 papers while retaining the
+structure an expert normally acquires through repeated reading.
 
-The unit of interaction should be a claim, not a paper. A minimal claim object needs:
+The first proof should use one bounded topic but process the whole relevant corpus, not a hand-picked
+dozen papers.
+
+### Stage A: Paper Fingerprints
+
+Compile each relevant paper into a source-linked record:
 
 ```text
-question
-claim
-supporting evidence
-counterevidence
-confidence
-scope / failure boundary
-prerequisites
-related claims
-user judgment
+paper_id / version / source boundary
+problem and why it matters
+prior method or baseline
+actual delta
+mechanism
+evidence and comparison conditions
+failure boundary
+closest related papers
+candidate research move
+confidence and review status
+```
+
+Use abstracts for high-recall triage only. Claims about mechanism, improvement, ablation, cost, or
+limitations require full-text evidence with a page/table/section locator.
+
+### Stage B: Research Moves and Lineages
+
+Do not ask the user to read 1,000 isolated fingerprints. Collapse them into repeated moves such as:
+
+```text
+retrieve better / write selectively / preserve execution state
+verify before completion / recover after failure / change the environment
+trade generality for task structure / improve benchmark validity
+```
+
+For each move, preserve:
+
+- the problem pressure that caused it;
+- the strongest baseline;
+- what variants actually change;
+- comparable evidence and non-comparable claims;
+- regressions and unsolved boundaries;
+- chronological lineage and independent replications.
+
+### Stage C: Field Grammar
+
+Compile the moves into a small, challengeable model:
+
+```text
+problem pressures -> available moves -> evidence conditions
+-> known trade-offs -> unresolved gaps -> design predictions
+```
+
+The field grammar must answer a new target without relying on paper titles:
+
+1. what class of problem is this;
+2. which moves have been tried;
+3. why they work or fail;
+4. what evidence would distinguish alternatives;
+5. what appears missing and is worth testing.
+
+### Stage D: User Compression
+
+Only after the compiled model is useful, add persistent user state:
+
+```text
+accepted / rejected / uncertain claims
+user rationale
+counterevidence seen
+application attempts
 last reviewed / next review
 ```
 
-The first path should contain a small number of consequential claims from the existing audit, for example:
+Recall is not measured by page completion. The user should be able to reconstruct the field grammar,
+diagnose an unseen design, and update a belief when contrary evidence appears.
 
-- strong retrieval can dominate write complexity on static QA;
-- current-value maintenance is distinct from retrieval;
-- execution-state memory is different from historical recall;
-- automatic experience promotion requires evidence and rollback;
-- memory can have negative marginal value inside an Agent scaffold.
+### P0 Acceptance
 
-The interaction should progressively reveal:
+The first vertical slice is successful only if:
 
-```text
-problem -> current hypothesis -> evidence -> conflicting evidence
--> user judgment -> application to a new design -> delayed recall
-```
+- corpus coverage and exclusions are explicit;
+- every consequential statement links to evidence or is marked as inference;
+- repeated papers are compressed without erasing meaningful disagreement;
+- the user can inspect a paper's position in seconds;
+- the compiled model surfaces at least one non-obvious gap or testable prediction;
+- a timed user session shows better diagnosis than raw title/abstract browsing.
 
-Persist the user's state. Merely rendering these sections is not enough.
+The frontend is optional. JSON/Markdown plus a rigorous review session is enough for the first proof.
 
-### P0 Data Work
+## P1: Implement the Agent Evaluation Runner
 
-Define a versioned, inspectable local schema before adding a database. Suggested initial files:
+The design contract is complete at
+`research/evaluation/k1412-agent-evaluation-v1.md`. Implementation belongs in
+`https://git.k1412.top/wuyang/zk-data-agent.git`.
 
-```text
-data/learning/claims.json
-data/learning/user-model.json
-data/learning/reviews.jsonl
-```
+Implement in this order:
 
-Requirements:
+1. extract a reusable runner from `scripts/eval-live-work.py`;
+2. load versioned task JSON, workspace fixtures, budgets, and deterministic validators;
+3. export complete run manifests, sanitized traces, and result JSONL;
+4. run paired baseline/variant repetitions in randomized order;
+5. add fault injection, multi-turn context, memory, and delegation task adapters;
+6. aggregate with `tools/evaluation/score_agent_runs.py`;
+7. inspect gains, regressions, both-fail, false completions, and safety events before deciding.
 
-- every claim links back to evidence-ledger entries;
-- user notes and AI synthesis remain distinguishable;
-- claim updates preserve history;
-- confidence changes record a reason;
-- review records are append-only;
-- no generated summary can silently replace source evidence.
+Start with the two task contracts marked `runnable-with-current-live-script`. Do not claim that the
+remaining contracts run until their listed runner requirements exist.
 
-### P0 AI Roles
+### Evaluation Release Boundary
 
-Use AI above the per-paper level:
+A candidate must not ship if it introduces:
 
-- evidence auditor: check whether a claim is actually supported;
-- contradiction finder: surface papers that challenge the current model;
-- synthesis maintainer: update the claim network when evidence changes;
-- research partner: help the user reason through an unfamiliar target;
-- review generator: create recall/application prompts from validated claims.
+- a safety or isolation violation;
+- more false completions;
+- an unrecorded model, prompt, tool, fixture, context, or budget change;
+- a critical-family regression hidden by average success;
+- gains that disappear under independent validation.
 
-Use Luna for cheap classification, Terra for retrieval briefs and bounded dialogue, and Sol only for explicit synthesis/critique. The user also has a company GPT-5.5 data-agent API with effectively unlimited tokens, but integration must wait until its endpoint, auth, rate behavior, and data policy are documented.
+No live benchmark has been run yet. The latest evaluation papers are only abstract-level candidates and
+need full-text audit before they can change the contract.
 
-### P0 Validation
+## P1: Continue Collection Through Questions
 
-Do not judge success by page completion or time spent. Validate whether the user can:
+The corpus now contains 1,125 papers, 1,110 of them queued. More collection is useful only when paired
+with processing:
 
-1. explain five claims without paper names;
-2. identify the evidence boundary of each claim;
-3. diagnose a new Agent memory design;
-4. predict when memory will hurt;
-5. recover the model after 1, 3, and 7 days;
-6. change a belief when presented with counterevidence.
+- run a narrow weekly increment from the last explicit date boundary;
+- retain broad candidates and truthful status labels;
+- prioritize papers that challenge a current research move or evaluation assumption;
+- audit the latest evaluation cluster first;
+- refresh Beijing jobs and official industry practice separately.
 
-## P1: Connect Research to the Frontend
-
-The current Atlas/search/reader can remain, but it should feed the stateful path:
-
-- selecting a topic opens its unresolved questions and claims, not only a paper list;
-- selecting a paper shows which claims it supports or challenges;
-- search results can be promoted into an evidence review queue;
-- paper chat can create a candidate claim, never an automatically accepted fact;
-- the primary desktop focus should shift to the active research/learning task;
-- mobile should preserve the same conceptual flow with fewer simultaneous panes.
-
-Avoid another navigation-heavy redesign before the data and learning loop exist.
-
-## P1: Continue Collection Selectively
-
-Current gaps:
-
-- only 6 JD items, last checked 2026-07-08;
-- only 9 industry items;
-- 959 of 974 paper items are still queued;
-- paper source approval and automated collection policy need an explicit user decision;
-- the corpus is heavily biased toward May-July 2026 and broad arXiv query terms.
-
-Next collection work should be question-driven:
-
-- refresh Beijing DeepSeek/Tencent/ByteDance/Alibaba/Baidu roles and record changes;
-- add official company material that explains memory, evaluation, coding agents, or computer use;
-- deepen only papers that challenge or extend current claims;
-- do not add another thousand papers without a processing and evidence plan.
-
-After any item changes:
+After item changes:
 
 ```bash
 python3 tools/collection/build_index.py
 python3 tools/check_project.py
 ```
 
-## P2: Research Refresh Protocol
+## P2: Productize Only Validated Workflows
 
-When refreshing the memory audit or starting another topic:
+The Atlas/search/reader can remain as an orientation surface. New frontend work should follow a proven
+workflow:
 
-1. define exact date windows and approved sources;
-2. save the broad candidate snapshot;
-3. separate deterministic filters from model triage;
-4. identify comparable benchmarks and controlled studies;
-5. read experiments, ablations, costs, and limitations in full text;
-6. update an evidence ledger before writing conclusions;
-7. preserve negative results and failed hypotheses;
-8. record the run under `experiments/` and update project status.
+- a topic opens unresolved questions and compiled moves, not only a paper list;
+- a paper shows the move it supports, challenges, or merely resembles;
+- the primary view is the active research decision;
+- candidate claims require evidence review before becoming durable knowledge;
+- user judgments and review history remain inspectable and reversible.
 
-Do not compare raw scores across papers unless model, embedding, prompt, visible information, and budget are controlled.
+Avoid another graph or dashboard redesign until either the paper compiler or evaluation runner produces
+something the user repeatedly uses.
 
 ## P2: Operations
 
 Still missing:
 
-- public-domain deployment documentation outside this repository;
-- authentication/authorization for a public or multi-user site;
-- automated backup for future personal learning state;
-- CI that runs `tools/check_project.py` on push;
-- tests for web API behavior and frontend flows.
-
-The repository now has an enabled user-level systemd service managed through `web/manage.sh`. It restarts on failure and survives user sessions because linger is enabled on the current host, but it is not a complete production deployment: proxy/TLS/auth configuration remains external.
+- CI for `tools/check_project.py` and unit tests;
+- authentication and authorization for any public multi-user state;
+- automated backup for future personal learning data;
+- repository-managed documentation of proxy/TLS deployment;
+- web API and frontend flow tests.
 
 ## Explicit Non-Goals
 
+- Do not present 1,110 queued papers as understood knowledge.
+- Do not revive the 12-paper lecture as a field synthesis.
+- Do not use per-paper summaries as the final knowledge unit.
+- Do not build a leaderboard or single composite Agent score.
+- Do not let an Agent or the same judge model certify its own success.
+- Do not hide regressions behind average improvement.
 - Do not make the graph more decorative as the next milestone.
-- Do not present 974 queued papers as knowledge already learned.
-- Do not revive the 12-paper lecture as the primary path.
-- Do not use per-paper summary/translation as the main AI value proposition.
-- Do not let AI write durable memory solely because a statement repeats in traces.
-- Do not hide model, embedding, context, or cost changes inside an architecture comparison.
 
-## First Continuation Task
+## First Continuation Tasks
 
-The best next task for another AI is:
+For the knowledge-base repository:
 
-1. read the memory findings and evidence ledger;
-2. define the claim/user-state/review schemas;
-3. compile 5-8 audited claims into a stateful local dataset;
-4. implement one progressive learning/research view;
-5. run it with the user and record what changes in their understanding;
-6. keep or discard the interaction model based on that evidence.
+1. choose one bounded topic and enumerate the complete relevant corpus;
+2. define the fingerprint schema and evidence locators;
+3. compile and audit a representative batch;
+4. test whether research-move compression preserves important distinctions;
+5. run the result with the user before building UI.
 
-That is the shortest path from the current corpus browser toward the actual goal.
+For `zk-data-agent`:
+
+1. port the two currently runnable task contracts;
+2. extract the live runner and add one deterministic validator;
+3. export valid result JSONL;
+4. run a smoke comparison only after deterministic layers pass;
+5. record the decision and failures, not just the score.