Refresh papers and define Agent evaluation
This commit is contained in:
wuyang
+149
-118
@@ -1,167 +1,198 @@
|
||||
# Next Work
|
||||
|
||||
last_reviewed: 2026-07-12
|
||||
last_reviewed: 2026-07-27
|
||||
|
||||
## P0: Build One Stateful Learning Path
|
||||
## P0: Prove the Paper Compiler
|
||||
|
||||
Use Agent Memory as the first vertical slice, but do not reuse the failed 45-minute lecture format.
|
||||
The main project goal is now a high-efficiency knowledge funnel, not a paper browser and not a fixed
|
||||
course. It should let one person process a corpus on the scale of 1,000 papers while retaining the
|
||||
structure an expert normally acquires through repeated reading.
|
||||
|
||||
The unit of interaction should be a claim, not a paper. A minimal claim object needs:
|
||||
The first proof should use one bounded topic but process the whole relevant corpus, not a hand-picked
|
||||
dozen papers.
|
||||
|
||||
### Stage A: Paper Fingerprints
|
||||
|
||||
Compile each relevant paper into a source-linked record:
|
||||
|
||||
```text
|
||||
question
|
||||
claim
|
||||
supporting evidence
|
||||
counterevidence
|
||||
confidence
|
||||
scope / failure boundary
|
||||
prerequisites
|
||||
related claims
|
||||
user judgment
|
||||
paper_id / version / source boundary
|
||||
problem and why it matters
|
||||
prior method or baseline
|
||||
actual delta
|
||||
mechanism
|
||||
evidence and comparison conditions
|
||||
failure boundary
|
||||
closest related papers
|
||||
candidate research move
|
||||
confidence and review status
|
||||
```
|
||||
|
||||
Use abstracts for high-recall triage only. Claims about mechanism, improvement, ablation, cost, or
|
||||
limitations require full-text evidence with a page/table/section locator.
|
||||
|
||||
### Stage B: Research Moves and Lineages
|
||||
|
||||
Do not ask the user to read 1,000 isolated fingerprints. Collapse them into repeated moves such as:
|
||||
|
||||
```text
|
||||
retrieve better / write selectively / preserve execution state
|
||||
verify before completion / recover after failure / change the environment
|
||||
trade generality for task structure / improve benchmark validity
|
||||
```
|
||||
|
||||
For each move, preserve:
|
||||
|
||||
- the problem pressure that caused it;
|
||||
- the strongest baseline;
|
||||
- what variants actually change;
|
||||
- comparable evidence and non-comparable claims;
|
||||
- regressions and unsolved boundaries;
|
||||
- chronological lineage and independent replications.
|
||||
|
||||
### Stage C: Field Grammar
|
||||
|
||||
Compile the moves into a small, challengeable model:
|
||||
|
||||
```text
|
||||
problem pressures -> available moves -> evidence conditions
|
||||
-> known trade-offs -> unresolved gaps -> design predictions
|
||||
```
|
||||
|
||||
The field grammar must answer a new target without relying on paper titles:
|
||||
|
||||
1. what class of problem is this;
|
||||
2. which moves have been tried;
|
||||
3. why they work or fail;
|
||||
4. what evidence would distinguish alternatives;
|
||||
5. what appears missing and is worth testing.
|
||||
|
||||
### Stage D: User Compression
|
||||
|
||||
Only after the compiled model is useful, add persistent user state:
|
||||
|
||||
```text
|
||||
accepted / rejected / uncertain claims
|
||||
user rationale
|
||||
counterevidence seen
|
||||
application attempts
|
||||
last reviewed / next review
|
||||
```
|
||||
|
||||
The first path should contain a small number of consequential claims from the existing audit, for example:
|
||||
Recall is not measured by page completion. The user should be able to reconstruct the field grammar,
|
||||
diagnose an unseen design, and update a belief when contrary evidence appears.
|
||||
|
||||
- strong retrieval can dominate write complexity on static QA;
|
||||
- current-value maintenance is distinct from retrieval;
|
||||
- execution-state memory is different from historical recall;
|
||||
- automatic experience promotion requires evidence and rollback;
|
||||
- memory can have negative marginal value inside an Agent scaffold.
|
||||
### P0 Acceptance
|
||||
|
||||
The interaction should progressively reveal:
|
||||
The first vertical slice is successful only if:
|
||||
|
||||
```text
|
||||
problem -> current hypothesis -> evidence -> conflicting evidence
|
||||
-> user judgment -> application to a new design -> delayed recall
|
||||
```
|
||||
- corpus coverage and exclusions are explicit;
|
||||
- every consequential statement links to evidence or is marked as inference;
|
||||
- repeated papers are compressed without erasing meaningful disagreement;
|
||||
- the user can inspect a paper's position in seconds;
|
||||
- the compiled model surfaces at least one non-obvious gap or testable prediction;
|
||||
- a timed user session shows better diagnosis than raw title/abstract browsing.
|
||||
|
||||
Persist the user's state. Merely rendering these sections is not enough.
|
||||
The frontend is optional. JSON/Markdown plus a rigorous review session is enough for the first proof.
|
||||
|
||||
### P0 Data Work
|
||||
## P1: Implement the Agent Evaluation Runner
|
||||
|
||||
Define a versioned, inspectable local schema before adding a database. Suggested initial files:
|
||||
The design contract is complete at
|
||||
`research/evaluation/k1412-agent-evaluation-v1.md`. Implementation belongs in
|
||||
`https://git.k1412.top/wuyang/zk-data-agent.git`.
|
||||
|
||||
```text
|
||||
data/learning/claims.json
|
||||
data/learning/user-model.json
|
||||
data/learning/reviews.jsonl
|
||||
```
|
||||
Implement in this order:
|
||||
|
||||
Requirements:
|
||||
1. extract a reusable runner from `scripts/eval-live-work.py`;
|
||||
2. load versioned task JSON, workspace fixtures, budgets, and deterministic validators;
|
||||
3. export complete run manifests, sanitized traces, and result JSONL;
|
||||
4. run paired baseline/variant repetitions in randomized order;
|
||||
5. add fault injection, multi-turn context, memory, and delegation task adapters;
|
||||
6. aggregate with `tools/evaluation/score_agent_runs.py`;
|
||||
7. inspect gains, regressions, both-fail, false completions, and safety events before deciding.
|
||||
|
||||
- every claim links back to evidence-ledger entries;
|
||||
- user notes and AI synthesis remain distinguishable;
|
||||
- claim updates preserve history;
|
||||
- confidence changes record a reason;
|
||||
- review records are append-only;
|
||||
- no generated summary can silently replace source evidence.
|
||||
Start with the two task contracts marked `runnable-with-current-live-script`. Do not claim that the
|
||||
remaining contracts run until their listed runner requirements exist.
|
||||
|
||||
### P0 AI Roles
|
||||
### Evaluation Release Boundary
|
||||
|
||||
Use AI above the per-paper level:
|
||||
A candidate must not ship if it introduces:
|
||||
|
||||
- evidence auditor: check whether a claim is actually supported;
|
||||
- contradiction finder: surface papers that challenge the current model;
|
||||
- synthesis maintainer: update the claim network when evidence changes;
|
||||
- research partner: help the user reason through an unfamiliar target;
|
||||
- review generator: create recall/application prompts from validated claims.
|
||||
- a safety or isolation violation;
|
||||
- more false completions;
|
||||
- an unrecorded model, prompt, tool, fixture, context, or budget change;
|
||||
- a critical-family regression hidden by average success;
|
||||
- gains that disappear under independent validation.
|
||||
|
||||
Use Luna for cheap classification, Terra for retrieval briefs and bounded dialogue, and Sol only for explicit synthesis/critique. The user also has a company GPT-5.5 data-agent API with effectively unlimited tokens, but integration must wait until its endpoint, auth, rate behavior, and data policy are documented.
|
||||
No live benchmark has been run yet. The latest evaluation papers are only abstract-level candidates and
|
||||
need full-text audit before they can change the contract.
|
||||
|
||||
### P0 Validation
|
||||
## P1: Continue Collection Through Questions
|
||||
|
||||
Do not judge success by page completion or time spent. Validate whether the user can:
|
||||
The corpus now contains 1,125 papers, 1,110 of them queued. More collection is useful only when paired
|
||||
with processing:
|
||||
|
||||
1. explain five claims without paper names;
|
||||
2. identify the evidence boundary of each claim;
|
||||
3. diagnose a new Agent memory design;
|
||||
4. predict when memory will hurt;
|
||||
5. recover the model after 1, 3, and 7 days;
|
||||
6. change a belief when presented with counterevidence.
|
||||
- run a narrow weekly increment from the last explicit date boundary;
|
||||
- retain broad candidates and truthful status labels;
|
||||
- prioritize papers that challenge a current research move or evaluation assumption;
|
||||
- audit the latest evaluation cluster first;
|
||||
- refresh Beijing jobs and official industry practice separately.
|
||||
|
||||
## P1: Connect Research to the Frontend
|
||||
|
||||
The current Atlas/search/reader can remain, but it should feed the stateful path:
|
||||
|
||||
- selecting a topic opens its unresolved questions and claims, not only a paper list;
|
||||
- selecting a paper shows which claims it supports or challenges;
|
||||
- search results can be promoted into an evidence review queue;
|
||||
- paper chat can create a candidate claim, never an automatically accepted fact;
|
||||
- the primary desktop focus should shift to the active research/learning task;
|
||||
- mobile should preserve the same conceptual flow with fewer simultaneous panes.
|
||||
|
||||
Avoid another navigation-heavy redesign before the data and learning loop exist.
|
||||
|
||||
## P1: Continue Collection Selectively
|
||||
|
||||
Current gaps:
|
||||
|
||||
- only 6 JD items, last checked 2026-07-08;
|
||||
- only 9 industry items;
|
||||
- 959 of 974 paper items are still queued;
|
||||
- paper source approval and automated collection policy need an explicit user decision;
|
||||
- the corpus is heavily biased toward May-July 2026 and broad arXiv query terms.
|
||||
|
||||
Next collection work should be question-driven:
|
||||
|
||||
- refresh Beijing DeepSeek/Tencent/ByteDance/Alibaba/Baidu roles and record changes;
|
||||
- add official company material that explains memory, evaluation, coding agents, or computer use;
|
||||
- deepen only papers that challenge or extend current claims;
|
||||
- do not add another thousand papers without a processing and evidence plan.
|
||||
|
||||
After any item changes:
|
||||
After item changes:
|
||||
|
||||
```bash
|
||||
python3 tools/collection/build_index.py
|
||||
python3 tools/check_project.py
|
||||
```
|
||||
|
||||
## P2: Research Refresh Protocol
|
||||
## P2: Productize Only Validated Workflows
|
||||
|
||||
When refreshing the memory audit or starting another topic:
|
||||
The Atlas/search/reader can remain as an orientation surface. New frontend work should follow a proven
|
||||
workflow:
|
||||
|
||||
1. define exact date windows and approved sources;
|
||||
2. save the broad candidate snapshot;
|
||||
3. separate deterministic filters from model triage;
|
||||
4. identify comparable benchmarks and controlled studies;
|
||||
5. read experiments, ablations, costs, and limitations in full text;
|
||||
6. update an evidence ledger before writing conclusions;
|
||||
7. preserve negative results and failed hypotheses;
|
||||
8. record the run under `experiments/` and update project status.
|
||||
- a topic opens unresolved questions and compiled moves, not only a paper list;
|
||||
- a paper shows the move it supports, challenges, or merely resembles;
|
||||
- the primary view is the active research decision;
|
||||
- candidate claims require evidence review before becoming durable knowledge;
|
||||
- user judgments and review history remain inspectable and reversible.
|
||||
|
||||
Do not compare raw scores across papers unless model, embedding, prompt, visible information, and budget are controlled.
|
||||
Avoid another graph or dashboard redesign until either the paper compiler or evaluation runner produces
|
||||
something the user repeatedly uses.
|
||||
|
||||
## P2: Operations
|
||||
|
||||
Still missing:
|
||||
|
||||
- public-domain deployment documentation outside this repository;
|
||||
- authentication/authorization for a public or multi-user site;
|
||||
- automated backup for future personal learning state;
|
||||
- CI that runs `tools/check_project.py` on push;
|
||||
- tests for web API behavior and frontend flows.
|
||||
|
||||
The repository now has an enabled user-level systemd service managed through `web/manage.sh`. It restarts on failure and survives user sessions because linger is enabled on the current host, but it is not a complete production deployment: proxy/TLS/auth configuration remains external.
|
||||
- CI for `tools/check_project.py` and unit tests;
|
||||
- authentication and authorization for any public multi-user state;
|
||||
- automated backup for future personal learning data;
|
||||
- repository-managed documentation of proxy/TLS deployment;
|
||||
- web API and frontend flow tests.
|
||||
|
||||
## Explicit Non-Goals
|
||||
|
||||
- Do not present 1,110 queued papers as understood knowledge.
|
||||
- Do not revive the 12-paper lecture as a field synthesis.
|
||||
- Do not use per-paper summaries as the final knowledge unit.
|
||||
- Do not build a leaderboard or single composite Agent score.
|
||||
- Do not let an Agent or the same judge model certify its own success.
|
||||
- Do not hide regressions behind average improvement.
|
||||
- Do not make the graph more decorative as the next milestone.
|
||||
- Do not present 974 queued papers as knowledge already learned.
|
||||
- Do not revive the 12-paper lecture as the primary path.
|
||||
- Do not use per-paper summary/translation as the main AI value proposition.
|
||||
- Do not let AI write durable memory solely because a statement repeats in traces.
|
||||
- Do not hide model, embedding, context, or cost changes inside an architecture comparison.
|
||||
|
||||
## First Continuation Task
|
||||
## First Continuation Tasks
|
||||
|
||||
The best next task for another AI is:
|
||||
For the knowledge-base repository:
|
||||
|
||||
1. read the memory findings and evidence ledger;
|
||||
2. define the claim/user-state/review schemas;
|
||||
3. compile 5-8 audited claims into a stateful local dataset;
|
||||
4. implement one progressive learning/research view;
|
||||
5. run it with the user and record what changes in their understanding;
|
||||
6. keep or discard the interaction model based on that evidence.
|
||||
1. choose one bounded topic and enumerate the complete relevant corpus;
|
||||
2. define the fingerprint schema and evidence locators;
|
||||
3. compile and audit a representative batch;
|
||||
4. test whether research-move compression preserves important distinctions;
|
||||
5. run the result with the user before building UI.
|
||||
|
||||
That is the shortest path from the current corpus browser toward the actual goal.
|
||||
For `zk-data-agent`:
|
||||
|
||||
1. port the two currently runnable task contracts;
|
||||
2. extract the live runner and add one deterministic validator;
|
||||
3. export valid result JSONL;
|
||||
4. run a smoke comparison only after deterministic layers pass;
|
||||
5. record the decision and failures, not just the score.
|
||||
|
||||
Reference in New Issue
Block a user