wuyang/agent
2
2
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refresh papers and define Agent evaluation

Browse Source
This commit is contained in:
wuyang
2026-07-27 16:28:23 +08:00
parent 8e4ff3779b
commit df475e8d90
180 changed files with 31313 additions and 160 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/06-project-status.md
+32 -7
View File
@@ -1,7 +1,7 @@
# Project Status
last_reviewed: 2026-07-12
baseline_before_handoff: `ab758e7`
last_reviewed: 2026-07-27
baseline_before_update: `8e4ff37`
branch: `main`
remote: `https://git.k1412.top/wuyang/agent.git`
@@ -16,19 +16,23 @@ The long-term product is not “a place to browse papers.” It should help the
## Current Snapshot
The deterministic index currently contains 989 items:
The deterministic index currently contains 1,140 items:
| Collection | Count | State |
| --- | ---: | --- |
| papers | 974 | 959 queued, 15 skimmed |
| papers | 1,125 | 1,110 queued, 15 skimmed |
| jobs | 6 | 4 active, 2 unknown |
| industry | 9 | 8 analyzed, 1 queued |
The expanded arXiv run saw 1,506 candidates, promoted 970 high-recall candidates, and retained 536 reserve candidates. These are collection counts, not quality or reading-completion counts.
The 2026-07-08 expanded arXiv run saw 1,506 candidates, promoted 970 high-recall candidates,
and retained 536 reserve candidates. The 2026-07-27 incremental refresh covered July 9-27, saw
249 unique candidates, selected 157 high-relevance records, skipped 6 existing papers, and added
151 paper items. The newest submission returned was 2026-07-24. These are collection counts, not
quality or reading-completion counts.
The memory topic has two intentionally different counts:
- 296 indexed items have the broad `memory` topic, including non-paper streams;
- 336 indexed items now have the broad `memory` topic, including non-paper streams;
- 293 local paper items were resolved for the dedicated memory audit.
The memory research snapshot contains:
@@ -59,6 +63,8 @@ The strongest current conclusion is that recent progress is concentrated in bett
| collection runs | `collection-runs/` | queries, scope, counts, method notes |
| collection tools | `tools/collection/` | item creation, arXiv collection, promotion, indexing, URL checks |
| research | `research/memory/` | field findings and evidence ledger |
| Agent evaluation | `research/evaluation/` | executable design contract for task suites, validators, paired comparisons, and release gates |
| evaluation data/tools | `data/evaluation/`, `tools/evaluation/` | versioned task contracts and JSONL result scorer |
| research data | `data/research/` | committed corpus, screening, landscape and CSV ledger |
| research tools | `tools/research/` | paginated search, abstract fetch, screening, classification, landscape build |
| experiments | `experiments/knowledge-compilation/` | failed pilot and completed field audit |
@@ -92,6 +98,20 @@ paginated arXiv queries + local memory-tagged papers
Do not merge these two lines silently. The first is a broad browsing corpus; the second is an evidence audit with explicit boundaries.
The Agent evaluation design follows a third line:
```text
zk-data-agent runtime/events + Zero decision-loop constraints
-> versioned task contracts
-> isolated repeated baseline/variant runs
-> independent validators + trace rules
-> JSONL result records
-> gain/regression/safety decision
```
Only the task contracts and scorer exist in this repository. The reusable runner, fixtures, validators,
and real-model comparison still need to be implemented in `zk-data-agent`.
## Web Application
Capabilities currently implemented:
@@ -121,7 +141,11 @@ public mapping: https://lab.k1412.top/
health: /api/health
```
At the beginning of the 2026-07-12 handoff audit, no process was listening on 18080 and the public domain returned 502. The handoff added and enabled a user-level systemd service through `web/manage.sh`; local and public health then returned 200 with 974 papers and `ollama_ready: true`. Runtime status is still external state and must always be checked live.
At the beginning of the 2026-07-12 handoff audit, no process was listening on 18080 and the public domain returned 502. The handoff added and enabled a user-level systemd service through `web/manage.sh`;
local and public health then returned 200. Runtime status is external state and must always be checked
live. After a corpus update, restart the service because the index is loaded at process startup.
On 2026-07-27 both the Tailscale endpoint and `https://lab.k1412.top/api/health` returned the refreshed
1,125-paper count with `ollama_ready: true`.
The live Ollama inventory checked on 2026-07-12 included `ChatGPT-5.6:Luna`, `Terra`, `Sol`, `auto`, `embed`, and `gpt-4o:latest`. The web defaults were updated from stale `light/fast/large` tags to configurable Luna/Terra/Sol tiers.
@@ -144,6 +168,7 @@ Offline validation:
```bash
python3 tools/check_project.py
python3 -m unittest tools.evaluation.test_score_agent_runs
git diff --check
```