Refresh papers and define Agent evaluation

2026-07-27 16:28:23 +08:00
parent 8e4ff3779b
commit df475e8d90
180 changed files with 31313 additions and 160 deletions
@@ -0,0 +1,87 @@
+# Collection Run: 2026-07-27 arXiv Incremental Refresh
+
+status: completed
+
+## Scope
+
+- source: arXiv official search pages
+- date window: 2026-07-09 to 2026-07-27
+- query groups: 15
+- unique candidates seen: 249
+- high-relevance candidates selected: 157
+- already-existing selected papers: 6
+- new paper files written: 151
+- reserve candidates: 92
+
+No submissions later than 2026-07-24 appeared in the result set. July 25-26 were a weekend; the run
+still keeps 2026-07-27 as its explicit upper boundary.
+
+## Outputs
+
+- `data/arxiv-agent-candidates-2026-07-09-to-2026-07-27.json`
+- `data/arxiv-agent-papers-2026-07-09-to-2026-07-27.json`
+- `papers/corpus-summary-2026-07-27.md`
+- 151 generated `papers/items/*.md` files
+
+## Selection
+
+The same high-recall rule as the previous expansion was retained:
+
+```text
+collection_score >= 13
+relevance == high
+```
+
+All selected files remain `queued`. Selection means relevant enough to retain, not read or validated.
+
+## Selected Auto-Tag Counts
+
+| Topic | Count |
+| --- | ---: |
+| agent-evaluation | 137 |
+| tool-use | 119 |
+| rag | 73 |
+| planning | 60 |
+| reasoning | 56 |
+| workflow-agent | 49 |
+| agent-safety | 49 |
+| computer-use | 48 |
+| coding-agent | 46 |
+| memory | 42 |
+| multi-agent | 36 |
+
+Tags overlap and are recall-oriented.
+
+## Collection Reliability Finding
+
+The first two attempts failed because `export.arxiv.org` timed out and then returned HTTP 503. The
+collector previously retried only HTTP 429 and `URLError`.
+
+The collection tool was improved to:
+
+- retry 429 and transient 5xx responses;
+- retry read timeout and HTTP connection failures;
+- expose a configurable request timeout;
+- print query/page progress;
+- support the official `arxiv.org/search/` HTML result pages as a fallback backend.
+
+The successful run used `--backend search-html`. The parser extracts the arXiv ID, title, full displayed
+abstract, authors, categories and submitted date before applying the existing classifier. Candidate and
+selected manifests intentionally omit abstracts, preserving the existing repository contract.
+
+A post-run review removed an unsafe early-stop assumption: arXiv announced-date ordering can mix old
+cross-listed papers into a recent page, so original submission date cannot terminate pagination. A full
+rerun with the corrected collector returned the same 249 unique candidates and 157 selected records.
+
+## Evaluation-Relevant Signals
+
+Abstract-level signals in the refresh include:
+
+- evaluation protocols and dynamic environments, not only static success rates;
+- trajectory review, failure attribution and recovery tooling;
+- deterministic verification and evidence grounding;
+- cost-aware evaluation and deployment realism;
+- explicit separation of gains from tasks regressed by a new Agent component.
+
+These are triage signals. Full-text evidence review is still required before turning them into field
+conclusions.