Document and harden project handoff

docs/07-exploration-history.md

@@ -0,0 +1,159 @@
+# Exploration History
+
+last_reviewed: 2026-07-12
+
+This is the decision history, including failed routes. Preserve it so future work does not rediscover the same lessons.
+
+## 1. Repository and Content Streams
+
+On 2026-07-08 the workspace was initialized as a long-lived Agent knowledge base, not a one-off report.
+
+The first structure included stable knowledge, projects, experiments, references, and templates. It was then expanded into three first-class collection streams:
+
+- `jobs/`: Beijing-focused Agent/LLM roles over a rolling six-month window;
+- `papers/`: research evidence and reading queue;
+- `industry/`: large-company reports, blogs, products, repositories, and benchmarks.
+
+Decision retained: papers, jobs, and industry must not be mixed because they answer different questions. Their common organization layer is metadata/index/search, not a deep directory taxonomy.
+
+Relevant commits: `47cea9d`, `1b6cc1f`, `36c2499`.
+
+## 2. First Collection and Expanded Corpus
+
+The first sweep created 21 items and the source/schema workflow. The expanded arXiv sweep then recalled 1,506 candidates and promoted 970 high-relevance candidates, bringing the paper corpus to 974.
+
+What worked:
+
+- source registries and explicit source quality;
+- keeping candidate and promoted manifests;
+- generated frontmatter plus a deterministic index;
+- preserving `queued` as a truthful unread state.
+
+What did not work:
+
+- treating the resulting count as understanding;
+- assuming auto tags were a stable taxonomy;
+- using non-paginated/latest-only query results to infer historical trends.
+
+Relevant artifacts: `collection-runs/`, `papers/corpus-summary-2026-07-08.md`, `data/arxiv-agent-*.json`.
+
+Relevant commits: `1cbc98e`, `391b3a4`.
+
+## 3. Paper Browser to Knowledge Atlas
+
+The first frontend exposed search, filtering, paper preview, abstract fetch, translation, summary, and deep analysis. It evolved into a knowledge Atlas with topic nodes, co-occurrence, timelines, routes, desktop/mobile layouts, and an ECharts graph.
+
+Further iterations fixed:
+
+- cache busting and stale deployment confusion;
+- an oversized Atlas payload;
+- desktop scrolling and reader focus;
+- mobile map centering;
+- interactive graph visuals;
+- app version visibility.
+
+This produced a usable corpus browser, but the user later rated its practical learning value at effectively 0/100. The reason was not mainly visual quality: exploration was stateless, and nothing helped the user retain, compress, challenge, or apply knowledge.
+
+Decision retained: the Atlas may remain an orientation surface, but it must not be treated as the product or as evidence of learning.
+
+Relevant commits: `e783a7d` through `15b5cec`.
+
+## 4. Search Agent and Per-Paper AI
+
+Search was upgraded from keyword matching to a compact Agent loop:
+
+```text
+interpret query -> expand Chinese/English terms -> retrieve locally
+-> aggregate dates/topics/trends -> generate brief -> allow follow-up
+```
+
+The UI initially exposed every fixed pipeline step. That looked mechanical, so it was simplified into one active status row with a changing task summary. Paper reading also gained multi-turn chat.
+
+What worked:
+
+- AI-assisted query interpretation and bilingual expansion;
+- progressive activity updates instead of a blocking spinner;
+- contextual follow-up over a bounded result set;
+- serialized model use and caching.
+
+What remained insufficient:
+
+- AI actions were still centered on individual papers or a single search;
+- the system did not maintain claims, contradictions, the user's evolving model, or review state;
+- a generated brief disappeared into another stateless browsing session.
+
+Decision retained: future AI should operate at the claim/evidence/model level, not merely summarize or translate documents.
+
+Relevant commits: `894c914`, `4892e7a`.
+
+## 5. Ollama Integration Lessons
+
+The application originally sent client-side context/residency parameters. The user identified that a `num_ctx` mismatch could restart a model and clear residency, and that `keep_alive` behavior could unload it.
+
+The fix was to let the Ollama server own both settings. The application now only sends generation options such as temperature and output length, serializes calls, and caches results.
+
+On 2026-07-12 the live model inventory no longer contained the old `light/fast/large` tags used by the web app. The defaults were changed to configurable tiers backed by the currently available Luna/Terra/Sol tags, and health now reports model mismatches.
+
+Decision retained: always verify `/api/tags`; never infer deployed model aliases from old documentation.
+
+Relevant commits: `47c301c`, `ac6e0e0`, and the 2026-07-12 handoff update.
+
+## 6. Learning-System Reframing
+
+The user clarified the real objective with three levels:
+
+1. complete memory of important knowledge;
+2. compression into a coherent network/system;
+3. abstraction that can explain why approaches exist and generate recommendations for new goals.
+
+The desired system is a scaffold from raw papers to internalized judgment. The user expects Codex or another strong AI to be an active research partner, not merely an orchestrator for small local models.
+
+This reframing invalidated “paper site” as the primary product definition.
+
+## 7. Failed 12-Paper Learning Pilot
+
+The first response to the reframing compiled 12 Agent Memory papers into a map, chapter, evidence matrix, active recall, and machine-readable module.
+
+The user rejected it because:
+
+- 12 papers were too few for a field-level trend;
+- it felt like a lecture rather than joint research;
+- it did not answer what 200+ papers were doing;
+- it did not compare current failures with six months earlier;
+- it did not establish whether architecture gains were attributable.
+
+The experiment is explicitly `failed` in `experiments/knowledge-compilation/2026-07-10-agent-memory-pilot.md`. Its artifacts remain under `learning/agent-memory/` only as historical material.
+
+Decision retained: a polished educational package is not valuable until the underlying field audit and the user's own learning interaction are validated.
+
+Relevant commit: `36c631e`.
+
+## 8. Agent Memory Field Audit
+
+The failed pilot was replaced with a broader research audit:
+
+- five paginated arXiv queries;
+- 516 unique abstract-level candidates;
+- complete local screening as triage;
+- exact recent/prior six-month windows;
+- 32 downloaded/text-located papers;
+- 23 papers in an evidence ledger with explicit non-extrapolation boundaries.
+
+The most useful findings were negative or diagnostic:
+
+- retrieval quality often matters more than elaborate write structures for static QA;
+- model and embedding choices can reverse architecture conclusions;
+- update/forgetting and automatic writing remain unsolved;
+- execution-state memory is the clearest task-specific capability direction;
+- memory can interfere with other Agent components;
+- cost and no-memory baselines are mandatory.
+
+This audit is the current research baseline. Read `research/memory/findings.md` and `research/memory/evidence-ledger.md` rather than the old learning module.
+
+Relevant commit: `ab758e7`.
+
+## 9. Current Position
+
+The repository now has a strong collection substrate, a functioning exploration prototype, and one evidence-led field audit. It does not yet have the stateful personal learning/internalization layer that motivated the project.
+
+The next meaningful experiment is not another graph redesign or another bulk paper sweep. It is one end-to-end learning path that records a user's claims, evidence judgments, contradictions, applications, and later recall, using the memory audit as its first evidence base.