agent/docs/07-exploration-history.md

# Exploration History

last_reviewed: 2026-07-28

This is the decision history, including failed routes. Preserve it so future work does not rediscover the same lessons.

## 1. Repository and Content Streams

On 2026-07-08 the workspace was initialized as a long-lived Agent knowledge base, not a one-off report.

The first structure included stable knowledge, projects, experiments, references, and templates. It was then expanded into three first-class collection streams:

- `jobs/`: Beijing-focused Agent/LLM roles over a rolling six-month window;
- `papers/`: research evidence and reading queue;
- `industry/`: large-company reports, blogs, products, repositories, and benchmarks.

Decision retained: papers, jobs, and industry must not be mixed because they answer different questions. Their common organization layer is metadata/index/search, not a deep directory taxonomy.

Relevant commits: `47cea9d`, `1b6cc1f`, `36c2499`.

## 2. First Collection and Expanded Corpus

The first sweep created 21 items and the source/schema workflow. The expanded arXiv sweep then recalled 1,506 candidates and promoted 970 high-relevance candidates, bringing the paper corpus to 974.

What worked:

- source registries and explicit source quality;
- keeping candidate and promoted manifests;
- generated frontmatter plus a deterministic index;
- preserving `queued` as a truthful unread state.

What did not work:

- treating the resulting count as understanding;
- assuming auto tags were a stable taxonomy;
- using non-paginated/latest-only query results to infer historical trends.

Relevant artifacts: `collection-runs/`, `papers/corpus-summary-2026-07-08.md`, `data/arxiv-agent-*.json`.

Relevant commits: `1cbc98e`, `391b3a4`.

## 3. Paper Browser to Knowledge Atlas

The first frontend exposed search, filtering, paper preview, abstract fetch, translation, summary, and deep analysis. It evolved into a knowledge Atlas with topic nodes, co-occurrence, timelines, routes, desktop/mobile layouts, and an ECharts graph.

Further iterations fixed:

- cache busting and stale deployment confusion;
- an oversized Atlas payload;
- desktop scrolling and reader focus;
- mobile map centering;
- interactive graph visuals;
- app version visibility.

This produced a usable corpus browser, but the user later rated its practical learning value at effectively 0/100. The reason was not mainly visual quality: exploration was stateless, and nothing helped the user retain, compress, challenge, or apply knowledge.

Decision retained: the Atlas may remain an orientation surface, but it must not be treated as the product or as evidence of learning.

Relevant commits: `e783a7d` through `15b5cec`.

## 4. Search Agent and Per-Paper AI

Search was upgraded from keyword matching to a compact Agent loop:

```text
interpret query -> expand Chinese/English terms -> retrieve locally
-> aggregate dates/topics/trends -> generate brief -> allow follow-up
```

The UI initially exposed every fixed pipeline step. That looked mechanical, so it was simplified into one active status row with a changing task summary. Paper reading also gained multi-turn chat.

What worked:

- AI-assisted query interpretation and bilingual expansion;
- progressive activity updates instead of a blocking spinner;
- contextual follow-up over a bounded result set;
- serialized model use and caching.

What remained insufficient:

- AI actions were still centered on individual papers or a single search;
- the system did not maintain claims, contradictions, the user's evolving model, or review state;
- a generated brief disappeared into another stateless browsing session.

Decision retained: future AI should operate at the claim/evidence/model level, not merely summarize or translate documents.

Relevant commits: `894c914`, `4892e7a`.

## 5. Ollama Integration Lessons

The application originally sent client-side context/residency parameters. The user identified that a `num_ctx` mismatch could restart a model and clear residency, and that `keep_alive` behavior could unload it.

The fix was to let the Ollama server own both settings. The application now only sends generation options such as temperature and output length, serializes calls, and caches results.

On 2026-07-12 the live model inventory no longer contained the old `light/fast/large` tags used by the web app. The defaults were changed to configurable tiers backed by the currently available Luna/Terra/Sol tags, and health now reports model mismatches.

Decision retained: always verify `/api/tags`; never infer deployed model aliases from old documentation.

Relevant commits: `47c301c`, `ac6e0e0`, and the 2026-07-12 handoff update.

On 2026-07-28 the runtime moved to llama.cpp after repeated Ollama runner churn and
resource-residency ambiguity. Luna was isolated on the NAS AMD iGPU through Vulkan,
while Terra and Sol became two API behaviors over one fixed 27B CUDA runner on the
V100. The web app now uses the servers' OpenAI-compatible endpoints, fixes context
at deployment time, and controls thinking by tier.

Decision retained: product tiers do not require separate physical model copies.
Keep resource ownership in the serving layer and let Terra/Sol share one resident
runner when their weights are identical.

## 6. Learning-System Reframing

The user clarified the real objective with three levels:

1. complete memory of important knowledge;
2. compression into a coherent network/system;
3. abstraction that can explain why approaches exist and generate recommendations for new goals.

The desired system is a scaffold from raw papers to internalized judgment. The user expects Codex or another strong AI to be an active research partner, not merely an orchestrator for small local models.

This reframing invalidated “paper site” as the primary product definition.

## 7. Failed 12-Paper Learning Pilot

The first response to the reframing compiled 12 Agent Memory papers into a map, chapter, evidence matrix, active recall, and machine-readable module.

The user rejected it because:

- 12 papers were too few for a field-level trend;
- it felt like a lecture rather than joint research;
- it did not answer what 200+ papers were doing;
- it did not compare current failures with six months earlier;
- it did not establish whether architecture gains were attributable.

The experiment is explicitly `failed` in `experiments/knowledge-compilation/2026-07-10-agent-memory-pilot.md`. Its artifacts remain under `learning/agent-memory/` only as historical material.

Decision retained: a polished educational package is not valuable until the underlying field audit and the user's own learning interaction are validated.

Relevant commit: `36c631e`.

## 8. Agent Memory Field Audit

The failed pilot was replaced with a broader research audit:

- five paginated arXiv queries;
- 516 unique abstract-level candidates;
- complete local screening as triage;
- exact recent/prior six-month windows;
- 32 downloaded/text-located papers;
- 23 papers in an evidence ledger with explicit non-extrapolation boundaries.

The most useful findings were negative or diagnostic:

- retrieval quality often matters more than elaborate write structures for static QA;
- model and embedding choices can reverse architecture conclusions;
- update/forgetting and automatic writing remain unsolved;
- execution-state memory is the clearest task-specific capability direction;
- memory can interfere with other Agent components;
- cost and no-memory baselines are mandatory.

This audit is the current research baseline. Read `research/memory/findings.md` and `research/memory/evidence-ledger.md` rather than the old learning module.

Relevant commit: `ab758e7`.

## 9. Position After the Memory Audit

The repository now has a strong collection substrate, a functioning exploration prototype, and one evidence-led field audit. It does not yet have the stateful personal learning/internalization layer that motivated the project.

At this point, the proposed next experiment was an end-to-end learning path that records a user's claims,
evidence judgments, contradictions, applications, and later recall.

## 10. From Learning Page to Knowledge Funnel

The user then made the throughput problem more precise: the target is not a nicer way to read papers one
by one. It is to preserve as much of an expert's understanding as possible while reducing the time needed
to process roughly 1,000 papers, ideally to one concentrated day.

The important expert behavior is:

```text
paper -> what problem / why now / prior method / actual change
-> evidence / failure boundary / place in the field
-> compare with neighboring work
-> notice missing methods and generate new hypotheses
```

This reframing changes the primary artifact from a course or browser to a paper compiler:

1. extract compact, source-linked paper fingerprints;
2. collapse repeated work into research moves and lineages;
3. compare evidence and regressions inside each move;
4. compile a field grammar that the user can inspect, challenge, and apply;
5. persist the user's judgments and delayed review only after the compiled knowledge is useful.

Decision retained: the frontend is optional at this stage. The first proof should be an inspectable data
pipeline and a high-density research artifact, not another navigation surface.

## 11. Incremental Refresh and Agent Evaluation Design

On 2026-07-27 the paper boundary was refreshed from July 9 through July 27:

- 249 unique arXiv candidates;
- 157 high-relevance selected records;
- 151 new queued paper items;
- 1,125 total paper items after indexing.

The arXiv export API timed out and returned 503, so the collector gained transient-error retries,
configurable timeouts, progress output, and an official search-page fallback. The successful run used
that fallback. This is an operational improvement, not evidence that the selected papers were read.

The same work inspected `zk-data-agent` at commit `e2e7a7b` and the Zero product flow. The requested
output was clarified as an Agent evaluation scheme, not a new Agent implementation.

What the target already had:

- isolated real workspaces and layered tests;
- versioned run events and complete tool lifecycle traces;
- evidence-gated completion;
- one real-model live evaluation script.

What it lacked:

- a versioned task suite with fixtures and deterministic validators;
- repeated paired baseline/variant runs;
- complete reproducibility manifests;
- unified failure labels and explicit false-completion checks;
- separate gain and regression accounting.

The resulting v1 contract lives under `research/evaluation/`, with ten task contracts and a JSONL scorer.
No real-model comparison was run, and most tasks correctly remain marked as requiring target-repository
runner work.

Decision retained: evaluation exists to decide whether one controlled Agent change should ship. Do not
compress safety, false completion, gains, regressions, latency, and cost into one decorative score, and
do not build a dashboard before the runner and validators exist.

## 12. Evaluation Explanation Site

The user requested a web presentation of the evaluation conclusion and artifacts. A separate static
report was created under `evaluation-site/` instead of adding another mode to the Atlas.

The report exposes:

- the central release-decision conclusion and evidence boundary;
- an adjustable paired-outcome simulation using the scorer's gain/regression logic;
- all six evaluation layers;
- searchable status-aware summaries of the ten task contracts;
- direct links to the design, contracts, scorer, experiment, corpus increment, and paper library.

The simulator is explicitly not a benchmark result. It exists to make the decision rule inspectable.
The repository artifacts remain the source of truth.

Desktop and 390px mobile views were checked with a real Chromium browser. Programmatic checks covered
horizontal overflow, JavaScript errors, safety-gate rejection, current-task filtering, memory search,
and layer switching.

The first Docker build attempt exposed two deployment details:

- Docker Hub returned a transient EOF while resolving an uncached base tag; the already-present,
  current `nginx:1.29.5-alpine` image was used and the exact base digest was recorded by BuildKit.
- copying individual assets with `--chmod=0644` caused the automatically created parent directory to
  become non-traversable. The Dockerfile now creates `/usr/share/nginx/html/assets` as `0755` before
  copying files as `0644`.

Decision retained: an explanation report is acceptable before the runner because it displays the
contract and its missing evidence. It must not evolve into a score dashboard until real Result JSONL
exists.

The report was deployed as immutable image
`docker.k1412.top/wuyang/agent-eval:20260727T084425Z-cd1eacb` through Unraid Compose Manager on NAS port
12005. NPM route and Let's Encrypt HTTPS were enabled for `https://agent-eval.k1412.top/`, and the site
was registered in the k1412 homepage `projects` category.

## 13. Stop Implementing, Research Completion First

After reviewing the evaluation report, the user rejected the work as useful research. The failure was
not that the site lacked another chart. The design had moved from code inspection to task contracts,
scoring, and presentation without first establishing what current Agent research can actually prove
about completion and recovery.

The user explicitly paused experiments and requested general conclusions before any new web form.
The project therefore audited a problem-driven 40-paper full-text pool spanning false success, state
and procedure verification, evaluator validity, test adequacy, tool failure, root-cause attribution,
recovery routing, and stopping. Thirty-seven papers entered an evidence ledger; three remained
historical context.

The main corrections are:

- task completion is an evidence contract, not an Agent statement or one benchmark bit;
- concrete capabilities include state change, side-effect control, procedure compliance, coordination,
  failure detection, localization, recovery routing, restoration, re-verification, and repeated
  reliability;
- deterministic state checks, trace rules, learned judges, and humans have different responsibilities;
- verification only creates an explicit failure; it does not provide diagnosis or recovery;
- fixed retry, reflection, replanning, and repair loops all have documented regressions;
- the benchmark protocol and verifier must themselves be audited for weak tests, leakage, drift,
  model changes, and reward hacking.

The cross-paper synthesis proposes a completion certificate, structured failure object, and
state-dependent recovery router. These are marked as system inferences, not experimental results.

Decision retained: do not implement the Agent evaluation runner or redesign its report until the user
accepts the research decomposition. If implementation resumes, rewrite the task contracts around
outcome, invariant, procedure, semantic residual, uncertainty, and explicit recovery stages.

Current source of truth:

- `research/completion-verification/findings.md`
- `research/completion-verification/evidence-ledger.md`
- `experiments/knowledge-compilation/2026-07-27-agent-completion-research-audit.md`

## 14. Research Document, Not a New Product Shape

The user asked to read the completion research as a web page because the Markdown artifacts were
harder to traverse. This did not reopen the larger website/product-design question. The implementation
therefore treats the page as a document rendering of committed research:

- the first view states the conclusion and evidence boundary;
- eight conclusions lead into capability decomposition and two distinct chains for closure and recovery;
- each argument exposes its direct numbers and non-extrapolation boundary progressively;
- the 2023-2026 timeline separates measurement progress from unsolved general capability;
- the completion certificate, failure object, and recovery router remain labeled as system inference;
- all 40 source papers are searchable, with 37 audited and 3 context-only records;
- evidence data is generated from the Markdown ledger and checked by source hash.

The former v1 evaluation explanation page remains at `/evaluation-v1.html`. It was not deleted or
silently rewritten.

Desktop 1440×1000 and mobile 390×844 Chrome checks found no horizontal overflow or console/page
errors. Core filtering returned 26 records; recovery filtering returned 12; public evidence search and
expansion worked. The production image
`docker.k1412.top/wuyang/agent-eval:20260727T165727Z-4c88faa` was deployed on the existing NAS port
12005 and existing HTTPS route.

Decision retained: use `https://agent-eval.k1412.top/` to challenge the research with the user.
Do not interpret successful document presentation as validation of the proposed evaluation system or
as permission to resume runner implementation.