Add Agent evaluation report site

This commit is contained in:
wuyang
2026-07-27 16:44:10 +08:00
parent df475e8d90
commit cd1eacbd9e
15 changed files with 2487 additions and 0 deletions
+17
View File
@@ -65,6 +65,7 @@ The strongest current conclusion is that recent progress is concentrated in bett
| research | `research/memory/` | field findings and evidence ledger |
| Agent evaluation | `research/evaluation/` | executable design contract for task suites, validators, paired comparisons, and release gates |
| evaluation data/tools | `data/evaluation/`, `tools/evaluation/` | versioned task contracts and JSONL result scorer |
| evaluation report | `evaluation-site/` | Chinese interactive explanation of the contract, decision gates, tasks, and artifacts |
| research data | `data/research/` | committed corpus, screening, landscape and CSV ledger |
| research tools | `tools/research/` | paginated search, abstract fetch, screening, classification, landscape build |
| experiments | `experiments/knowledge-compilation/` | failed pilot and completed field audit |
@@ -112,6 +113,9 @@ zk-data-agent runtime/events + Zero decision-loop constraints
Only the task contracts and scorer exist in this repository. The reusable runner, fixtures, validators,
and real-model comparison still need to be implemented in `zk-data-agent`.
`evaluation-site/` visualizes this contract but does not add evidence. Its adjustable decision values are
explicitly labeled as a method simulation, not a real benchmark.
## Web Application
Capabilities currently implemented:
@@ -149,6 +153,19 @@ On 2026-07-27 both the Tailscale endpoint and `https://lab.k1412.top/api/health`
The live Ollama inventory checked on 2026-07-12 included `ChatGPT-5.6:Luna`, `Terra`, `Sol`, `auto`, `embed`, and `gpt-4o:latest`. The web defaults were updated from stale `light/fast/large` tags to configurable Luna/Terra/Sol tiers.
The static Agent evaluation report is packaged separately:
```text
source: evaluation-site/
public mapping: https://agent-eval.k1412.top/
container port: 8080
health: /health
deployment: Unraid Compose Manager, app slug agent-eval
```
It uses no Ollama or runtime API. The source report remains usable even when the research application is
offline.
## Runtime Constraints
- Python standard library only; no package installation is required for the current web server and collection scripts.
+32
View File
@@ -223,3 +223,35 @@ runner work.
Decision retained: evaluation exists to decide whether one controlled Agent change should ship. Do not
compress safety, false completion, gains, regressions, latency, and cost into one decorative score, and
do not build a dashboard before the runner and validators exist.
## 12. Evaluation Explanation Site
The user requested a web presentation of the evaluation conclusion and artifacts. A separate static
report was created under `evaluation-site/` instead of adding another mode to the Atlas.
The report exposes:
- the central release-decision conclusion and evidence boundary;
- an adjustable paired-outcome simulation using the scorer's gain/regression logic;
- all six evaluation layers;
- searchable status-aware summaries of the ten task contracts;
- direct links to the design, contracts, scorer, experiment, corpus increment, and paper library.
The simulator is explicitly not a benchmark result. It exists to make the decision rule inspectable.
The repository artifacts remain the source of truth.
Desktop and 390px mobile views were checked with a real Chromium browser. Programmatic checks covered
horizontal overflow, JavaScript errors, safety-gate rejection, current-task filtering, memory search,
and layer switching.
The first Docker build attempt exposed two deployment details:
- Docker Hub returned a transient EOF while resolving an uncached base tag; the already-present,
current `nginx:1.29.5-alpine` image was used and the exact base digest was recorded by BuildKit.
- copying individual assets with `--chmod=0644` caused the automatically created parent directory to
become non-traversable. The Dockerfile now creates `/usr/share/nginx/html/assets` as `0755` before
copying files as `0644`.
Decision retained: an explanation report is acceptable before the runner because it displays the
contract and its missing evidence. It must not evolve into a score dashboard until real Result JSONL
exists.
+4
View File
@@ -127,6 +127,10 @@ A candidate must not ship if it introduces:
No live benchmark has been run yet. The latest evaluation papers are only abstract-level candidates and
need full-text audit before they can change the contract.
The report at `https://agent-eval.k1412.top/` explains the contract and simulates its decision rules.
It is not evidence that any Agent variant passed. Future result views must consume the same validator
JSONL instead of embedding numbers in frontend code.
## P1: Continue Collection Through Questions
The corpus now contains 1,125 papers, 1,110 of them queued. More collection is useful only when paired