Add Agent evaluation report site
This commit is contained in:
@@ -65,6 +65,7 @@ The strongest current conclusion is that recent progress is concentrated in bett
|
||||
| research | `research/memory/` | field findings and evidence ledger |
|
||||
| Agent evaluation | `research/evaluation/` | executable design contract for task suites, validators, paired comparisons, and release gates |
|
||||
| evaluation data/tools | `data/evaluation/`, `tools/evaluation/` | versioned task contracts and JSONL result scorer |
|
||||
| evaluation report | `evaluation-site/` | Chinese interactive explanation of the contract, decision gates, tasks, and artifacts |
|
||||
| research data | `data/research/` | committed corpus, screening, landscape and CSV ledger |
|
||||
| research tools | `tools/research/` | paginated search, abstract fetch, screening, classification, landscape build |
|
||||
| experiments | `experiments/knowledge-compilation/` | failed pilot and completed field audit |
|
||||
@@ -112,6 +113,9 @@ zk-data-agent runtime/events + Zero decision-loop constraints
|
||||
Only the task contracts and scorer exist in this repository. The reusable runner, fixtures, validators,
|
||||
and real-model comparison still need to be implemented in `zk-data-agent`.
|
||||
|
||||
`evaluation-site/` visualizes this contract but does not add evidence. Its adjustable decision values are
|
||||
explicitly labeled as a method simulation, not a real benchmark.
|
||||
|
||||
## Web Application
|
||||
|
||||
Capabilities currently implemented:
|
||||
@@ -149,6 +153,19 @@ On 2026-07-27 both the Tailscale endpoint and `https://lab.k1412.top/api/health`
|
||||
|
||||
The live Ollama inventory checked on 2026-07-12 included `ChatGPT-5.6:Luna`, `Terra`, `Sol`, `auto`, `embed`, and `gpt-4o:latest`. The web defaults were updated from stale `light/fast/large` tags to configurable Luna/Terra/Sol tiers.
|
||||
|
||||
The static Agent evaluation report is packaged separately:
|
||||
|
||||
```text
|
||||
source: evaluation-site/
|
||||
public mapping: https://agent-eval.k1412.top/
|
||||
container port: 8080
|
||||
health: /health
|
||||
deployment: Unraid Compose Manager, app slug agent-eval
|
||||
```
|
||||
|
||||
It uses no Ollama or runtime API. The source report remains usable even when the research application is
|
||||
offline.
|
||||
|
||||
## Runtime Constraints
|
||||
|
||||
- Python standard library only; no package installation is required for the current web server and collection scripts.
|
||||
|
||||
@@ -223,3 +223,35 @@ runner work.
|
||||
Decision retained: evaluation exists to decide whether one controlled Agent change should ship. Do not
|
||||
compress safety, false completion, gains, regressions, latency, and cost into one decorative score, and
|
||||
do not build a dashboard before the runner and validators exist.
|
||||
|
||||
## 12. Evaluation Explanation Site
|
||||
|
||||
The user requested a web presentation of the evaluation conclusion and artifacts. A separate static
|
||||
report was created under `evaluation-site/` instead of adding another mode to the Atlas.
|
||||
|
||||
The report exposes:
|
||||
|
||||
- the central release-decision conclusion and evidence boundary;
|
||||
- an adjustable paired-outcome simulation using the scorer's gain/regression logic;
|
||||
- all six evaluation layers;
|
||||
- searchable status-aware summaries of the ten task contracts;
|
||||
- direct links to the design, contracts, scorer, experiment, corpus increment, and paper library.
|
||||
|
||||
The simulator is explicitly not a benchmark result. It exists to make the decision rule inspectable.
|
||||
The repository artifacts remain the source of truth.
|
||||
|
||||
Desktop and 390px mobile views were checked with a real Chromium browser. Programmatic checks covered
|
||||
horizontal overflow, JavaScript errors, safety-gate rejection, current-task filtering, memory search,
|
||||
and layer switching.
|
||||
|
||||
The first Docker build attempt exposed two deployment details:
|
||||
|
||||
- Docker Hub returned a transient EOF while resolving an uncached base tag; the already-present,
|
||||
current `nginx:1.29.5-alpine` image was used and the exact base digest was recorded by BuildKit.
|
||||
- copying individual assets with `--chmod=0644` caused the automatically created parent directory to
|
||||
become non-traversable. The Dockerfile now creates `/usr/share/nginx/html/assets` as `0755` before
|
||||
copying files as `0644`.
|
||||
|
||||
Decision retained: an explanation report is acceptable before the runner because it displays the
|
||||
contract and its missing evidence. It must not evolve into a score dashboard until real Result JSONL
|
||||
exists.
|
||||
|
||||
@@ -127,6 +127,10 @@ A candidate must not ship if it introduces:
|
||||
No live benchmark has been run yet. The latest evaluation papers are only abstract-level candidates and
|
||||
need full-text audit before they can change the contract.
|
||||
|
||||
The report at `https://agent-eval.k1412.top/` explains the contract and simulates its decision rules.
|
||||
It is not evidence that any Agent variant passed. Future result views must consume the same validator
|
||||
JSONL instead of embedding numbers in frontend code.
|
||||
|
||||
## P1: Continue Collection Through Questions
|
||||
|
||||
The corpus now contains 1,125 papers, 1,110 of them queued. More collection is useful only when paired
|
||||
|
||||
Reference in New Issue
Block a user