wuyang/zk-data-agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove unused project artifacts

Browse Source
This commit is contained in:
武阳
2026-05-06 16:21:38 +08:00
parent 7d4ae3e7ba
commit c46f8ae4f1
37 changed files with 0 additions and 7956 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/README.md
-373
View File
@@ -1,373 +0,0 @@
# Benchmarks — claw-code-agent
This directory contains two benchmark systems:
1. **Local task benchmarks** (`benchmarks/run.py`) — custom tasks that test agent capabilities directly
2. **Standard evaluation suites** (`benchmarks/run_suite.py`) — implementations of well-known AI evaluation benchmarks
---
## Quick Start
```bash
# From the repository root:
# Set up your model endpoint
export OPENAI_API_KEY="your-key"
export OPENAI_MODEL="gpt-4" # or your model name
export OPENAI_BASE_URL="http://localhost:8000/v1" # if using local vLLM/ollama
# Run a quick smoke test (5 problems from HumanEval)
python3 -m benchmarks.run_suite --suite humaneval --limit 5 -v
# Run all benchmarks
python3 -m benchmarks.run_suite --all -o results.json
```
---
## Standard Evaluation Suites
### Available Benchmarks
| Suite | Category | # Problems (built-in) | Description |
|-------|----------|----------------------|-------------|
| **HumanEval** | Coding | 20 | Code generation from Python docstrings |
| **MBPP** | Coding | 15 | Basic Python programming problems |
| **SWE-Bench** | Coding | 5 | Resolve real-world GitHub issues |
| **Aider** | Coding | 6 | Code editing and refactoring tasks |
| **LiveCodeBench** | Coding | 5 | Competitive programming problems |
| **Codeforces** | Coding | 10 | Competitive programming with ELO rating |
| **MATH** | Math | 15 | Competition mathematics problems |
| **GSM8K** | Math | 15 | Grade school math word problems |
| **AIME** | Math | 10 | Challenging competition math (integers 0–999) |
| **IFEval** | Instruction Following | 10 | Verifiable instruction-following evaluation |
| **BFCL** | Instruction Following | 7 | Function/tool calling evaluation |
| **MMLU-Pro** | Knowledge | 10 | Professional-level multiple-choice QA (10 choices) |
| **GPQA-Diamond** | Knowledge | 10 | Graduate-level science QA (diamond difficulty) |
| **MMMLU** | Knowledge | 10 | Multilingual MMLU across languages |
| **HLE** | Knowledge | 10 | Humanity's Last Exam (extremely hard) |
| **BigBench-Hard** | Reasoning | 10 | BIG-Bench Extra Hard reasoning tasks |
| **Tau2** | Reasoning | 10 | Tool-augmented reasoning (retail/airline/finance) |
### Commands
#### List All Suites
```bash
python3 -m benchmarks.run_suite --list
```
#### Run a Specific Suite
```bash
# Coding benchmarks
python3 -m benchmarks.run_suite --suite humaneval
python3 -m benchmarks.run_suite --suite mbpp
python3 -m benchmarks.run_suite --suite swe-bench
python3 -m benchmarks.run_suite --suite aider
python3 -m benchmarks.run_suite --suite livecodebench
python3 -m benchmarks.run_suite --suite codeforces
# Math benchmarks
python3 -m benchmarks.run_suite --suite math
python3 -m benchmarks.run_suite --suite gsm8k
python3 -m benchmarks.run_suite --suite aime
# Knowledge benchmarks
python3 -m benchmarks.run_suite --suite mmlu-pro
python3 -m benchmarks.run_suite --suite gpqa-diamond
python3 -m benchmarks.run_suite --suite mmmlu
python3 -m benchmarks.run_suite --suite hle
# Reasoning benchmarks
python3 -m benchmarks.run_suite --suite bigbench-hard
python3 -m benchmarks.run_suite --suite tau2
# Instruction following benchmarks
python3 -m benchmarks.run_suite --suite ifeval
python3 -m benchmarks.run_suite --suite bfcl
```
#### Run by Category
```bash
# All coding benchmarks
python3 -m benchmarks.run_suite --category coding
# All math benchmarks
python3 -m benchmarks.run_suite --category math
# All knowledge benchmarks (MMLU-Pro, GPQA, MMMLU, HLE)
python3 -m benchmarks.run_suite --category knowledge
# All reasoning benchmarks (BigBench-Hard, Tau2)
python3 -m benchmarks.run_suite --category reasoning
# All instruction following benchmarks
python3 -m benchmarks.run_suite --category instruction-following
```
#### Run ALL Suites
```bash
python3 -m benchmarks.run_suite --all
```
#### Limit Problems (Quick Testing)
```bash
# Run only first 3 problems from each suite
python3 -m benchmarks.run_suite --all --limit 3
# Quick HumanEval smoke test
python3 -m benchmarks.run_suite --suite humaneval --limit 5 -v
```
#### Save Results to JSON
```bash
python3 -m benchmarks.run_suite --all -o results.json
python3 -m benchmarks.run_suite --suite humaneval -o humaneval_results.json
```
#### Verbose Output
```bash
python3 -m benchmarks.run_suite --suite humaneval -v
```
#### Custom Timeout
```bash
# 10 minutes per problem
python3 -m benchmarks.run_suite --suite swe-bench --timeout 600
```
#### Use Full Datasets (JSONL)
```bash
# Download or place your datasets in benchmarks/data/
python3 -m benchmarks.run_suite --suite humaneval --data-dir ./benchmarks/data/
# Or specify a custom directory
python3 -m benchmarks.run_suite --suite humaneval --data-dir /path/to/datasets/
```
---
## Using Full Datasets
Each suite looks for a JSONL file in the data directory. If the file isn't found, it falls back to the built-in subset.
| Suite | Expected File | Format |
|-------|--------------|--------|
| HumanEval | `humaneval.jsonl` | `{"task_id", "prompt", "canonical_solution", "test", "entry_point"}` |
| MBPP | `mbpp.jsonl` | `{"task_id", "text", "code", "test_list"}` |
| SWE-Bench | `swe_bench.jsonl` | `{"instance_id", "problem_statement", "setup_code", "test_cmd"}` |
| Aider | `aider.jsonl` | `{"id", "instruction", "setup_code", "test_code"}` |
| LiveCodeBench | `livecodebench.jsonl` | `{"id", "title", "description", "test_cases", "function_name"}` |
| Codeforces | `codeforces.jsonl` | `{"id", "rating", "title", "problem", "test_cases"}` |
| MATH | `math.jsonl` | `{"id", "problem", "answer", "subject", "level"}` |
| GSM8K | `gsm8k.jsonl` | `{"id", "question", "answer"}` |
| AIME | `aime.jsonl` or `aime_2026.jsonl` | `{"id", "problem", "answer"}` |
| IFEval | `ifeval.jsonl` | `{"id", "instruction", "checks"}` |
| BFCL | `bfcl.jsonl` | `{"id", "instruction", "expected_function", "setup_code", "test_code"}` |
| MMLU-Pro | `mmlu_pro.jsonl` | `{"id", "subject", "question", "choices", "answer"}` |
| GPQA-Diamond | `gpqa.jsonl` | `{"id", "subject", "question", "choices", "answer"}` |
| MMMLU | `mmmlu.jsonl` | `{"id", "language", "subject", "question", "choices", "answer"}` |
| HLE | `hle.jsonl` | `{"id", "subject", "question", "answer", "answer_type"}` |
| BigBench-Hard | `bigbench_hard.jsonl` | `{"id", "task", "question", "choices", "answer"}` |
| Tau2 | `tau2.jsonl` | `{"id", "domain", "question", "choices", "answer"}` |
### Downloading Full Datasets
```bash
# Download all datasets (builtin + official where available)
python3 -m benchmarks.download_datasets --all
# Download builtin-only (no network, uses embedded problems)
python3 -m benchmarks.download_datasets --all --builtin-only
# Download specific suites
python3 -m benchmarks.download_datasets --suite humaneval --suite mmlu-pro --suite gpqa-diamond
# Force re-download
python3 -m benchmarks.download_datasets --all --force
```
**Datasets with HuggingFace downloaders:**
HumanEval, GSM8K, MBPP, MATH, MMLU-Pro, GPQA-Diamond, BigBench-Hard, MMMLU, HLE
**Builtin-only datasets (no HuggingFace source):**
SWE-Bench, Aider, LiveCodeBench, AIME, IFEval, BFCL, Tau2, Codeforces
---
## Local Task Benchmarks
The original local benchmark system tests the agent on custom tasks.
```bash
# Run all local tasks
python3 -m benchmarks.run
# Run a single task
python3 -m benchmarks.run --task file-create-basic
# Run a category
python3 -m benchmarks.run --category bugfix
# Run by difficulty
python3 -m benchmarks.run --difficulty easy
# List available tasks
python3 -m benchmarks.run --list
# Verbose + save results
python3 -m benchmarks.run -v -o local_results.json
```
---
## All Commands Reference
```bash
# ─── STANDARD EVALUATION SUITES ────────────────────────────────────────
# List suites
python3 -m benchmarks.run_suite --list
# Individual suites
python3 -m benchmarks.run_suite --suite humaneval # Coding: docstring → code
python3 -m benchmarks.run_suite --suite mbpp # Coding: basic Python
python3 -m benchmarks.run_suite --suite swe-bench # Coding: GitHub issues
python3 -m benchmarks.run_suite --suite aider # Coding: code editing
python3 -m benchmarks.run_suite --suite livecodebench # Coding: competitive programming
python3 -m benchmarks.run_suite --suite codeforces # Coding: competitive + ELO
python3 -m benchmarks.run_suite --suite math # Math: competition math
python3 -m benchmarks.run_suite --suite gsm8k # Math: grade school
python3 -m benchmarks.run_suite --suite aime # Math: AIME competition
python3 -m benchmarks.run_suite --suite mmlu-pro # Knowledge: 14 subjects, 10 choices
python3 -m benchmarks.run_suite --suite gpqa-diamond # Knowledge: graduate science
python3 -m benchmarks.run_suite --suite mmmlu # Knowledge: multilingual MMLU
python3 -m benchmarks.run_suite --suite hle # Knowledge: Humanity's Last Exam
python3 -m benchmarks.run_suite --suite bigbench-hard # Reasoning: BIG-Bench Hard
python3 -m benchmarks.run_suite --suite tau2 # Reasoning: tool-augmented
python3 -m benchmarks.run_suite --suite ifeval # Instruction: format following
python3 -m benchmarks.run_suite --suite bfcl # Instruction: function calling
# Category runs
python3 -m benchmarks.run_suite --category coding # All coding
python3 -m benchmarks.run_suite --category math # All math
python3 -m benchmarks.run_suite --category knowledge # MMLU-Pro, GPQA, MMMLU, HLE
python3 -m benchmarks.run_suite --category reasoning # BigBench, Tau2
python3 -m benchmarks.run_suite --category instruction-following # IFEval, BFCL
# Full run
python3 -m benchmarks.run_suite --all # All suites (~108 problems)
python3 -m benchmarks.run_suite --all --limit 3 # Quick: 3 per suite
python3 -m benchmarks.run_suite --all -v -o results.json # Full + verbose + save
# Options
python3 -m benchmarks.run_suite --suite humaneval --limit 5 # Limit problems
python3 -m benchmarks.run_suite --suite humaneval --timeout 600 # Custom timeout (seconds)
python3 -m benchmarks.run_suite --suite humaneval --data-dir ./data # Custom data directory
python3 -m benchmarks.run_suite --suite humaneval -v # Verbose mode
python3 -m benchmarks.run_suite --suite humaneval -o out.json # Save to JSON
# ─── LOCAL TASK BENCHMARKS ─────────────────────────────────────────────
python3 -m benchmarks.run --list # List local tasks
python3 -m benchmarks.run # Run all local tasks
python3 -m benchmarks.run --task file-create-basic # Single task
python3 -m benchmarks.run --category bugfix # Category filter
python3 -m benchmarks.run --difficulty easy # Difficulty filter
python3 -m benchmarks.run -v -o local_results.json # Verbose + save
```
---
## Environment Variables
| Variable | Description | Example |
|----------|-------------|---------|
| `OPENAI_API_KEY` | API key for the model provider | `sk-...` |
| `OPENAI_MODEL` | Model name to use | `gpt-4`, `qwen2.5-coder-32b` |
| `OPENAI_BASE_URL` | API base URL (for local models) | `http://localhost:8000/v1` |
---
## Output Format
Results are saved as JSON with this structure:
```json
{
"benchmark_run": "claw-code-agent-evaluation",
"timestamp": "2025-01-15T10:30:00",
"suites": [
{
"suite_name": "HumanEval",
"total": 20,
"passed": 15,
"failed": 5,
"score_pct": 75.0,
"duration_sec": 450.5,
"model": "gpt-4",
"results": [
{
"problem_id": "HumanEval/0",
"passed": true,
"duration_sec": 12.3,
"error": ""
}
]
}
],
"summary": {
"total_suites": 10,
"total_problems": 108,
"total_passed": 85,
"overall_score_pct": 78.7
}
}
```
---
## Architecture
```
benchmarks/
├── __init__.py
├── run.py # Local task benchmark runner
├── run_suite.py # Standard evaluation suite runner (CLI)
├── download_datasets.py # Dataset downloader (HuggingFace + builtins)
├── README.md # This file
├── data/ # Dataset files (JSONL) — not committed
│ └── .gitkeep
├── tasks/
│ ├── __init__.py
│ └── definitions.py # Local task definitions
└── suites/
├── __init__.py
├── base.py # Base class for all suites
├── humaneval.py # HumanEval benchmark
├── mbpp.py # MBPP benchmark
├── swe_bench.py # SWE-Bench benchmark
├── aider.py # Aider benchmark
├── livecodebench.py # LiveCodeBench benchmark
├── codeforces.py # Codeforces (ELO scoring)
├── math_bench.py # MATH benchmark
├── gsm8k.py # GSM8K benchmark
├── aime.py # AIME benchmark
├── mmlu_pro.py # MMLU-Pro (10-choice QA)
├── gpqa.py # GPQA Diamond (science)
├── mmmlu.py # MMMLU (multilingual)
├── hle.py # HLE (Humanity's Last Exam)
├── bigbench.py # BigBench Extra Hard
├── tau2.py # Tau2 (tool-augmented)
├── ifeval.py # IFEval benchmark
└── bfcl.py # BFCL benchmark
```
benchmarks/__init__.py
-1
View File
@@ -1 +0,0 @@
"""Claw Code Agent local benchmark suite."""
benchmarks/data/.gitkeep
View File
benchmarks/download_datasets.py
-591
View File
@@ -1,591 +0,0 @@
#!/usr/bin/env python3
"""
Download or export benchmark datasets into benchmarks/data.
Uses the HuggingFace `datasets` library for reliable full downloads.
Falls back to the REST API or builtins if `datasets` is not installed.
"""
from __future__ import annotations
import argparse
import gzip
import json
import re
import urllib.parse
import urllib.request
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Any, Callable
from benchmarks.suites.aider import _BUILTIN_PROBLEMS as _AIDER_BUILTINS
from benchmarks.suites.aime import _BUILTIN_PROBLEMS as _AIME_BUILTINS
from benchmarks.suites.bfcl import _BUILTIN_PROBLEMS as _BFCL_BUILTINS
from benchmarks.suites.bigbench import _BUILTIN_PROBLEMS as _BIGBENCH_BUILTINS
from benchmarks.suites.codeforces import _BUILTIN_PROBLEMS as _CODEFORCES_BUILTINS
from benchmarks.suites.gpqa import _BUILTIN_PROBLEMS as _GPQA_BUILTINS
from benchmarks.suites.gsm8k import _BUILTIN_PROBLEMS as _GSM8K_BUILTINS
from benchmarks.suites.hle import _BUILTIN_PROBLEMS as _HLE_BUILTINS
from benchmarks.suites.humaneval import _BUILTIN_PROBLEMS as _HUMANEVAL_BUILTINS
from benchmarks.suites.ifeval import _BUILTIN_PROBLEMS as _IFEVAL_BUILTINS
from benchmarks.suites.livecodebench import _BUILTIN_PROBLEMS as _LIVECODEBENCH_BUILTINS
from benchmarks.suites.math_bench import _BUILTIN_PROBLEMS as _MATH_BUILTINS
from benchmarks.suites.mbpp import _BUILTIN_PROBLEMS as _MBPP_BUILTINS
from benchmarks.suites.mmmlu import _BUILTIN_PROBLEMS as _MMMLU_BUILTINS
from benchmarks.suites.mmlu_pro import _BUILTIN_PROBLEMS as _MMLU_PRO_BUILTINS
from benchmarks.suites.swe_bench import _BUILTIN_PROBLEMS as _SWE_BUILTINS
from benchmarks.suites.tau2 import _BUILTIN_PROBLEMS as _TAU2_BUILTINS
HUMANEVAL_GZ_URL = "https://raw.githubusercontent.com/openai/human-eval/master/data/HumanEval.jsonl.gz"
DEFAULT_DATA_DIR = Path(__file__).resolve().parent / "data"
# Legacy REST API support (fallback only)
HF_DATASET_VIEWER_BASE = "https://datasets-server.huggingface.co"
JsonFetcher = Callable[[str, dict[str, object], dict[str, str] | None, float], object]
@dataclass
class DownloadResult:
suite: str
rows: int
path: str
source: str
note: str = ""
def fetch_bytes(url: str, timeout: float, headers: dict[str, str] | None = None) -> bytes:
request = urllib.request.Request(url, headers=headers or {})
with urllib.request.urlopen(request, timeout=timeout) as response:
return response.read()
def fetch_json(
endpoint: str,
params: dict[str, object],
headers: dict[str, str] | None,
timeout: float,
) -> object:
query = urllib.parse.urlencode(params, doseq=True)
url = f"{HF_DATASET_VIEWER_BASE}/{endpoint}"
if query:
url = f"{url}?{query}"
raw = fetch_bytes(url, timeout=timeout, headers=headers)
return json.loads(raw.decode("utf-8"))
def _write_jsonl(path: Path, rows: list[dict[str, Any]]) -> int:
path.parent.mkdir(parents=True, exist_ok=True)
with open(path, "w", encoding="utf-8") as handle:
for row in rows:
handle.write(json.dumps(row, ensure_ascii=False) + "\n")
return len(rows)
# ---------------------------------------------------------------------------
# HuggingFace `datasets` library helpers
# ---------------------------------------------------------------------------
def _load_hf_dataset(
dataset_name: str,
config: str | None = None,
split: str = "test",
) -> list[dict[str, Any]]:
"""Load a dataset using the HuggingFace `datasets` library."""
from datasets import load_dataset # type: ignore[import-untyped]
kwargs: dict[str, Any] = {}
if config:
kwargs["name"] = config
ds = load_dataset(dataset_name, split=split, **kwargs)
return [dict(row) for row in ds] # type: ignore[union-attr]
def _try_load_hf(
dataset_name: str,
config: str | None = None,
split_preference: tuple[str, ...] = ("test", "validation", "train"),
) -> list[dict[str, Any]]:
"""Try loading with preferred splits, falling back through the list."""
for split in split_preference:
try:
rows = _load_hf_dataset(dataset_name, config=config, split=split)
if rows:
print(f" Loaded {len(rows)} rows from {dataset_name} [{split}]")
return rows
except (ValueError, KeyError):
continue
raise ValueError(f"No valid split found for {dataset_name}")
def _fetch_hf_rows(
dataset: str,
*,
config_preference: tuple[str, ...] = (),
split_preference: tuple[str, ...] = ("test", "validation", "train"),
json_fetcher: JsonFetcher = fetch_json,
timeout: float = 60.0,
headers: dict[str, str] | None = None,
) -> list[dict[str, Any]]:
"""Legacy REST-API fetcher (kept for backward compatibility with tests)."""
splits_payload = json_fetcher("splits", {"dataset": dataset}, headers, timeout)
splits = list((splits_payload or {}).get("splits", [])) # type: ignore[assignment]
if not splits:
return []
chosen: dict[str, Any] | None = None
for config_name in config_preference:
for split_name in split_preference:
chosen = next(
(
item for item in splits
if item.get("config") == config_name and item.get("split") == split_name
),
None,
)
if chosen is not None:
break
if chosen is not None:
break
if chosen is None:
for split_name in split_preference:
chosen = next((item for item in splits if item.get("split") == split_name), None)
if chosen is not None:
break
if chosen is None:
chosen = splits[0]
rows: list[dict[str, Any]] = []
offset = 0
length = 100
while True:
payload = json_fetcher(
"rows",
{
"dataset": chosen["dataset"],
"config": chosen["config"],
"split": chosen["split"],
"offset": offset,
"length": length,
},
headers,
timeout,
)
batch = [item.get("row", {}) for item in (payload or {}).get("rows", [])] # type: ignore[union-attr]
rows.extend(batch)
total = int((payload or {}).get("num_rows_total", len(rows))) # type: ignore[union-attr]
offset += len(batch)
if not batch or offset >= total:
break
return rows
# ---------------------------------------------------------------------------
# Answer extraction helpers
# ---------------------------------------------------------------------------
def _extract_gsm8k_answer(text: str) -> str:
if "####" in text:
text = text.split("####", 1)[1]
numbers = re.findall(r"-?\d[\d,]*\.?\d*", text.replace("$", ""))
if numbers:
return numbers[-1].replace(",", "")
return text.strip().replace(",", "")
def _extract_math_answer(solution: str) -> str:
boxed_fraction = re.search(r"\\boxed\{\\frac\{([^}]+)\}\{([^}]+)\}\}", solution, flags=re.DOTALL)
if boxed_fraction:
return f"{boxed_fraction.group(1).strip()}/{boxed_fraction.group(2).strip()}"
boxed = re.search(r"\\boxed\{([^{}]+)\}", solution, flags=re.DOTALL)
value = boxed.group(1) if boxed else solution
value = value.strip()
value = value.replace("\\frac{", "").replace("}{", "/").replace("}", "")
value = value.replace("$", "").replace(",", "").strip()
fraction = re.search(r"-?\d+\s*/\s*-?\d+", value)
if fraction:
return fraction.group(0).replace(" ", "")
numbers = re.findall(r"-?\d+(?:/\d+)?(?:\.\d+)?", value)
return numbers[-1] if numbers else value
# ---------------------------------------------------------------------------
# Individual dataset downloaders (using `datasets` library)
# ---------------------------------------------------------------------------
def _download_humaneval(output_path: Path, *, timeout: float) -> DownloadResult:
raw = fetch_bytes(HUMANEVAL_GZ_URL, timeout=timeout)
if raw[:2] == b"\x1f\x8b":
raw = gzip.decompress(raw)
lines = [json.loads(line) for line in raw.decode("utf-8").splitlines() if line.strip()]
rows = [
{
"task_id": item["task_id"],
"prompt": item["prompt"],
"canonical_solution": item.get("canonical_solution", ""),
"test": item["test"],
"entry_point": item["entry_point"],
}
for item in lines
]
count = _write_jsonl(output_path, rows)
return DownloadResult("humaneval", count, str(output_path), "official")
def _download_gsm8k(
output_path: Path,
*,
timeout: float,
json_fetcher: JsonFetcher | None = None,
) -> DownloadResult:
if json_fetcher is not None:
# Legacy path for tests
rows = _fetch_hf_rows(
"openai/gsm8k",
config_preference=("main",),
split_preference=("test",),
json_fetcher=json_fetcher,
timeout=timeout,
)
else:
rows = _try_load_hf("openai/gsm8k", config="main", split_preference=("test",))
normalized = [
{
"id": f"gsm8k-{index + 1:04d}",
"question": row["question"],
"answer": _extract_gsm8k_answer(str(row["answer"])),
}
for index, row in enumerate(rows)
]
count = _write_jsonl(output_path, normalized)
return DownloadResult("gsm8k", count, str(output_path), "official")
def _download_mbpp(output_path: Path, *, timeout: float) -> DownloadResult:
try:
rows = _try_load_hf("google-research-datasets/mbpp", config="sanitized", split_preference=("test", "validation"))
except Exception:
rows = _try_load_hf("google-research-datasets/mbpp", config="full", split_preference=("test", "validation"))
normalized = [
{
"task_id": row.get("task_id", index + 1),
"text": row.get("text") or row.get("prompt") or "",
"code": row.get("code", ""),
"test_list": row.get("test_list") or row.get("test_setup_code", []),
}
for index, row in enumerate(rows)
]
count = _write_jsonl(output_path, normalized)
return DownloadResult("mbpp", count, str(output_path), "official")
def _download_math(output_path: Path, *, timeout: float) -> DownloadResult:
rows = _try_load_hf("DigitalLearningGmbH/MATH-lighteval", split_preference=("test", "train"))
normalized = [
{
"id": row.get("problem_id", f"math-{index + 1:04d}"),
"problem": row.get("problem", ""),
"answer": _extract_math_answer(str(row.get("solution", row.get("answer", "")))),
"subject": row.get("type", row.get("subject", "unknown")),
"level": row.get("level", 0),
}
for index, row in enumerate(rows)
]
count = _write_jsonl(output_path, normalized)
return DownloadResult("math", count, str(output_path), "official")
def _download_mmlu_pro(output_path: Path, *, timeout: float) -> DownloadResult:
rows = _try_load_hf("TIGER-Lab/MMLU-Pro", split_preference=("test", "validation"))
letters = "ABCDEFGHIJ"
normalized = []
for index, row in enumerate(rows):
answer_raw = row.get("answer", "")
if isinstance(answer_raw, int) and answer_raw < len(letters):
answer = letters[answer_raw]
else:
answer = str(answer_raw)
normalized.append({
"id": f"mmlu-pro-{index + 1:04d}",
"subject": row.get("category", row.get("subject", "unknown")),
"question": row.get("question", ""),
"choices": row.get("options", row.get("choices", [])),
"answer": answer,
})
count = _write_jsonl(output_path, normalized)
return DownloadResult("mmlu-pro", count, str(output_path), "official")
def _download_gpqa(output_path: Path, *, timeout: float) -> DownloadResult:
rows = _try_load_hf("Idavidrein/gpqa", config="gpqa_diamond", split_preference=("train",))
normalized = []
for index, row in enumerate(rows):
choices = [
row.get("Correct Answer", ""),
row.get("Incorrect Answer 1", ""),
row.get("Incorrect Answer 2", ""),
row.get("Incorrect Answer 3", ""),
]
normalized.append({
"id": f"gpqa-{index + 1:04d}",
"subject": row.get("Subdomain", row.get("domain", "science")),
"question": row.get("Question", ""),
"choices": choices,
"answer": "A",
})
count = _write_jsonl(output_path, normalized)
return DownloadResult("gpqa-diamond", count, str(output_path), "official")
def _download_bigbench_hard(output_path: Path, *, timeout: float) -> DownloadResult:
from datasets import load_dataset # type: ignore[import-untyped]
configs = [
"boolean_expressions", "causal_judgement", "date_understanding",
"disambiguation_qa", "dyck_languages", "formal_fallacies",
"geometric_shapes", "hyperbaton", "logical_deduction_three_objects",
"logical_deduction_five_objects", "logical_deduction_seven_objects",
"movie_recommendation", "multistep_arithmetic_two", "navigate",
"object_counting", "penguins_in_a_table",
"reasoning_about_colored_objects", "ruin_names",
"salient_translation_error_detection", "snarks",
"sports_understanding", "temporal_sequences",
"tracking_shuffled_objects_three_objects",
"tracking_shuffled_objects_five_objects",
"tracking_shuffled_objects_seven_objects",
"web_of_lies", "word_sorting",
]
all_rows: list[dict[str, Any]] = []
for config in configs:
try:
ds = load_dataset("lukaemon/bbh", config, split="test")
for row in ds:
row_dict = dict(row) # type: ignore[arg-type]
row_dict["task"] = config
all_rows.append(row_dict)
except Exception:
continue
print(f" Loaded {len(all_rows)} rows from lukaemon/bbh [{len(configs)} tasks]")
normalized = []
for index, row in enumerate(all_rows):
target = row.get("target", row.get("answer", ""))
normalized.append({
"id": f"bbh-{index + 1:05d}",
"task": row.get("task", "unknown"),
"question": row.get("input", row.get("question", "")),
"choices": [],
"answer": str(target).strip(),
})
count = _write_jsonl(output_path, normalized)
return DownloadResult("bigbench-hard", count, str(output_path), "official")
def _download_mmmlu(output_path: Path, *, timeout: float) -> DownloadResult:
rows = _try_load_hf("openai/MMMLU", split_preference=("test", "validation"))
letters = "ABCD"
normalized = []
for index, row in enumerate(rows):
answer_raw = row.get("answer", "")
if isinstance(answer_raw, int) and answer_raw < len(letters):
answer = letters[answer_raw]
else:
answer = str(answer_raw)
normalized.append({
"id": f"mmmlu-{index + 1:04d}",
"language": row.get("language", "en"),
"subject": row.get("subject", "unknown"),
"question": row.get("question", ""),
"choices": row.get("choices", row.get("options", [])),
"answer": answer,
})
count = _write_jsonl(output_path, normalized)
return DownloadResult("mmmlu", count, str(output_path), "official")
def _download_hle(output_path: Path, *, timeout: float) -> DownloadResult:
rows = _try_load_hf("cais/hle", split_preference=("test", "validation", "train"))
normalized = []
for index, row in enumerate(rows):
entry: dict[str, Any] = {
"id": f"hle-{index + 1:04d}",
"subject": row.get("category", row.get("subject", "general")),
"question": row.get("question", ""),
"answer": str(row.get("answer", "")),
}
if row.get("choices") or row.get("options"):
entry["answer_type"] = "multiple_choice"
entry["choices"] = row.get("choices", row.get("options", []))
else:
entry["answer_type"] = "exact"
normalized.append(entry)
count = _write_jsonl(output_path, normalized)
return DownloadResult("hle", count, str(output_path), "official")
def _export_builtin(output_path: Path, suite: str, rows: list[dict[str, Any]], *, source: str = "builtin", note: str = "") -> DownloadResult:
count = _write_jsonl(output_path, rows)
return DownloadResult(suite, count, str(output_path), source, note)
def _builtin_rows(suite: str) -> list[dict[str, Any]]:
mapping: dict[str, list[dict[str, Any]]] = {
"humaneval": list(_HUMANEVAL_BUILTINS),
"mbpp": list(_MBPP_BUILTINS),
"gsm8k": list(_GSM8K_BUILTINS),
"math": list(_MATH_BUILTINS),
"swe-bench": list(_SWE_BUILTINS),
"aider": list(_AIDER_BUILTINS),
"livecodebench": list(_LIVECODEBENCH_BUILTINS),
"aime": list(_AIME_BUILTINS),
"ifeval": list(_IFEVAL_BUILTINS),
"bfcl": list(_BFCL_BUILTINS),
"mmlu-pro": list(_MMLU_PRO_BUILTINS),
"gpqa-diamond": list(_GPQA_BUILTINS),
"bigbench-hard": list(_BIGBENCH_BUILTINS),
"mmmlu": list(_MMMLU_BUILTINS),
"hle": list(_HLE_BUILTINS),
"tau2": list(_TAU2_BUILTINS),
"codeforces": list(_CODEFORCES_BUILTINS),
}
return list(mapping[suite])
def prepare_suite(
suite: str,
*,
data_dir: Path,
force: bool,
builtin_only: bool,
official_only: bool,
timeout: float,
) -> DownloadResult:
output_map = {
"mmlu-pro": "mmlu_pro.jsonl",
"gpqa-diamond": "gpqa.jsonl",
"bigbench-hard": "bigbench_hard.jsonl",
"mmmlu": "mmmlu.jsonl",
"hle": "hle.jsonl",
"tau2": "tau2.jsonl",
"codeforces": "codeforces.jsonl",
}
output_path = data_dir / output_map.get(suite, f"{suite}.jsonl")
if output_path.exists() and not force:
lines = [line for line in output_path.read_text(encoding="utf-8").splitlines() if line.strip()]
return DownloadResult(suite, len(lines), str(output_path), "existing")
data_dir.mkdir(parents=True, exist_ok=True)
builtin_only_suites = {"swe-bench", "aider", "livecodebench", "aime", "ifeval", "bfcl", "tau2", "codeforces"}
official_downloaders = {
"humaneval": _download_humaneval,
"gsm8k": _download_gsm8k,
"mbpp": _download_mbpp,
"math": _download_math,
"mmlu-pro": _download_mmlu_pro,
"gpqa-diamond": _download_gpqa,
"bigbench-hard": _download_bigbench_hard,
"mmmlu": _download_mmmlu,
"hle": _download_hle,
}
if suite in builtin_only_suites or builtin_only:
return _export_builtin(output_path, suite, _builtin_rows(suite))
downloader = official_downloaders.get(suite)
if downloader is None:
return _export_builtin(output_path, suite, _builtin_rows(suite))
try:
return downloader(output_path, timeout=timeout)
except Exception as exc:
if official_only:
raise
note = f"official download failed: {exc}"
print(f" WARNING: {note}")
print(f" Falling back to {len(_builtin_rows(suite))} built-in problems.")
print(f" To get full data, install `datasets`: pip install datasets")
return _export_builtin(
output_path,
suite,
_builtin_rows(suite),
source="builtin-fallback",
note=note,
)
def _write_manifest(data_dir: Path, results: list[DownloadResult]) -> Path:
manifest_path = data_dir / "manifest.json"
payload = {
"generated_at": __import__("time").strftime("%Y-%m-%dT%H:%M:%S"),
"results": [asdict(item) for item in results],
}
manifest_path.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
return manifest_path
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description="Download benchmark datasets for claw-code-agent.")
parser.add_argument("--suite", action="append", default=[], help="Suite to prepare. Can be repeated.")
parser.add_argument("--all", action="store_true", help="Prepare all known suites.")
parser.add_argument("--list", action="store_true", help="List known suites.")
parser.add_argument("--data-dir", type=Path, default=DEFAULT_DATA_DIR, help="Output data directory.")
parser.add_argument("--force", action="store_true", help="Overwrite existing files.")
parser.add_argument("--builtin-only", action="store_true", help="Skip official downloads and export builtins only.")
parser.add_argument("--official-only", action="store_true", help="Do not fall back to builtins.")
parser.add_argument("--timeout", type=float, default=60.0, help="Network timeout in seconds.")
return parser
def main() -> None:
parser = build_parser()
args = parser.parse_args()
known = [
"humaneval",
"mbpp",
"gsm8k",
"math",
"swe-bench",
"aider",
"livecodebench",
"aime",
"ifeval",
"bfcl",
"mmlu-pro",
"gpqa-diamond",
"bigbench-hard",
"mmmlu",
"hle",
"tau2",
"codeforces",
]
if args.list:
for name in known:
print(name)
return
suites = list(args.suite)
if args.all:
suites = known
if not suites:
parser.error("specify --suite or --all")
results = [
prepare_suite(
suite,
data_dir=args.data_dir,
force=args.force,
builtin_only=args.builtin_only,
official_only=args.official_only,
timeout=args.timeout,
)
for suite in suites
]
manifest = _write_manifest(args.data_dir, results)
print(f"Wrote {len(results)} suite files to {args.data_dir}")
print(f"Manifest: {manifest}")
if __name__ == "__main__":
main()
benchmarks/run.py
-353
View File
@@ -1,353 +0,0 @@
#!/usr/bin/env python3
"""
Local benchmark runner for claw-code-agent.
Runs the REAL agent binary against a suite of coding tasks and scores
pass/fail automatically. No Docker required.
Usage:
# Run all tasks
python3 -m benchmarks.run
# Run a single task
python3 -m benchmarks.run --task file-create-basic
# Run a category
python3 -m benchmarks.run --category bugfix
# Run a difficulty level
python3 -m benchmarks.run --difficulty easy
# List available tasks
python3 -m benchmarks.run --list
# Verbose output
python3 -m benchmarks.run --verbose
"""
from __future__ import annotations
import argparse
import json
import os
import shutil
import subprocess
import sys
import tempfile
import time
from dataclasses import dataclass
from pathlib import Path
from benchmarks.tasks.definitions import TASKS, BenchmarkTask, get_task
from benchmarks.suites.base import make_temp_workspace
# ---------------------------------------------------------------------------
# Result tracking
# ---------------------------------------------------------------------------
@dataclass
class TaskResult:
task_id: str
category: str
difficulty: str
passed: bool
duration_sec: float
agent_exit_code: int
verify_exit_code: int
error: str = ""
# ---------------------------------------------------------------------------
# Task execution
# ---------------------------------------------------------------------------
def _run_shell(cmd: str, cwd: str, timeout: float = 30.0) -> tuple[int, str]:
"""Run a shell command, return (exit_code, combined_output)."""
try:
proc = subprocess.run(
cmd,
shell=True,
cwd=cwd,
capture_output=True,
text=True,
timeout=timeout,
)
output = (proc.stdout + proc.stderr).strip()
return proc.returncode, output
except subprocess.TimeoutExpired:
return 1, f"[TIMEOUT after {timeout}s]"
except Exception as exc:
return 1, str(exc)
def run_task(
task: BenchmarkTask,
*,
project_root: str,
agent_timeout: float = 300.0,
verbose: bool = False,
) -> TaskResult:
"""Run a single benchmark task end-to-end."""
# Create isolated temp workspace
workspace = make_temp_workspace("claw_bench", task.category, task.id)
if verbose:
print(f" workspace: {workspace}")
try:
# --- SETUP ---
if task.setup:
code, out = _run_shell(task.setup, cwd=workspace)
if code != 0:
return TaskResult(
task_id=task.id,
category=task.category,
difficulty=task.difficulty,
passed=False,
duration_sec=0.0,
agent_exit_code=-1,
verify_exit_code=-1,
error=f"Setup failed: {out}",
)
# --- RUN AGENT ---
agent_cmd = (
f"{sys.executable} -m src.main agent "
f"{_shell_quote(task.instruction)} "
f"--cwd {_shell_quote(workspace)} "
f"--allow-write "
f"--allow-shell"
)
if verbose:
print(f" agent cmd: {agent_cmd[:120]}...")
start = time.time()
agent_code, agent_out = _run_shell(
agent_cmd,
cwd=project_root,
timeout=agent_timeout,
)
duration = time.time() - start
if verbose:
print(f" agent exit={agent_code} duration={duration:.1f}s")
if agent_out:
# Print last few lines of agent output
lines = agent_out.split("\n")
for line in lines[-5:]:
print(f" > {line}")
# --- VERIFY ---
verify_code, verify_out = _run_shell(task.verify, cwd=workspace, timeout=30.0)
if verbose:
status = "PASS" if verify_code == 0 else "FAIL"
print(f" verify exit={verify_code} -> {status}")
if verify_code != 0 and verify_out:
print(f" verify output: {verify_out[:200]}")
return TaskResult(
task_id=task.id,
category=task.category,
difficulty=task.difficulty,
passed=(verify_code == 0),
duration_sec=duration,
agent_exit_code=agent_code,
verify_exit_code=verify_code,
error=verify_out if verify_code != 0 else "",
)
finally:
# Clean up workspace
shutil.rmtree(workspace, ignore_errors=True)
def _shell_quote(s: str) -> str:
"""Quote a string for shell use."""
import shlex
return shlex.quote(s)
# ---------------------------------------------------------------------------
# Reporting
# ---------------------------------------------------------------------------
def print_results(results: list[TaskResult]) -> None:
"""Print a formatted results table."""
total = len(results)
passed = sum(1 for r in results if r.passed)
failed = total - passed
print()
print("=" * 72)
print(" CLAW CODE AGENT — BENCHMARK RESULTS")
print("=" * 72)
print()
print(f" {'Task ID':<30} {'Category':<12} {'Diff':<8} {'Result':<8} {'Time':>6}")
print(f" {'─' * 30} {'─' * 12} {'─' * 8} {'─' * 8} {'─' * 6}")
for r in results:
status = "PASS" if r.passed else "FAIL"
icon = " ✅" if r.passed else " ❌"
time_str = f"{r.duration_sec:.1f}s"
print(f"{icon} {r.task_id:<30} {r.category:<12} {r.difficulty:<8} {status:<8} {time_str:>6}")
print()
print("─" * 72)
print(f" Total: {total} | Passed: {passed} | Failed: {failed} | Score: {passed}/{total} ({100*passed/total:.0f}%)")
print("─" * 72)
# Breakdown by category
categories: dict[str, list[TaskResult]] = {}
for r in results:
categories.setdefault(r.category, []).append(r)
print()
print(" Category Breakdown:")
for cat, cat_results in sorted(categories.items()):
cat_passed = sum(1 for r in cat_results if r.passed)
cat_total = len(cat_results)
bar = "█" * cat_passed + "░" * (cat_total - cat_passed)
print(f" {cat:<14} {bar} {cat_passed}/{cat_total}")
# Breakdown by difficulty
difficulties: dict[str, list[TaskResult]] = {}
for r in results:
difficulties.setdefault(r.difficulty, []).append(r)
print()
print(" Difficulty Breakdown:")
for diff in ("easy", "medium", "hard"):
if diff in difficulties:
diff_results = difficulties[diff]
diff_passed = sum(1 for r in diff_results if r.passed)
diff_total = len(diff_results)
print(f" {diff:<14} {diff_passed}/{diff_total} ({100*diff_passed/diff_total:.0f}%)")
total_time = sum(r.duration_sec for r in results)
print()
print(f" Total time: {total_time:.1f}s")
print()
def save_results(results: list[TaskResult], output_path: str) -> None:
"""Save results to JSON."""
data = {
"benchmark": "claw-code-agent-local",
"timestamp": time.strftime("%Y-%m-%dT%H:%M:%S"),
"model": os.environ.get("OPENAI_MODEL", "unknown"),
"total": len(results),
"passed": sum(1 for r in results if r.passed),
"score_pct": round(100 * sum(1 for r in results if r.passed) / len(results), 1) if results else 0,
"results": [
{
"task_id": r.task_id,
"category": r.category,
"difficulty": r.difficulty,
"passed": r.passed,
"duration_sec": round(r.duration_sec, 2),
"agent_exit_code": r.agent_exit_code,
"verify_exit_code": r.verify_exit_code,
"error": r.error,
}
for r in results
],
}
with open(output_path, "w") as f:
json.dump(data, f, indent=2)
print(f" Results saved to {output_path}")
# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def main() -> None:
parser = argparse.ArgumentParser(description="Claw Code Agent local benchmark")
parser.add_argument("--task", help="Run a single task by ID")
parser.add_argument("--category", help="Run tasks in a category")
parser.add_argument("--difficulty", choices=["easy", "medium", "hard"], help="Run tasks by difficulty")
parser.add_argument("--list", action="store_true", help="List available tasks")
parser.add_argument("--verbose", "-v", action="store_true", help="Verbose output")
parser.add_argument("--timeout", type=float, default=300.0, help="Agent timeout per task in seconds (default: 300)")
parser.add_argument("--output", "-o", help="Save results to JSON file")
args = parser.parse_args()
if args.list:
print(f"\n {'ID':<30} {'Category':<12} {'Difficulty':<10}")
print(f" {'─' * 30} {'─' * 12} {'─' * 10}")
for t in TASKS:
print(f" {t.id:<30} {t.category:<12} {t.difficulty:<10}")
print(f"\n Total: {len(TASKS)} tasks\n")
return
# Select tasks
tasks_to_run: list[BenchmarkTask] = []
if args.task:
t = get_task(args.task)
if t is None:
print(f"Unknown task: {args.task}")
print("Use --list to see available tasks")
sys.exit(1)
tasks_to_run = [t]
else:
tasks_to_run = list(TASKS)
if args.category:
tasks_to_run = [t for t in tasks_to_run if t.category == args.category]
if args.difficulty:
tasks_to_run = [t for t in tasks_to_run if t.difficulty == args.difficulty]
if not tasks_to_run:
print("No tasks matched the filters.")
sys.exit(1)
# Find project root
project_root = str(Path(__file__).resolve().parent.parent)
# Check environment
model = os.environ.get("OPENAI_MODEL", "not set")
base_url = os.environ.get("OPENAI_BASE_URL", "not set")
print()
print("=" * 72)
print(" CLAW CODE AGENT — LOCAL BENCHMARK")
print("=" * 72)
print(f" Model: {model}")
print(f" Base URL: {base_url}")
print(f" Tasks: {len(tasks_to_run)}")
print(f" Timeout: {args.timeout}s per task")
print("=" * 72)
print()
# Run tasks
results: list[TaskResult] = []
for i, task in enumerate(tasks_to_run, 1):
print(f"[{i}/{len(tasks_to_run)}] {task.id} ({task.category}, {task.difficulty})")
result = run_task(
task,
project_root=project_root,
agent_timeout=args.timeout,
verbose=args.verbose,
)
results.append(result)
status = "PASS ✅" if result.passed else "FAIL ❌"
print(f" -> {status} ({result.duration_sec:.1f}s)")
print()
# Report
print_results(results)
if args.output:
save_results(results, args.output)
if __name__ == "__main__":
main()
benchmarks/run_suite.py
-278
View File
@@ -1,278 +0,0 @@
#!/usr/bin/env python3
"""
Standard evaluation benchmark runner for claw-code-agent.
Runs the agent against well-known evaluation suites and reports scores.
Usage:
# List available benchmark suites
python3 -m benchmarks.run_suite --list
# Run a specific suite (built-in subset)
python3 -m benchmarks.run_suite --suite humaneval
python3 -m benchmarks.run_suite --suite mbpp
python3 -m benchmarks.run_suite --suite swe-bench
python3 -m benchmarks.run_suite --suite aider
python3 -m benchmarks.run_suite --suite livecodebench
python3 -m benchmarks.run_suite --suite math
python3 -m benchmarks.run_suite --suite gsm8k
python3 -m benchmarks.run_suite --suite aime
python3 -m benchmarks.run_suite --suite ifeval
python3 -m benchmarks.run_suite --suite bfcl
# Run ALL suites
python3 -m benchmarks.run_suite --all
# Run by category
python3 -m benchmarks.run_suite --category coding
python3 -m benchmarks.run_suite --category math
python3 -m benchmarks.run_suite --category instruction-following
# Limit problems per suite (for quick testing)
python3 -m benchmarks.run_suite --suite humaneval --limit 5
# Verbose output + save results
python3 -m benchmarks.run_suite --suite humaneval -v -o results.json
"""
from __future__ import annotations
import argparse
import json
import sys
import time
from benchmarks.suites.base import BenchmarkSuite, SuiteReport
# Import all suites
from benchmarks.suites.humaneval import HumanEvalBenchmark
from benchmarks.suites.mbpp import MBPPBenchmark
from benchmarks.suites.swe_bench import SWEBenchBenchmark
from benchmarks.suites.aider import AiderBenchmark
from benchmarks.suites.livecodebench import LiveCodeBenchBenchmark
from benchmarks.suites.math_bench import MATHBenchmark
from benchmarks.suites.gsm8k import GSM8KBenchmark
from benchmarks.suites.aime import AIMEBenchmark
from benchmarks.suites.ifeval import IFEvalBenchmark
from benchmarks.suites.bfcl import BFCLBenchmark
from benchmarks.suites.mmlu_pro import MMLUProBenchmark
from benchmarks.suites.gpqa import GPQABenchmark
from benchmarks.suites.bigbench import BigBenchHardBenchmark
from benchmarks.suites.mmmlu import MMMMLUBenchmark
from benchmarks.suites.hle import HLEBenchmark
from benchmarks.suites.tau2 import Tau2Benchmark
from benchmarks.suites.codeforces import CodeforcesBenchmark
# ---------------------------------------------------------------------------
# Registry
# ---------------------------------------------------------------------------
SUITE_REGISTRY: dict[str, type[BenchmarkSuite]] = {
"humaneval": HumanEvalBenchmark,
"mbpp": MBPPBenchmark,
"swe-bench": SWEBenchBenchmark,
"aider": AiderBenchmark,
"livecodebench": LiveCodeBenchBenchmark,
"math": MATHBenchmark,
"gsm8k": GSM8KBenchmark,
"aime": AIMEBenchmark,
"ifeval": IFEvalBenchmark,
"bfcl": BFCLBenchmark,
"mmlu-pro": MMLUProBenchmark,
"gpqa-diamond": GPQABenchmark,
"bigbench-hard": BigBenchHardBenchmark,
"mmmlu": MMMMLUBenchmark,
"hle": HLEBenchmark,
"tau2": Tau2Benchmark,
"codeforces": CodeforcesBenchmark,
}
CATEGORY_MAP: dict[str, list[str]] = {
"coding": ["humaneval", "mbpp", "swe-bench", "aider", "livecodebench", "codeforces"],
"math": ["math", "gsm8k", "aime"],
"instruction-following": ["ifeval", "bfcl"],
"knowledge": ["mmlu-pro", "gpqa-diamond", "mmmlu", "hle"],
"reasoning": ["bigbench-hard", "tau2"],
}
# ---------------------------------------------------------------------------
# Multi-suite reporting
# ---------------------------------------------------------------------------
def print_combined_report(reports: list[SuiteReport]) -> None:
"""Print a combined summary of all suite runs."""
print()
print("=" * 80)
print(" COMBINED BENCHMARK REPORT")
print("=" * 80)
print()
print(f" {'Suite':<20} {'Category':<22} {'Passed':>8} {'Total':>8} {'Score':>8} {'Time':>8}")
print(f" {'─' * 20} {'─' * 22} {'─' * 8} {'─' * 8} {'─' * 8} {'─' * 8}")
total_passed = 0
total_problems = 0
total_time = 0.0
for r in reports:
suite_cls = SUITE_REGISTRY.get(r.suite_name.lower().replace("-", "").replace("_", ""))
cat = suite_cls.category if suite_cls else "unknown"
print(
f" {r.suite_name:<20} {cat:<22} "
f"{r.passed:>8} {r.total:>8} "
f"{r.score_pct:>7.1f}% {r.duration_sec:>7.1f}s"
)
total_passed += r.passed
total_problems += r.total
total_time += r.duration_sec
print()
print("─" * 80)
overall_pct = round(100.0 * total_passed / total_problems, 1) if total_problems else 0.0
print(
f" OVERALL: {total_passed}/{total_problems} ({overall_pct}%) "
f"in {total_time:.1f}s"
)
print("─" * 80)
# Category breakdown
print()
print(" Category Breakdown:")
for cat_name, suite_names in CATEGORY_MAP.items():
cat_reports = [r for r in reports if r.suite_name.lower().replace("-", "").replace("_", "") in
[s.replace("-", "").replace("_", "") for s in suite_names]]
if cat_reports:
cp = sum(r.passed for r in cat_reports)
ct = sum(r.total for r in cat_reports)
cpct = round(100.0 * cp / ct, 1) if ct else 0.0
bar_len = 20
filled = round(bar_len * cp / ct) if ct else 0
bar = "█" * filled + "░" * (bar_len - filled)
print(f" {cat_name:<22} {bar} {cp}/{ct} ({cpct}%)")
print()
def save_combined_report(reports: list[SuiteReport], path: str) -> None:
data = {
"benchmark_run": "claw-code-agent-evaluation",
"timestamp": time.strftime("%Y-%m-%dT%H:%M:%S"),
"suites": [r.to_dict() for r in reports],
"summary": {
"total_suites": len(reports),
"total_problems": sum(r.total for r in reports),
"total_passed": sum(r.passed for r in reports),
"overall_score_pct": round(
100.0 * sum(r.passed for r in reports) / sum(r.total for r in reports), 1
) if any(r.total for r in reports) else 0.0,
},
}
with open(path, "w") as fh:
json.dump(data, fh, indent=2)
print(f" Combined results saved to {path}")
# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def main() -> None:
parser = argparse.ArgumentParser(
description="Run standard evaluation benchmarks against claw-code-agent",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=(
"Examples:\n"
" python3 -m benchmarks.run_suite --list\n"
" python3 -m benchmarks.run_suite --suite humaneval\n"
" python3 -m benchmarks.run_suite --suite humaneval --limit 5 -v\n"
" python3 -m benchmarks.run_suite --category coding\n"
" python3 -m benchmarks.run_suite --all\n"
" python3 -m benchmarks.run_suite --all -o results.json\n"
),
)
parser.add_argument("--suite", action="append", default=[],
help="Run a specific benchmark suite (can be repeated)")
parser.add_argument("--category", choices=list(CATEGORY_MAP.keys()),
help="Run all suites in a category")
parser.add_argument("--all", action="store_true",
help="Run ALL benchmark suites")
parser.add_argument("--list", action="store_true",
help="List available benchmark suites")
parser.add_argument("--limit", type=int, default=None,
help="Max problems per suite (for quick testing)")
parser.add_argument("--timeout", type=float, default=300.0,
help="Agent timeout per problem in seconds (default: 300)")
parser.add_argument("--verbose", "-v", action="store_true",
help="Verbose output")
parser.add_argument("--output", "-o",
help="Save results to JSON file")
parser.add_argument("--data-dir",
help="Directory containing dataset files (JSONL)")
parser.add_argument("--artifacts-dir",
help="Directory where per-problem artifacts will be saved")
parser.add_argument("--save-passing-artifacts", action="store_true",
help="Also save artifacts for passing problems")
args = parser.parse_args()
if args.list:
print()
print(" Available Benchmark Suites:")
print(" " + "─" * 68)
print(f" {'Name':<18} {'Category':<22} {'Description'}")
print(" " + "─" * 68)
for name, cls in SUITE_REGISTRY.items():
print(f" {name:<18} {cls.category:<22} {cls.description}")
print()
print(" Categories:")
for cat, suites in CATEGORY_MAP.items():
print(f" {cat}: {', '.join(suites)}")
print()
return
# Determine which suites to run
suite_names: list[str] = []
if args.all:
suite_names = list(SUITE_REGISTRY.keys())
elif args.category:
suite_names = CATEGORY_MAP.get(args.category, [])
elif args.suite:
for s in args.suite:
if s not in SUITE_REGISTRY:
print(f"Error: unknown suite '{s}'. Available: {', '.join(SUITE_REGISTRY)}")
sys.exit(1)
suite_names = list(args.suite)
else:
parser.print_help()
print("\nError: specify --suite, --category, or --all")
sys.exit(1)
if not suite_names:
print("No suites matched.")
sys.exit(1)
# Run suites
reports: list[SuiteReport] = []
for name in suite_names:
cls = SUITE_REGISTRY[name]
suite = cls(
data_dir=args.data_dir,
limit=args.limit,
agent_timeout=args.timeout,
verbose=args.verbose,
artifacts_dir=args.artifacts_dir,
save_passing_artifacts=args.save_passing_artifacts,
)
report = suite.run_all()
reports.append(report)
# Combined report
if len(reports) > 1:
print_combined_report(reports)
if args.output:
save_combined_report(reports, args.output)
if __name__ == "__main__":
main()
benchmarks/run_terminal_bench_local.py
-541
View File
@@ -1,541 +0,0 @@
#!/usr/bin/env python3
"""
Run Terminal-Bench tasks locally with Apptainer and claw-code-agent.
This runner is designed for cluster setups where Docker is unavailable but
Apptainer is available and the model server runs on the same node.
"""
from __future__ import annotations
import argparse
import fnmatch
import json
import os
import re
import shlex
import shutil
import subprocess
import sys
import time
import tomllib
from dataclasses import asdict, dataclass, field
from pathlib import Path
from typing import Any
DEFAULT_TASKS_DIR = Path.home() / ".cache/harbor/tasks/packages/terminal-bench"
DEFAULT_JOBS_DIR = Path("jobs/terminal_bench_local")
@dataclass
class TerminalBenchTask:
task_dir: Path
name: str
short_name: str
instruction: str
docker_image: str | None
agent_timeout_sec: float | None
verifier_timeout_sec: float
workdir: str
has_docker_compose: bool
raw_config: dict[str, Any] = field(default_factory=dict)
@dataclass
class LocalTrialResult:
task_name: str
short_name: str
passed: bool
reward: float | None
duration_sec: float
status: str
image: str | None
workdir: str
trial_dir: str
error: str = ""
agent_return_code: int | None = None
verifier_return_code: int | None = None
metadata: dict[str, Any] = field(default_factory=dict)
def _load_toml(path: Path) -> dict[str, Any]:
return tomllib.loads(path.read_text(encoding="utf-8"))
def strip_canary(text: str) -> str:
lines = text.splitlines()
index = 0
while index < len(lines):
stripped = lines[index].strip()
if stripped.startswith("<!--") and "canary" in stripped.lower():
index += 1
continue
if stripped.startswith("#") and "canary" in stripped.lower():
index += 1
continue
break
while index < len(lines) and not lines[index].strip():
index += 1
return "\n".join(lines[index:])
def parse_dockerfile_workdir(dockerfile_path: Path) -> str:
if not dockerfile_path.exists():
return "/workspace"
workdir = "/workspace"
pattern = re.compile(r"^\s*WORKDIR\s+(.+?)\s*$", re.IGNORECASE)
for raw_line in dockerfile_path.read_text(encoding="utf-8").splitlines():
line = raw_line.strip()
if not line or line.startswith("#"):
continue
match = pattern.match(line)
if not match:
continue
value = match.group(1).strip()
if value.startswith("["):
continue
if value.startswith('"') and value.endswith('"'):
value = value[1:-1]
workdir = value
return workdir
def load_task(task_dir: Path) -> TerminalBenchTask:
config = _load_toml(task_dir / "task.toml")
task_config = config.get("task") or {}
environment = config.get("environment") or {}
agent = config.get("agent") or {}
verifier = config.get("verifier") or {}
name = task_config.get("name", task_dir.name)
short_name = name.split("/", 1)[-1]
instruction = strip_canary((task_dir / "instruction.md").read_text(encoding="utf-8"))
environment_dir = task_dir / "environment"
return TerminalBenchTask(
task_dir=task_dir,
name=name,
short_name=short_name,
instruction=instruction,
docker_image=environment.get("docker_image"),
agent_timeout_sec=agent.get("timeout_sec"),
verifier_timeout_sec=float(verifier.get("timeout_sec", 600.0)),
workdir=parse_dockerfile_workdir(environment_dir / "Dockerfile"),
has_docker_compose=(environment_dir / "docker-compose.yaml").exists(),
raw_config=config,
)
def discover_tasks(root: Path) -> list[TerminalBenchTask]:
if (root / "task.toml").exists():
return [load_task(root)]
tasks: list[TerminalBenchTask] = []
for config_path in sorted(root.rglob("task.toml")):
try:
tasks.append(load_task(config_path.parent))
except Exception:
continue
return tasks
def filter_tasks(
tasks: list[TerminalBenchTask],
*,
include_patterns: list[str],
exclude_patterns: list[str],
limit: int | None,
) -> list[TerminalBenchTask]:
selected: list[TerminalBenchTask] = []
for task in tasks:
candidates = {task.name, task.short_name, task.task_dir.name}
if include_patterns and not any(
fnmatch.fnmatchcase(candidate, pattern)
for pattern in include_patterns
for candidate in candidates
):
continue
if any(
fnmatch.fnmatchcase(candidate, pattern)
for pattern in exclude_patterns
for candidate in candidates
):
continue
selected.append(task)
if limit is not None:
selected = selected[:limit]
return selected
def run_shell(
cmd: str,
*,
timeout: float | None = None,
env: dict[str, str] | None = None,
) -> subprocess.CompletedProcess[str]:
return subprocess.run(
cmd,
shell=True,
text=True,
capture_output=True,
timeout=timeout,
env=env,
)
def safe_name(value: str) -> str:
return re.sub(r"[^A-Za-z0-9_.-]+", "_", value).strip("._") or "item"
def ensure_apptainer_image(task: TerminalBenchTask, images_dir: Path, force_pull: bool) -> Path:
if not task.docker_image:
raise ValueError("task has no docker_image")
images_dir.mkdir(parents=True, exist_ok=True)
image_path = images_dir / f"{safe_name(task.name)}.sif"
if image_path.exists() and not force_pull:
return image_path
cmd = f"apptainer pull --force {shlex.quote(str(image_path))} docker://{shlex.quote(task.docker_image)}"
result = run_shell(cmd, timeout=3600.0)
if result.returncode != 0:
raise RuntimeError(
f"apptainer pull failed for {task.docker_image}\nstdout:\n{result.stdout}\nstderr:\n{result.stderr}"
)
return image_path
def seed_workspace(task: TerminalBenchTask, image_path: Path, workspace_dir: Path) -> None:
workspace_dir.mkdir(parents=True, exist_ok=True)
marker = workspace_dir / ".seed_complete"
if marker.exists():
return
copy_cmd = (
"set -euo pipefail; "
f"if [ -d {shlex.quote(task.workdir)} ]; then "
f"mkdir -p /mnt/workspace && cp -a {shlex.quote(task.workdir)}/. /mnt/workspace/; "
"fi"
)
cmd = (
f"apptainer exec --bind {shlex.quote(str(workspace_dir))}:/mnt/workspace:rw "
f"{shlex.quote(str(image_path))} "
f"bash -lc {shlex.quote(copy_cmd)}"
)
result = run_shell(cmd, timeout=1800.0)
if result.returncode != 0:
raise RuntimeError(
f"failed to seed workspace from {task.workdir}\nstdout:\n{result.stdout}\nstderr:\n{result.stderr}"
)
marker.write_text("ok\n", encoding="utf-8")
def read_reward(verifier_dir: Path) -> tuple[float | None, dict[str, Any]]:
reward_txt = verifier_dir / "reward.txt"
reward_json = verifier_dir / "reward.json"
if reward_txt.exists():
raw = reward_txt.read_text(encoding="utf-8").strip()
return float(raw), {"reward_source": "reward.txt"}
if reward_json.exists():
payload = json.loads(reward_json.read_text(encoding="utf-8"))
reward = payload.get("reward")
if reward is None:
numeric = [value for value in payload.values() if isinstance(value, (int, float))]
reward = float(numeric[0]) if numeric else None
return (float(reward) if reward is not None else None), {"reward_source": "reward.json", "reward_payload": payload}
return None, {}
def build_host_agent_command(
*,
task: TerminalBenchTask,
workspace_dir: Path,
repo_dir: Path,
agent_logs_dir: Path,
) -> str:
instruction_file = agent_logs_dir / "instruction.txt"
instruction_file.write_text(task.instruction, encoding="utf-8")
stdout_path = agent_logs_dir / "stdout.txt"
stderr_path = agent_logs_dir / "stderr.txt"
return (
"set -euo pipefail; "
f"cd {shlex.quote(str(repo_dir))}; "
f"instruction=$(cat {shlex.quote(str(instruction_file))}); "
f"{shlex.quote(sys.executable)} -m src.main agent \"$instruction\" "
f"--cwd {shlex.quote(str(workspace_dir))} --allow-write --allow-shell "
f"> {shlex.quote(str(stdout_path))} 2> {shlex.quote(str(stderr_path))}"
)
def build_verifier_exec_command(
*,
task: TerminalBenchTask,
image_path: Path,
workspace_dir: Path,
task_dir: Path,
verifier_logs_dir: Path,
env: dict[str, str],
fakeroot: bool = False,
) -> str:
binds = [
f"{workspace_dir}:{task.workdir}:rw",
f"{task_dir / 'tests'}:/tests:ro",
f"{verifier_logs_dir}:/logs/verifier:rw",
]
bind_flags = " ".join(f"--bind {shlex.quote(spec)}" for spec in binds)
# When using fakeroot (no-sudo HPC), inject env vars needed for apt-get
# and SSL inside the container.
if fakeroot:
fakeroot_env = {
"TMPDIR": "/tmp",
"DEBIAN_FRONTEND": "noninteractive",
"CURL_CA_BUNDLE": "/etc/ssl/certs/ca-certificates.crt",
"SSL_CERT_FILE": "/etc/ssl/certs/ca-certificates.crt",
"REQUESTS_CA_BUNDLE": "/etc/ssl/certs/ca-certificates.crt",
}
env = {**fakeroot_env, **env}
env_flags = " ".join(
f"--env {shlex.quote(key)}={shlex.quote(value)}"
for key, value in sorted(env.items())
)
inner = (
"set -euo pipefail; "
"chmod +x /tests/test.sh 2>/dev/null || true; "
f"cd {shlex.quote(task.workdir)}; "
"bash /tests/test.sh > /logs/verifier/test-stdout.txt 2> /logs/verifier/test-stderr.txt"
)
# --fakeroot: simulate root inside the container (for apt-get etc.)
# --writable-tmpfs: in-memory overlay so packages can be installed
# --contain: prevent host dirs from leaking into container
fakeroot_flags = "--fakeroot --writable-tmpfs --contain " if fakeroot else ""
return (
f"apptainer exec --cleanenv {fakeroot_flags}{bind_flags} {env_flags} "
f"--cwd {shlex.quote(task.workdir)} {shlex.quote(str(image_path))} "
f"bash -lc {shlex.quote(inner)}"
)
def run_trial(
task: TerminalBenchTask,
*,
repo_dir: Path,
jobs_dir: Path,
images_dir: Path,
force_pull: bool,
keep_images: bool,
timeout_multiplier: float,
fakeroot: bool = False,
) -> LocalTrialResult:
timestamp = time.strftime("%Y%m%d-%H%M%S")
trial_dir = jobs_dir / f"{timestamp}_{safe_name(task.short_name)}"
workspace_dir = trial_dir / "workspace"
agent_logs_dir = trial_dir / "agent"
verifier_logs_dir = trial_dir / "verifier"
for path in (trial_dir, workspace_dir, agent_logs_dir, verifier_logs_dir):
path.mkdir(parents=True, exist_ok=True)
start = time.time()
result = LocalTrialResult(
task_name=task.name,
short_name=task.short_name,
passed=False,
reward=None,
duration_sec=0.0,
status="pending",
image=task.docker_image,
workdir=task.workdir,
trial_dir=str(trial_dir),
)
if task.has_docker_compose:
result.status = "skipped"
result.error = "docker-compose task is not supported by the local Apptainer runner"
result.duration_sec = time.time() - start
return result
if not task.docker_image:
result.status = "skipped"
result.error = "task has no prebuilt docker_image in task.toml"
result.duration_sec = time.time() - start
return result
image_path = ensure_apptainer_image(task, images_dir, force_pull)
if not keep_images:
result.metadata["image_path"] = str(image_path)
seed_workspace(task, image_path, workspace_dir)
env = {
key: os.environ[key]
for key in ("OPENAI_API_KEY", "OPENAI_BASE_URL", "OPENAI_MODEL")
if os.environ.get(key)
}
env.update({str(k): str(v) for k, v in (task.raw_config.get("environment") or {}).get("env", {}).items()})
verifier_env = {str(k): str(v) for k, v in (task.raw_config.get("verifier") or {}).get("env", {}).items()}
agent_cmd = build_host_agent_command(
task=task,
workspace_dir=workspace_dir,
repo_dir=repo_dir,
agent_logs_dir=agent_logs_dir,
)
verifier_cmd = build_verifier_exec_command(
task=task,
image_path=image_path,
workspace_dir=workspace_dir,
task_dir=task.task_dir,
verifier_logs_dir=verifier_logs_dir,
env=verifier_env,
fakeroot=fakeroot,
)
agent_timeout = (task.agent_timeout_sec or 1800.0) * timeout_multiplier
verifier_timeout = task.verifier_timeout_sec * timeout_multiplier
try:
agent_proc = run_shell(agent_cmd, timeout=agent_timeout, env={**os.environ, **env})
result.agent_return_code = agent_proc.returncode
if agent_proc.returncode != 0:
result.status = "agent_failed"
stdout_path = agent_logs_dir / "stdout.txt"
stderr_path = agent_logs_dir / "stderr.txt"
stdout_text = stdout_path.read_text(encoding="utf-8", errors="ignore") if stdout_path.exists() else agent_proc.stdout
stderr_text = stderr_path.read_text(encoding="utf-8", errors="ignore") if stderr_path.exists() else agent_proc.stderr
result.error = (stdout_text + "\n" + stderr_text).strip()[:4000]
result.duration_sec = time.time() - start
return result
verifier_proc = run_shell(verifier_cmd, timeout=verifier_timeout)
result.verifier_return_code = verifier_proc.returncode
reward, reward_metadata = read_reward(verifier_logs_dir)
result.reward = reward
result.metadata.update(reward_metadata)
result.passed = verifier_proc.returncode == 0 and reward is not None and reward > 0.0
result.status = "passed" if result.passed else "failed"
if not result.passed:
stdout_text = (verifier_logs_dir / "test-stdout.txt").read_text(encoding="utf-8", errors="ignore") if (verifier_logs_dir / "test-stdout.txt").exists() else ""
stderr_text = (verifier_logs_dir / "test-stderr.txt").read_text(encoding="utf-8", errors="ignore") if (verifier_logs_dir / "test-stderr.txt").exists() else ""
result.error = (stdout_text + "\n" + stderr_text).strip()[:4000]
return result
except subprocess.TimeoutExpired as exc:
result.status = "timeout"
result.error = f"timeout after {exc.timeout}s"
return result
except Exception as exc:
result.status = "error"
result.error = str(exc)
return result
finally:
result.duration_sec = time.time() - start
def print_report(results: list[LocalTrialResult]) -> None:
total = len(results)
passed = sum(1 for item in results if item.passed)
failed = sum(1 for item in results if item.status == "failed")
skipped = sum(1 for item in results if item.status == "skipped")
print()
print("=" * 80)
print(" TERMINAL-BENCH LOCAL REPORT")
print("=" * 80)
for item in results:
icon = "PASS ✅" if item.passed else ("SKIP ⚪" if item.status == "skipped" else "FAIL ❌")
print(f" {icon:<8} {item.short_name:<32} {item.duration_sec:>7.1f}s {item.status}")
print("─" * 80)
print(f" Total: {total} Passed: {passed} Failed: {failed} Skipped: {skipped}")
score = (100.0 * passed / total) if total else 0.0
print(f" Score: {score:.1f}%")
print("─" * 80)
def save_results(path: Path, results: list[LocalTrialResult]) -> None:
payload = {
"benchmark": "terminal-bench-local-apptainer",
"timestamp": time.strftime("%Y-%m-%dT%H:%M:%S"),
"model": os.environ.get("OPENAI_MODEL", "unknown"),
"results": [asdict(item) for item in results],
}
path.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description="Run Terminal-Bench tasks locally with Apptainer and claw-code-agent.")
parser.add_argument("--tasks-dir", type=Path, default=DEFAULT_TASKS_DIR, help="Directory containing downloaded Harbor task packages.")
parser.add_argument("--include-task-name", "-i", action="append", default=[], help="Include only task names matching this glob pattern.")
parser.add_argument("--exclude-task-name", "-x", action="append", default=[], help="Exclude task names matching this glob pattern.")
parser.add_argument("--n-tasks", "-l", type=int, default=None, help="Maximum number of tasks to run after filtering.")
parser.add_argument("--jobs-dir", "-o", type=Path, default=DEFAULT_JOBS_DIR, help="Directory where trial outputs will be written.")
parser.add_argument("--images-dir", type=Path, default=DEFAULT_JOBS_DIR / "_images", help="Directory where pulled Apptainer images will be cached.")
parser.add_argument("--force-pull", action="store_true", help="Re-pull Apptainer images even if cached locally.")
parser.add_argument("--keep-images", action="store_true", help="Keep pulled SIF images in the image cache directory.")
parser.add_argument("--timeout-multiplier", type=float, default=1.0, help="Multiplier applied to agent and verifier timeouts.")
parser.add_argument("--output", type=Path, help="Optional JSON file for the run summary.")
parser.add_argument("--list", action="store_true", help="List discovered tasks and exit.")
parser.add_argument(
"--fakeroot",
action="store_true",
help="Use Apptainer --fakeroot + --writable-tmpfs for the verifier container. "
"Required on HPC systems without sudo where test.sh scripts need apt-get.",
)
return parser
def main() -> None:
parser = build_parser()
args = parser.parse_args()
if not args.tasks_dir.exists():
parser.error(f"tasks directory not found: {args.tasks_dir}")
tasks = discover_tasks(args.tasks_dir)
tasks = filter_tasks(
tasks,
include_patterns=args.include_task_name,
exclude_patterns=args.exclude_task_name,
limit=args.n_tasks,
)
if args.list:
for task in tasks:
print(task.short_name)
return
if not tasks:
parser.error("no tasks matched")
repo_dir = Path(__file__).resolve().parent.parent
results: list[LocalTrialResult] = []
print()
print("=" * 80)
print(" TERMINAL-BENCH LOCAL")
print("=" * 80)
print(f" Tasks: {len(tasks)}")
print(f" Jobs dir: {args.jobs_dir}")
if args.fakeroot:
print(" Fakeroot: enabled (no-sudo HPC mode)")
print("=" * 80)
print()
for index, task in enumerate(tasks, 1):
print(f"[{index}/{len(tasks)}] {task.short_name}")
result = run_trial(
task,
repo_dir=repo_dir,
jobs_dir=args.jobs_dir,
images_dir=args.images_dir,
force_pull=args.force_pull,
keep_images=args.keep_images,
timeout_multiplier=args.timeout_multiplier,
fakeroot=args.fakeroot,
)
results.append(result)
icon = "PASS ✅" if result.passed else ("SKIP ⚪" if result.status == "skipped" else "FAIL ❌")
print(f" -> {icon} ({result.duration_sec:.1f}s)")
print(f" trial_dir: {result.trial_dir}")
if result.error:
print(f" error: {result.error[:200]}")
print()
print_report(results)
if args.output:
save_results(args.output, results)
if __name__ == "__main__":
main()
benchmarks/suites/__init__.py
-1
View File
@@ -1 +0,0 @@
"""Standard evaluation benchmark suites for claw-code-agent."""
benchmarks/suites/aider.py
-300
View File
@@ -1,300 +0,0 @@
"""
Aider benchmark suite.
Aider's code editing benchmark measures how well an agent can apply
specified edits to existing codebases — refactoring, adding features,
and fixing bugs based on natural language instructions.
Reference: https://aider.chat/docs/leaderboards/
"""
from __future__ import annotations
import json
import os
import sys
import textwrap
from pathlib import Path
from typing import Any
from .base import BenchmarkResult, BenchmarkSuite
# ---------------------------------------------------------------------------
# Built-in mini dataset — Aider-style edit tasks
# ---------------------------------------------------------------------------
_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
{
"id": "aider-001",
"instruction": "Add a `__len__` method to the `TaskList` class that returns the number of tasks.",
"setup_code": textwrap.dedent("""\
cat > tasks.py << 'PYEOF'
class TaskList:
def __init__(self):
self._tasks = []
def add(self, task):
self._tasks.append(task)
def get_all(self):
return list(self._tasks)
PYEOF
"""),
"test_code": textwrap.dedent("""\
from tasks import TaskList
t = TaskList()
assert len(t) == 0
t.add("task1")
t.add("task2")
assert len(t) == 2
# Ensure original methods still work
assert t.get_all() == ["task1", "task2"]
print("ALL_TESTS_PASSED")
"""),
},
{
"id": "aider-002",
"instruction": (
"Refactor the `process_data` function to use a list comprehension "
"instead of the for loop. The behavior must remain identical."
),
"setup_code": textwrap.dedent("""\
cat > processor.py << 'PYEOF'
def process_data(items):
result = []
for item in items:
if item > 0:
result.append(item * 2)
return result
PYEOF
"""),
"test_code": textwrap.dedent("""\
import inspect
from processor import process_data
assert process_data([1, -2, 3, -4, 5]) == [2, 6, 10]
assert process_data([]) == []
assert process_data([-1, -2]) == []
# Check that a list comprehension is used
src = inspect.getsource(process_data)
assert 'for' in src and '[' in src, "Should use list comprehension"
assert src.count('for') <= 2, "Should not have a separate for loop"
print("ALL_TESTS_PASSED")
"""),
},
{
"id": "aider-003",
"instruction": (
"Add error handling to the `divide` function so it raises a "
"`ValueError` with message 'Division by zero' when b is 0, "
"instead of letting ZeroDivisionError propagate."
),
"setup_code": textwrap.dedent("""\
cat > mathops.py << 'PYEOF'
def divide(a, b):
return a / b
PYEOF
"""),
"test_code": textwrap.dedent("""\
from mathops import divide
assert divide(10, 2) == 5.0
assert divide(7, 2) == 3.5
try:
divide(1, 0)
assert False, "Should have raised ValueError"
except ValueError as e:
assert str(e) == 'Division by zero'
except ZeroDivisionError:
assert False, "Should raise ValueError, not ZeroDivisionError"
print("ALL_TESTS_PASSED")
"""),
},
{
"id": "aider-004",
"instruction": (
"Add a `to_dict` method to the `Config` class that returns a dictionary "
"of all configuration key-value pairs."
),
"setup_code": textwrap.dedent("""\
cat > config.py << 'PYEOF'
class Config:
def __init__(self):
self._data = {}
def set(self, key, value):
self._data[key] = value
def get(self, key, default=None):
return self._data.get(key, default)
PYEOF
"""),
"test_code": textwrap.dedent("""\
from config import Config
c = Config()
assert c.to_dict() == {}
c.set('host', 'localhost')
c.set('port', 8080)
d = c.to_dict()
assert d == {'host': 'localhost', 'port': 8080}
# Ensure it's a copy, not a reference
d['new'] = 'value'
assert 'new' not in c.to_dict()
print("ALL_TESTS_PASSED")
"""),
},
{
"id": "aider-005",
"instruction": (
"Add a `reverse` method to the `LinkedList` class that reverses the "
"list in-place."
),
"setup_code": textwrap.dedent("""\
cat > linkedlist.py << 'PYEOF'
class Node:
def __init__(self, val, next=None):
self.val = val
self.next = next
class LinkedList:
def __init__(self):
self.head = None
def append(self, val):
if not self.head:
self.head = Node(val)
return
curr = self.head
while curr.next:
curr = curr.next
curr.next = Node(val)
def to_list(self):
result = []
curr = self.head
while curr:
result.append(curr.val)
curr = curr.next
return result
PYEOF
"""),
"test_code": textwrap.dedent("""\
from linkedlist import LinkedList
ll = LinkedList()
ll.append(1)
ll.append(2)
ll.append(3)
ll.reverse()
assert ll.to_list() == [3, 2, 1]
# Test single element
ll2 = LinkedList()
ll2.append(42)
ll2.reverse()
assert ll2.to_list() == [42]
# Test empty
ll3 = LinkedList()
ll3.reverse()
assert ll3.to_list() == []
print("ALL_TESTS_PASSED")
"""),
},
{
"id": "aider-006",
"instruction": (
"Convert the `UserStore` class to use a context manager pattern. "
"Add `__enter__` and `__exit__` methods. `__enter__` should return self, "
"and `__exit__` should call the existing `close` method."
),
"setup_code": textwrap.dedent("""\
cat > userstore.py << 'PYEOF'
class UserStore:
def __init__(self):
self._users = {}
self._closed = False
def add(self, name, email):
self._users[name] = email
def get(self, name):
return self._users.get(name)
def close(self):
self._closed = True
@property
def is_closed(self):
return self._closed
PYEOF
"""),
"test_code": textwrap.dedent("""\
from userstore import UserStore
with UserStore() as store:
store.add("alice", "alice@example.com")
assert store.get("alice") == "alice@example.com"
assert not store.is_closed
assert store.is_closed
# Test exception in with block still calls close
try:
with UserStore() as store2:
store2.add("bob", "bob@example.com")
raise ValueError("test error")
except ValueError:
pass
assert store2.is_closed
print("ALL_TESTS_PASSED")
"""),
},
]
class AiderBenchmark(BenchmarkSuite):
"""Aider: code editing benchmark."""
name = "Aider"
description = "Code editing and refactoring tasks"
category = "coding"
def load_dataset(self) -> list[dict[str, Any]]:
jsonl_path = Path(self.data_dir) / "aider.jsonl"
if jsonl_path.exists():
problems: list[dict[str, Any]] = []
with open(jsonl_path) as fh:
for line in fh:
line = line.strip()
if line:
problems.append(json.loads(line))
if self.verbose:
print(f" Loaded {len(problems)} problems from {jsonl_path}")
return problems
if self.verbose:
print(f" {jsonl_path} not found — using built-in 6-problem subset")
return list(_BUILTIN_PROBLEMS)
def build_prompt(self, problem: dict[str, Any]) -> str:
return (
f"Edit the code in the workspace to satisfy this requirement:\n\n"
f"{problem['instruction']}\n\n"
f"Read the existing file(s), make the minimal changes needed, "
f"and save the updated file(s)."
)
def setup_workspace(self, problem: dict[str, Any], workspace: str) -> None:
setup = problem.get("setup_code", "")
if setup:
self._run_shell(setup, cwd=workspace, timeout=30.0)
# Write test file
test_code = problem.get("test_code", "")
if test_code:
with open(os.path.join(workspace, "test_harness.py"), "w") as fh:
fh.write(test_code)
def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
pid = problem.get("id", "unknown")
code, output = self._run_shell(
f"{sys.executable} test_harness.py", cwd=workspace, timeout=30.0
)
passed = code == 0 and "ALL_TESTS_PASSED" in output
return BenchmarkResult(
problem_id=pid, passed=passed, actual=output[:500],
error="" if passed else output[:500],
)
benchmarks/suites/aime.py
-158
View File
@@ -1,158 +0,0 @@
"""
AIME benchmark suite.
AIME (American Invitational Mathematics Examination) problems are
challenging high-school competition problems. All answers are integers
from 000 to 999.
Reference: https://artofproblemsolving.com/wiki/index.php/AIME
"""
from __future__ import annotations
import json
import os
import re
from pathlib import Path
from typing import Any
from .base import BenchmarkResult, BenchmarkSuite
# ---------------------------------------------------------------------------
# Built-in mini dataset (10 representative AIME-style problems)
# ---------------------------------------------------------------------------
_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
{
"id": "aime-001",
"problem": "Find the sum of all positive integers $n$ such that $n^2 - 19n + 99$ is a perfect square.",
"answer": "38",
},
{
"id": "aime-002",
"problem": "The number $2^{1993} + 3^{1993}$ is divisible by 5. What is the units digit of the quotient $\\frac{2^{1993}+3^{1993}}{5}$?",
"answer": "3",
},
{
"id": "aime-003",
"problem": "Let $S$ be the set of integers between 1 and $2^{40}$ whose binary expansions have exactly two 1's. If a number is chosen at random from $S$, what is the probability that it is divisible by 9? Express as $p/q$ where $p$ and $q$ are coprime, and find $p+q$.",
"answer": "913",
},
{
"id": "aime-004",
"problem": "What is the largest prime factor of $1{,}000{,}027$?",
"answer": "103",
},
{
"id": "aime-005",
"problem": "Compute the remainder when $3^{1000}$ is divided by $1000$.",
"answer": "1",
},
{
"id": "aime-006",
"problem": "How many four-digit numbers have the property that the digits sum to 10?",
"answer": "219",
},
{
"id": "aime-007",
"problem": "Find the number of ordered triples $(a, b, c)$ of positive integers satisfying $a + b + c = 20$ and $a \\leq b \\leq c$.",
"answer": "33",
},
{
"id": "aime-008",
"problem": "A fair coin is tossed 10 times. What is the probability that no two consecutive tosses are both heads? If the probability is $\\frac{m}{n}$ in lowest terms, find $m + n$.",
"answer": "73",
},
{
"id": "aime-009",
"problem": "Let $f(n) = n^2 + n + 1$. Find the remainder when $f(1) \\cdot f(2) \\cdot f(3) \\cdots f(10)$ is divided by 100.",
"answer": "10",
},
{
"id": "aime-010",
"problem": "What is the last three digits of $7^{999}$?",
"answer": "343",
},
]
class AIMEBenchmark(BenchmarkSuite):
"""AIME: American Invitational Mathematics Examination problems."""
name = "AIME"
description = "Challenging competition math problems (answers are integers 000–999)"
category = "math"
def load_dataset(self) -> list[dict[str, Any]]:
# Prefer aime_2026.jsonl for AIME 2026 specific problems
for fname in ("aime_2026.jsonl", "aime.jsonl"):
jsonl_path = Path(self.data_dir) / fname
if jsonl_path.exists():
problems: list[dict[str, Any]] = []
with open(jsonl_path) as fh:
for line in fh:
line = line.strip()
if line:
problems.append(json.loads(line))
if self.verbose:
print(f" Loaded {len(problems)} problems from {jsonl_path}")
return problems
if self.verbose:
print(" No AIME data file found — using built-in 10-problem subset")
return list(_BUILTIN_PROBLEMS)
def build_prompt(self, problem: dict[str, Any]) -> str:
prob = problem["problem"]
return (
f"Solve the following AIME-style competition math problem.\n\n"
f"Problem: {prob}\n\n"
f"The answer is an integer between 0 and 999. "
f"Write ONLY the integer answer to a file called answer.txt — "
f"no explanation, no work, just the number."
)
def recover_output_files(
self,
problem: dict[str, Any],
workspace: str,
agent_output: str,
metadata: dict[str, Any],
) -> None:
del problem
answer_path = Path(workspace) / "answer.txt"
if answer_path.exists():
return
numbers = re.findall(r"-?\d+", agent_output.replace(",", ""))
if numbers:
answer_path.write_text(numbers[-1] + "\n", encoding="utf-8")
metadata["recovered_answer_from_output"] = True
def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
pid = problem.get("id", "unknown")
expected = str(problem["answer"]).strip()
answer_file = os.path.join(workspace, "answer.txt")
if not os.path.exists(answer_file):
return BenchmarkResult(
problem_id=pid, passed=False, expected=expected,
error="answer.txt not found",
)
with open(answer_file) as fh:
actual_raw = fh.read().strip()
# Extract last integer from the answer
numbers = re.findall(r"-?\d+", actual_raw.replace(",", ""))
actual = numbers[-1] if numbers else actual_raw
try:
passed = int(actual) == int(expected)
except (ValueError, TypeError):
passed = actual == expected
return BenchmarkResult(
problem_id=pid, passed=passed,
expected=expected, actual=actual_raw,
error="" if passed else f"expected={expected}, got={actual_raw}",
)
benchmarks/suites/base.py
-340
View File
@@ -1,340 +0,0 @@
"""
Base class for benchmark suites and shared benchmark helpers.
"""
from __future__ import annotations
import json
import os
import re
import shutil
import subprocess
import sys
import tempfile
import time
from abc import ABC, abstractmethod
from dataclasses import asdict, dataclass, field
from pathlib import Path
from typing import Any
_SAFE_COMPONENT_RE = re.compile(r"[^A-Za-z0-9_.-]+")
def _safe_component(value: str) -> str:
cleaned = _SAFE_COMPONENT_RE.sub("_", value).strip("._")
return cleaned or "item"
def resolve_temp_root() -> Path:
root = Path(tempfile.gettempdir()).expanduser().resolve()
root.mkdir(parents=True, exist_ok=True)
return root
def make_temp_workspace(prefix: str, suite_name: str, problem_id: str) -> str:
temp_root = resolve_temp_root()
safe_prefix = _safe_component(prefix)
safe_suite = _safe_component(suite_name)
safe_problem = _safe_component(problem_id)
return tempfile.mkdtemp(
prefix=f"{safe_prefix}_{safe_suite}_{safe_problem}_",
dir=str(temp_root),
)
@dataclass
class BenchmarkResult:
problem_id: str
passed: bool
expected: str = ""
actual: str = ""
duration_sec: float = 0.0
error: str = ""
metadata: dict[str, Any] = field(default_factory=dict)
@dataclass
class SuiteReport:
suite_name: str
total: int
passed: int
failed: int
score_pct: float
duration_sec: float
model: str
results: list[BenchmarkResult]
timestamp: str = ""
def to_dict(self) -> dict[str, Any]:
return {
"suite_name": self.suite_name,
"total": self.total,
"passed": self.passed,
"failed": self.failed,
"score_pct": self.score_pct,
"duration_sec": round(self.duration_sec, 2),
"model": self.model,
"timestamp": self.timestamp,
"results": [asdict(r) for r in self.results],
}
class BenchmarkSuite(ABC):
name: str = "base"
description: str = ""
category: str = "general"
def __init__(
self,
*,
data_dir: str | None = None,
limit: int | None = None,
agent_timeout: float = 300.0,
verbose: bool = False,
artifacts_dir: str | None = None,
save_passing_artifacts: bool = False,
) -> None:
self.data_dir = data_dir or str(
Path(__file__).resolve().parent.parent / "data"
)
self.limit = limit
self.agent_timeout = agent_timeout
self.verbose = verbose
self.artifacts_dir = artifacts_dir
self.save_passing_artifacts = save_passing_artifacts
self.project_root = str(Path(__file__).resolve().parent.parent.parent)
@abstractmethod
def load_dataset(self) -> list[dict[str, Any]]:
...
@abstractmethod
def build_prompt(self, problem: dict[str, Any]) -> str:
...
@abstractmethod
def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
...
def _run_shell(
self,
cmd: str,
cwd: str,
timeout: float = 30.0,
) -> tuple[int, str]:
try:
proc = subprocess.run(
cmd,
shell=True,
cwd=cwd,
capture_output=True,
text=True,
timeout=timeout,
)
return proc.returncode, (proc.stdout + proc.stderr).strip()
except subprocess.TimeoutExpired:
return 1, f"[TIMEOUT after {timeout}s]"
except Exception as exc:
return 1, str(exc)
def run_agent(self, instruction: str, workspace: str) -> tuple[int, str, float]:
import shlex
agent_cmd = (
f"{sys.executable} -m src.main agent "
f"{shlex.quote(instruction)} "
f"--cwd {shlex.quote(workspace)} "
f"--allow-write "
f"--allow-shell"
)
if self.verbose:
print(f" agent cmd: {agent_cmd[:160]}...")
start = time.time()
code, output = self._run_shell(
agent_cmd,
cwd=self.project_root,
timeout=self.agent_timeout,
)
duration = time.time() - start
if self.verbose:
print(f" agent exit={code} duration={duration:.1f}s")
return code, output, duration
def setup_workspace(self, problem: dict[str, Any], workspace: str) -> None:
del problem, workspace
def recover_output_files(
self,
problem: dict[str, Any],
workspace: str,
agent_output: str,
metadata: dict[str, Any],
) -> None:
del problem, workspace, agent_output, metadata
def _artifact_root(self, index: int, problem_id: str) -> Path | None:
if not self.artifacts_dir:
return None
root = Path(self.artifacts_dir)
root.mkdir(parents=True, exist_ok=True)
return root / f"{index:03d}_{_safe_component(problem_id)}"
def _save_artifacts(
self,
*,
index: int,
problem: dict[str, Any],
prompt: str,
agent_output: str,
workspace: str,
result: BenchmarkResult,
agent_exit_code: int,
) -> None:
artifact_root = self._artifact_root(index, result.problem_id)
if artifact_root is None:
return
if result.passed and not self.save_passing_artifacts:
return
artifact_root.mkdir(parents=True, exist_ok=True)
(artifact_root / "problem.json").write_text(
json.dumps(problem, indent=2) + "\n",
encoding="utf-8",
)
(artifact_root / "prompt.txt").write_text(prompt, encoding="utf-8")
(artifact_root / "agent_output.txt").write_text(agent_output, encoding="utf-8")
workspace_dst = artifact_root / "workspace"
if workspace_dst.exists():
shutil.rmtree(workspace_dst, ignore_errors=True)
shutil.copytree(workspace, workspace_dst)
result_payload = asdict(result)
result_payload["agent_exit_code"] = agent_exit_code
result_payload["workspace"] = workspace
(artifact_root / "result.json").write_text(
json.dumps(result_payload, indent=2) + "\n",
encoding="utf-8",
)
result.metadata["artifact_path"] = str(artifact_root)
def run_all(self) -> SuiteReport:
problems = self.load_dataset()
if self.limit is not None:
problems = problems[: self.limit]
print()
print("=" * 72)
print(f" {self.name} BENCHMARK")
print(f" {self.description}")
print("=" * 72)
model = os.environ.get("OPENAI_MODEL", "unknown")
print(f" Model: {model}")
print(f" Problems: {len(problems)}")
print(f" Timeout: {self.agent_timeout}s per problem")
print("=" * 72)
print()
suite_start = time.time()
all_results: list[BenchmarkResult] = []
for index, problem in enumerate(problems, 1):
pid = str(problem.get("id", problem.get("task_id", f"problem-{index}")))
print(f"[{index}/{len(problems)}] {pid}")
workspace = make_temp_workspace("claw", self.name, pid)
prompt = ""
agent_output = ""
agent_exit_code = -1
try:
self.setup_workspace(problem, workspace)
prompt = self.build_prompt(problem)
agent_exit_code, agent_output, duration = self.run_agent(prompt, workspace)
result_metadata: dict[str, Any] = {"agent_exit_code": agent_exit_code}
self.recover_output_files(problem, workspace, agent_output, result_metadata)
result = self.evaluate(problem, workspace)
result.duration_sec = duration
result.metadata.update(result_metadata)
self._save_artifacts(
index=index,
problem=problem,
prompt=prompt,
agent_output=agent_output,
workspace=workspace,
result=result,
agent_exit_code=agent_exit_code,
)
status = "PASS ✅" if result.passed else "FAIL ❌"
print(f" -> {status} ({duration:.1f}s)")
except Exception as exc:
result = BenchmarkResult(
problem_id=pid,
passed=False,
error=str(exc),
metadata={"agent_exit_code": agent_exit_code},
)
self._save_artifacts(
index=index,
problem=problem,
prompt=prompt,
agent_output=agent_output,
workspace=workspace,
result=result,
agent_exit_code=agent_exit_code,
)
print(f" -> ERROR ❌ {exc}")
finally:
shutil.rmtree(workspace, ignore_errors=True)
all_results.append(result)
print()
suite_duration = time.time() - suite_start
passed = sum(1 for item in all_results if item.passed)
total = len(all_results)
report = SuiteReport(
suite_name=self.name,
total=total,
passed=passed,
failed=total - passed,
score_pct=round(100.0 * passed / total, 1) if total else 0.0,
duration_sec=suite_duration,
model=model,
results=all_results,
timestamp=time.strftime("%Y-%m-%dT%H:%M:%S"),
)
self._print_report(report)
return report
@staticmethod
def _print_report(report: SuiteReport) -> None:
print()
print("=" * 72)
print(f" {report.suite_name} — RESULTS")
print("=" * 72)
print()
for result in report.results:
icon = "✅" if result.passed else "❌"
print(f" {icon} {result.problem_id:<40} {result.duration_sec:.1f}s")
print()
print("─" * 72)
print(
f" Total: {report.total} | Passed: {report.passed} "
f"| Failed: {report.failed} | Score: {report.score_pct:.1f}%"
)
print(f" Total time: {report.duration_sec:.1f}s")
print("─" * 72)
print()
@staticmethod
def save_report(report: SuiteReport, path: str) -> None:
with open(path, "w", encoding="utf-8") as fh:
json.dump(report.to_dict(), fh, indent=2)
print(f" Report saved to {path}")
benchmarks/suites/bfcl.py
-341
View File
@@ -1,341 +0,0 @@
"""
BFCL (Berkeley Function Calling Leaderboard) benchmark suite.
BFCL tests whether an agent can correctly identify and call functions/tools
based on natural language instructions. This is directly relevant for
coding agents that use tool calling.
Reference: https://gorilla.cs.berkeley.edu/leaderboard.html
"""
from __future__ import annotations
import json
import os
import re
import sys
from pathlib import Path
from typing import Any
from .base import BenchmarkResult, BenchmarkSuite
# ---------------------------------------------------------------------------
# Built-in mini dataset — function-calling tasks
# ---------------------------------------------------------------------------
_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
{
"id": "bfcl-001",
"instruction": (
"You have access to a function `get_weather(city: str, unit: str = 'celsius') -> dict` "
"that returns weather info. Write a Python script called solution.py that calls "
"get_weather for 'San Francisco' with unit 'fahrenheit' and prints the result."
),
"expected_function": "get_weather",
"expected_args": {"city": "San Francisco", "unit": "fahrenheit"},
"setup_code": (
"cat > weather_api.py << 'PYEOF'\n"
"def get_weather(city: str, unit: str = 'celsius') -> dict:\n"
" return {'city': city, 'unit': unit, 'temp': 72 if unit == 'fahrenheit' else 22}\n"
"PYEOF"
),
"test_code": (
"import sys\n"
"sys.path.insert(0, '.')\n"
"# Check that solution.py exists and calls get_weather correctly\n"
"with open('solution.py') as f:\n"
" code = f.read()\n"
"assert 'get_weather' in code, 'Must call get_weather'\n"
"assert 'San Francisco' in code, 'Must use San Francisco'\n"
"assert 'fahrenheit' in code, 'Must use fahrenheit'\n"
"exec(open('solution.py').read())\n"
"print('ALL_TESTS_PASSED')\n"
),
},
{
"id": "bfcl-002",
"instruction": (
"You have access to these functions:\n"
"- `search_files(directory: str, pattern: str) -> list[str]`\n"
"- `read_file(path: str) -> str`\n"
"- `write_file(path: str, content: str) -> None`\n\n"
"Write a Python script called solution.py that:\n"
"1. Calls search_files('.', '*.txt') to find all text files\n"
"2. Reads each file using read_file\n"
"3. Writes a combined output using write_file to 'combined.txt'"
),
"expected_function": "search_files",
"expected_args": {"directory": ".", "pattern": "*.txt"},
"setup_code": (
"cat > file_api.py << 'PYEOF'\n"
"import glob\n"
"def search_files(directory: str, pattern: str) -> list:\n"
" import os\n"
" return [os.path.join(directory, f) for f in os.listdir(directory) if f.endswith('.txt') and f != 'combined.txt']\n"
"def read_file(path: str) -> str:\n"
" with open(path) as f:\n"
" return f.read()\n"
"def write_file(path: str, content: str) -> None:\n"
" with open(path, 'w') as f:\n"
" f.write(content)\n"
"PYEOF\n"
"echo 'hello' > a.txt\n"
"echo 'world' > b.txt\n"
),
"test_code": (
"import sys\n"
"sys.path.insert(0, '.')\n"
"with open('solution.py') as f:\n"
" code = f.read()\n"
"assert 'search_files' in code, 'Must call search_files'\n"
"assert 'read_file' in code, 'Must call read_file'\n"
"assert 'write_file' in code, 'Must call write_file'\n"
"exec(open('solution.py').read())\n"
"import os\n"
"assert os.path.exists('combined.txt'), 'combined.txt must exist'\n"
"content = open('combined.txt').read()\n"
"assert 'hello' in content and 'world' in content\n"
"print('ALL_TESTS_PASSED')\n"
),
},
{
"id": "bfcl-003",
"instruction": (
"You have a function `calculate(expression: str) -> float` that evaluates "
"a mathematical expression string. Write solution.py that:\n"
"1. Calculates '(15 + 25) * 3'\n"
"2. Calculates '100 / 4 - 5'\n"
"3. Prints both results"
),
"expected_function": "calculate",
"expected_args": {"expression": "(15 + 25) * 3"},
"setup_code": (
"cat > calc_api.py << 'PYEOF'\n"
"def calculate(expression: str) -> float:\n"
" return float(eval(expression))\n"
"PYEOF"
),
"test_code": (
"import sys\n"
"sys.path.insert(0, '.')\n"
"with open('solution.py') as f:\n"
" code = f.read()\n"
"assert 'calculate' in code, 'Must call calculate'\n"
"exec(open('solution.py').read())\n"
"print('ALL_TESTS_PASSED')\n"
),
},
{
"id": "bfcl-004",
"instruction": (
"You have these functions:\n"
"- `create_user(name: str, email: str, role: str = 'user') -> dict`\n"
"- `delete_user(user_id: int) -> bool`\n"
"- `list_users() -> list[dict]`\n\n"
"Write solution.py that creates a user named 'Alice' with email "
"'alice@example.com' and role 'admin', then lists all users."
),
"expected_function": "create_user",
"expected_args": {"name": "Alice", "email": "alice@example.com", "role": "admin"},
"setup_code": (
"cat > user_api.py << 'PYEOF'\n"
"_users = []\n"
"_next_id = 1\n"
"def create_user(name: str, email: str, role: str = 'user') -> dict:\n"
" global _next_id\n"
" user = {'id': _next_id, 'name': name, 'email': email, 'role': role}\n"
" _users.append(user)\n"
" _next_id += 1\n"
" return user\n"
"def delete_user(user_id: int) -> bool:\n"
" global _users\n"
" _users = [u for u in _users if u['id'] != user_id]\n"
" return True\n"
"def list_users() -> list:\n"
" return list(_users)\n"
"PYEOF"
),
"test_code": (
"import sys\n"
"sys.path.insert(0, '.')\n"
"with open('solution.py') as f:\n"
" code = f.read()\n"
"assert 'create_user' in code, 'Must call create_user'\n"
"assert 'Alice' in code, 'Must use Alice'\n"
"assert 'alice@example.com' in code, 'Must use correct email'\n"
"assert 'admin' in code, 'Must use admin role'\n"
"assert 'list_users' in code, 'Must call list_users'\n"
"print('ALL_TESTS_PASSED')\n"
),
},
{
"id": "bfcl-005",
"instruction": (
"You have a function `send_email(to: str, subject: str, body: str, "
"cc: list[str] = None) -> bool`. Write solution.py that sends an email "
"to 'boss@company.com' with subject 'Weekly Report', body 'Please see attached report.', "
"and cc=['team@company.com']."
),
"expected_function": "send_email",
"expected_args": {
"to": "boss@company.com",
"subject": "Weekly Report",
"body": "Please see attached report.",
"cc": ["team@company.com"],
},
"setup_code": (
"cat > email_api.py << 'PYEOF'\n"
"def send_email(to: str, subject: str, body: str, cc: list = None) -> bool:\n"
" print(f'Email sent to {to}, subject: {subject}')\n"
" return True\n"
"PYEOF"
),
"test_code": (
"import sys\n"
"sys.path.insert(0, '.')\n"
"with open('solution.py') as f:\n"
" code = f.read()\n"
"assert 'send_email' in code, 'Must call send_email'\n"
"assert 'boss@company.com' in code, 'Must use correct recipient'\n"
"assert 'Weekly Report' in code, 'Must use correct subject'\n"
"assert 'team@company.com' in code, 'Must include cc'\n"
"print('ALL_TESTS_PASSED')\n"
),
},
{
"id": "bfcl-006",
"instruction": (
"You have these functions:\n"
"- `connect_db(host: str, port: int, database: str) -> object`\n"
"- `execute_query(connection: object, query: str) -> list`\n"
"- `close_db(connection: object) -> None`\n\n"
"Write solution.py that connects to host='localhost', port=5432, database='mydb', "
"executes 'SELECT * FROM users', and closes the connection."
),
"expected_function": "connect_db",
"expected_args": {"host": "localhost", "port": 5432, "database": "mydb"},
"setup_code": (
"cat > db_api.py << 'PYEOF'\n"
"class FakeConnection:\n"
" def __init__(self, host, port, db):\n"
" self.host = host\n"
" self.port = port\n"
" self.db = db\n"
" self.closed = False\n"
"def connect_db(host: str, port: int, database: str) -> object:\n"
" return FakeConnection(host, port, database)\n"
"def execute_query(connection: object, query: str) -> list:\n"
" return [{'id': 1, 'name': 'test'}]\n"
"def close_db(connection: object) -> None:\n"
" connection.closed = True\n"
"PYEOF"
),
"test_code": (
"import sys\n"
"sys.path.insert(0, '.')\n"
"with open('solution.py') as f:\n"
" code = f.read()\n"
"assert 'connect_db' in code, 'Must call connect_db'\n"
"assert 'execute_query' in code, 'Must call execute_query'\n"
"assert 'close_db' in code, 'Must call close_db'\n"
"assert 'localhost' in code, 'Must use localhost'\n"
"assert '5432' in code, 'Must use port 5432'\n"
"assert 'mydb' in code, 'Must use mydb database'\n"
"print('ALL_TESTS_PASSED')\n"
),
},
{
"id": "bfcl-007",
"instruction": (
"You have a function `sort_data(data: list, key: str, reverse: bool = False) -> list`. "
"Write solution.py that sorts the list [{'name': 'Charlie', 'age': 30}, "
"{'name': 'Alice', 'age': 25}, {'name': 'Bob', 'age': 35}] by 'age' in "
"descending order (reverse=True) and prints the result."
),
"expected_function": "sort_data",
"expected_args": {"key": "age", "reverse": True},
"setup_code": (
"cat > sort_api.py << 'PYEOF'\n"
"def sort_data(data: list, key: str, reverse: bool = False) -> list:\n"
" return sorted(data, key=lambda x: x[key], reverse=reverse)\n"
"PYEOF"
),
"test_code": (
"import sys\n"
"sys.path.insert(0, '.')\n"
"with open('solution.py') as f:\n"
" code = f.read()\n"
"assert 'sort_data' in code, 'Must call sort_data'\n"
"assert 'age' in code, 'Must sort by age'\n"
"assert 'reverse' in code or 'True' in code, 'Must use reverse order'\n"
"print('ALL_TESTS_PASSED')\n"
),
},
]
class BFCLBenchmark(BenchmarkSuite):
"""BFCL: Berkeley Function Calling Leaderboard."""
name = "BFCL"
description = "Function/tool calling evaluation"
category = "instruction-following"
def load_dataset(self) -> list[dict[str, Any]]:
jsonl_path = Path(self.data_dir) / "bfcl.jsonl"
if jsonl_path.exists():
problems: list[dict[str, Any]] = []
with open(jsonl_path) as fh:
for line in fh:
line = line.strip()
if line:
problems.append(json.loads(line))
if self.verbose:
print(f" Loaded {len(problems)} problems from {jsonl_path}")
return problems
if self.verbose:
print(f" {jsonl_path} not found — using built-in 7-problem subset")
return list(_BUILTIN_PROBLEMS)
def build_prompt(self, problem: dict[str, Any]) -> str:
return problem["instruction"]
def setup_workspace(self, problem: dict[str, Any], workspace: str) -> None:
setup = problem.get("setup_code", "")
if setup:
self._run_shell(setup, cwd=workspace, timeout=30.0)
test_code = problem.get("test_code", "")
if test_code:
with open(os.path.join(workspace, "test_harness.py"), "w") as fh:
fh.write(test_code)
def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
pid = problem.get("id", "unknown")
sol = os.path.join(workspace, "solution.py")
if not os.path.exists(sol):
return BenchmarkResult(
problem_id=pid, passed=False, error="solution.py not found"
)
output = ""
test_harness = os.path.join(workspace, "test_harness.py")
if os.path.exists(test_harness):
code, output = self._run_shell(
f"{sys.executable} test_harness.py", cwd=workspace, timeout=30.0
)
passed = code == 0 and "ALL_TESTS_PASSED" in output
else:
# Basic check: verify the expected function is called
with open(sol) as fh:
code_content = fh.read()
expected_fn = problem.get("expected_function", "")
passed = expected_fn in code_content if expected_fn else True
return BenchmarkResult(
problem_id=pid, passed=passed,
actual=output[:500],
error="" if passed else (output[:500] if output else "function call not found"),
)
benchmarks/suites/bigbench.py
-176
View File
@@ -1,176 +0,0 @@
"""
BIG-Bench Extra Hard benchmark suite.
BIG-Bench Extra Hard (BBH) is a subset of the most challenging tasks from
the BIG-Bench benchmark that language models have historically struggled with.
Tasks include logical reasoning, causal judgment, date understanding, etc.
Reference: https://huggingface.co/datasets/maveriq/bigbenchhard
"""
from __future__ import annotations
import json
import os
import re
from pathlib import Path
from typing import Any
from .base import BenchmarkResult, BenchmarkSuite
# ---------------------------------------------------------------------------
# Built-in mini dataset (10 representative BIG-Bench Hard problems)
# ---------------------------------------------------------------------------
_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
{
"id": "bbh-001",
"task": "logical_deduction_three_objects",
"question": "The following paragraphs each describe a set of three objects arranged in a fixed order. The statements are logically consistent within each paragraph.\n\nOn a shelf, there are three books: a red book, a green book, and a blue book. The blue book is to the left of the green book. The red book is the second from the left.\n\nWhich book is the leftmost?",
"choices": ["The red book", "The green book", "The blue book"],
"answer": "C",
},
{
"id": "bbh-002",
"task": "causal_judgement",
"question": "How would a typical person answer each of the following questions about causation?\n\nBilly and Suzy each throw a rock at a bottle. Suzy's rock arrives first and shatters the bottle. Billy's rock arrives at where the bottle was a moment later.\n\nDid Billy cause the bottle to shatter?",
"choices": ["Yes", "No"],
"answer": "B",
},
{
"id": "bbh-003",
"task": "date_understanding",
"question": "Today is Christmas Eve of 1937. What is the date tomorrow in MM/DD/YYYY?",
"choices": ["12/11/1937", "12/25/1937", "01/04/1938", "12/04/1937", "12/25/2006", "09/25/1937"],
"answer": "B",
},
{
"id": "bbh-004",
"task": "disambiguation_qa",
"question": "In the following sentences, explain the antecedent of the pronoun (which thing the pronoun refers to), or state that it is ambiguous.\n\nSentence: The scientist thanked the librarian because she was helpful.\n\nWho was helpful?",
"choices": ["The scientist", "The librarian", "Ambiguous"],
"answer": "B",
},
{
"id": "bbh-005",
"task": "navigate",
"question": "If you follow these instructions, do you return to the starting point?\n\nTake 1 step east. Take 2 steps north. Take 1 step west. Take 2 steps south.",
"choices": ["Yes", "No"],
"answer": "A",
},
{
"id": "bbh-006",
"task": "penguins_in_a_table",
"question": "Here is a table where the first line is a header and each subsequent line is a penguin:\n\nname, age, height (cm), weight (kg)\nLouis, 7, 50, 11\nBernard, 5, 80, 13\nVincent, 9, 60, 11\nGwen, 8, 70, 15\n\nWhat is the name of the tallest penguin?",
"choices": ["Louis", "Bernard", "Vincent", "Gwen"],
"answer": "B",
},
{
"id": "bbh-007",
"task": "sports_understanding",
"question": "Is the following sentence plausible?\n\n\"Lebron James scored a touchdown.\"",
"choices": ["plausible", "not plausible"],
"answer": "B",
},
{
"id": "bbh-008",
"task": "boolean_expressions",
"question": "Evaluate the result of a random Boolean expression.\n\nnot ( ( not not True ) ) is",
"choices": ["True", "False"],
"answer": "B",
},
{
"id": "bbh-009",
"task": "web_of_lies",
"question": "Evaluate a random Boolean function expressed as a word problem.\n\nQuestion: Fidel tells the truth. Jerry says Fidel tells the truth. Vina says Jerry lies. Millicent says Vina tells the truth. Does Millicent tell the truth?",
"choices": ["Yes", "No"],
"answer": "B",
},
{
"id": "bbh-010",
"task": "tracking_shuffled_objects_three_objects",
"question": "Alice, Bob, and Claire are playing a game. At the start of the game, they are each holding a ball: Alice has a red ball, Bob has a blue ball, and Claire has a green ball.\n\nAs the game progresses, pairs of players trade balls. First, Alice and Bob swap balls. Then, Bob and Claire swap balls.\n\nAt the end of the game, what ball does Bob have?",
"choices": ["red ball", "blue ball", "green ball"],
"answer": "C",
},
]
class BigBenchHardBenchmark(BenchmarkSuite):
"""BIG-Bench Extra Hard: Challenging reasoning tasks from BIG-Bench."""
name = "BigBench-Extra-Hard"
description = "Challenging multi-step reasoning tasks from BIG-Bench"
category = "reasoning"
def load_dataset(self) -> list[dict[str, Any]]:
jsonl_path = Path(self.data_dir) / "bigbench_hard.jsonl"
if jsonl_path.exists():
problems: list[dict[str, Any]] = []
with open(jsonl_path) as fh:
for line in fh:
line = line.strip()
if line:
problems.append(json.loads(line))
if self.verbose:
print(f" Loaded {len(problems)} problems from {jsonl_path}")
return problems
if self.verbose:
print(f" {jsonl_path} not found — using built-in 10-problem subset")
return list(_BUILTIN_PROBLEMS)
def build_prompt(self, problem: dict[str, Any]) -> str:
question = problem["question"]
choices = problem.get("choices", [])
letters = "ABCDEFGHIJ"
choices_text = "\n".join(
f" {letters[i]}. {c}" for i, c in enumerate(choices)
)
return (
f"Answer the following reasoning question.\n\n"
f"Question: {question}\n\n"
f"Choices:\n{choices_text}\n\n"
f"Think step by step, then write ONLY the letter of the correct answer "
f"to a file called answer.txt — no explanation, just the single letter."
)
def recover_output_files(
self,
problem: dict[str, Any],
workspace: str,
agent_output: str,
metadata: dict[str, Any],
) -> None:
del problem
answer_path = Path(workspace) / "answer.txt"
if answer_path.exists():
return
match = re.search(r"\b([A-J])\b", agent_output)
if match:
answer_path.write_text(match.group(1) + "\n", encoding="utf-8")
metadata["recovered_answer_from_output"] = True
def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
pid = problem.get("id", "unknown")
expected = str(problem["answer"]).strip().upper()
answer_file = os.path.join(workspace, "answer.txt")
if not os.path.exists(answer_file):
return BenchmarkResult(
problem_id=pid, passed=False, expected=expected,
error="answer.txt not found",
)
with open(answer_file) as fh:
actual_raw = fh.read().strip()
match = re.search(r"\b([A-J])\b", actual_raw.upper())
actual = match.group(1) if match else actual_raw.strip().upper()
passed = actual == expected
return BenchmarkResult(
problem_id=pid, passed=passed,
expected=expected, actual=actual_raw,
error="" if passed else f"expected={expected}, got={actual_raw}",
)
benchmarks/suites/codeforces.py
-275
View File
@@ -1,275 +0,0 @@
"""
Codeforces benchmark suite.
Evaluates competitive programming ability using Codeforces-style problems.
Unlike pass/fail suites, this suite computes an ELO-like rating based on
problem difficulty and correctness.
Reference: https://codeforces.com/
"""
from __future__ import annotations
import json
import math
import os
import re
from pathlib import Path
from typing import Any
from .base import BenchmarkResult, BenchmarkSuite, SuiteReport
# ---------------------------------------------------------------------------
# Built-in mini dataset (10 Codeforces-style problems with difficulty ratings)
# ---------------------------------------------------------------------------
_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
{
"id": "cf-800-001",
"rating": 800,
"title": "Watermelon",
"problem": "Pete and Billy have a watermelon weighing w kilograms. They want to divide it into two parts, each weighing an even number of kilograms. Determine if this is possible.\n\nInput: A single integer w (1 ≤ w ≤ 100)\nOutput: Print YES if possible, NO otherwise.",
"test_cases": [
{"input": "8", "expected_output": "YES"},
{"input": "3", "expected_output": "NO"},
{"input": "1", "expected_output": "NO"},
{"input": "2", "expected_output": "NO"},
{"input": "4", "expected_output": "YES"},
],
},
{
"id": "cf-800-002",
"rating": 800,
"title": "Way Too Long Words",
"problem": "Abbreviate words longer than 10 characters. For such words, output the first letter, number of middle characters, and last letter.\n\nInput: First line is n (1 ≤ n ≤ 100). Next n lines each contain a word.\nOutput: For each word, output the abbreviation or the word itself if length ≤ 10.",
"test_cases": [
{"input": "4\nword\nlocalization\ninternationalization\npneumonoultramicroscopicsilicovolcanoconiosis", "expected_output": "word\nl10n\ni18n\np43s"},
{"input": "1\nabcdefghij", "expected_output": "abcdefghij"},
],
},
{
"id": "cf-1000-001",
"rating": 1000,
"title": "Nearly Lucky Number",
"problem": "A number is nearly lucky if the count of digits 4 and 7 in it is itself a lucky number (composed only of 4s and 7s). Given n, determine if it is nearly lucky.\n\nInput: A single integer n (1 ≤ n ≤ 10^18)\nOutput: YES or NO",
"test_cases": [
{"input": "40047", "expected_output": "NO"},
{"input": "7747774", "expected_output": "YES"},
{"input": "1000000000000000000", "expected_output": "NO"},
],
},
{
"id": "cf-1200-001",
"rating": 1200,
"title": "Beautiful Matrix",
"problem": "You have a 5×5 matrix with exactly one 1 and rest 0s. In one move, you can swap any two adjacent rows or columns. Find the minimum number of moves to place the 1 in the center (row 3, col 3).\n\nInput: 5 lines each with 5 space-separated integers.\nOutput: Minimum number of moves.",
"test_cases": [
{"input": "0 0 0 0 0\n0 0 0 0 1\n0 0 0 0 0\n0 0 0 0 0\n0 0 0 0 0", "expected_output": "3"},
{"input": "0 0 0 0 0\n0 0 0 0 0\n0 0 1 0 0\n0 0 0 0 0\n0 0 0 0 0", "expected_output": "0"},
],
},
{
"id": "cf-1400-001",
"rating": 1400,
"title": "Kefa and Park",
"problem": "Kefa wants to walk from the root (node 1) to any leaf in a tree. Along the path, there are cats at some nodes. Find the number of leaves reachable such that the path doesn't contain more than m consecutive cats.\n\nInput: First line: n m (n nodes, max m consecutive cats). Second line: n integers (0/1) for each node. Next n-1 lines: edges.\nOutput: Number of valid leaves.\n\nFor the built-in test, n=7 m=1, cats=[1,1,0,0,1,0,1], edges: 1-2, 1-3, 2-4, 2-5, 3-6, 3-7",
"test_cases": [
{"input": "7 1\n1 1 0 0 1 0 1\n1 2\n1 3\n2 4\n2 5\n3 6\n3 7", "expected_output": "2"},
],
},
{
"id": "cf-1600-001",
"rating": 1600,
"title": "Divisibility by Eight",
"problem": "Given a number (as a string of up to 100 digits), determine if you can delete some digits to get a non-empty number divisible by 8. Leading zeros are allowed in the result.\n\nInput: A string of digits.\nOutput: YES and the resulting number, or NO.",
"test_cases": [
{"input": "3121", "expected_output": "YES\n312"},
{"input": "123456789", "expected_output": "YES\n8"},
{"input": "3", "expected_output": "NO"},
],
},
{
"id": "cf-1800-001",
"rating": 1800,
"title": "Array Partition",
"problem": "Given an array a of n integers, determine if you can partition it into three non-empty contiguous parts such that max(part1) = min(part2) = max(part3).\n\nInput: First line n. Second line: a1...an.\nOutput: YES and the lengths of three parts, or NO.",
"test_cases": [
{"input": "5\n3 1 5 3 1", "expected_output": "YES"},
{"input": "3\n1 2 3", "expected_output": "NO"},
],
},
{
"id": "cf-2000-001",
"rating": 2000,
"title": "Count Binary Strings",
"problem": "Count the number of binary strings of length n where no two adjacent characters are both '1'. Output the answer modulo 10^9 + 7.\n\nInput: A single integer n (1 ≤ n ≤ 10^6)\nOutput: The count mod 10^9+7.",
"test_cases": [
{"input": "1", "expected_output": "2"},
{"input": "2", "expected_output": "3"},
{"input": "3", "expected_output": "5"},
{"input": "10", "expected_output": "144"},
],
},
{
"id": "cf-2200-001",
"rating": 2200,
"title": "Xor Sequences",
"problem": "Given an array of n distinct non-negative integers, find the number of pairs (i,j) where i < j such that a[i] XOR a[j] has an even number of set bits.\n\nInput: First line n. Second line: a1...an.\nOutput: Number of such pairs.",
"test_cases": [
{"input": "3\n1 2 3", "expected_output": "1"},
{"input": "4\n0 1 2 3", "expected_output": "2"},
],
},
{
"id": "cf-2500-001",
"rating": 2500,
"title": "Segment Tree Query",
"problem": "Given an array of n integers, answer q queries. Each query gives l, r and asks for the sum of elements from index l to r (1-indexed). Implement this efficiently.\n\nInput: First line: n q. Second line: a1...an. Next q lines: l r.\nOutput: q lines with answers.",
"test_cases": [
{"input": "5 3\n1 2 3 4 5\n1 3\n2 4\n1 5", "expected_output": "6\n9\n15"},
],
},
]
def _compute_elo(results: list[BenchmarkResult], problems: list[dict[str, Any]]) -> float:
"""Compute an approximate ELO rating from problem results and difficulty ratings."""
if not results:
return 0.0
# Build a mapping from problem_id to rating
rating_map = {str(p.get("id", "")): p.get("rating", 1200) for p in problems}
# Start with base rating 800
elo = 800.0
k_factor = 40.0
for result in results:
problem_rating = rating_map.get(result.problem_id, 1200)
# Expected score based on ELO difference
expected = 1.0 / (1.0 + math.pow(10, (problem_rating - elo) / 400.0))
actual_score = 1.0 if result.passed else 0.0
elo += k_factor * (actual_score - expected)
return round(max(0, elo))
class CodeforcesBenchmark(BenchmarkSuite):
"""Codeforces: Competitive programming with ELO-based scoring."""
name = "Codeforces"
description = "Competitive programming problems with ELO rating estimation"
category = "coding"
def load_dataset(self) -> list[dict[str, Any]]:
jsonl_path = Path(self.data_dir) / "codeforces.jsonl"
if jsonl_path.exists():
problems: list[dict[str, Any]] = []
with open(jsonl_path) as fh:
for line in fh:
line = line.strip()
if line:
problems.append(json.loads(line))
if self.verbose:
print(f" Loaded {len(problems)} problems from {jsonl_path}")
return problems
if self.verbose:
print(f" {jsonl_path} not found — using built-in 10-problem subset")
return list(_BUILTIN_PROBLEMS)
def build_prompt(self, problem: dict[str, Any]) -> str:
title = problem.get("title", "Problem")
text = problem["problem"]
test_cases = problem.get("test_cases", [])
examples = ""
for i, tc in enumerate(test_cases[:2], 1):
examples += f"\nExample {i}:\n Input: {tc['input']}\n Output: {tc['expected_output']}\n"
return (
f"Solve the following competitive programming problem.\n\n"
f"Title: {title}\n\n"
f"Problem:\n{text}\n"
f"{examples}\n"
f"Write a Python solution in a file called solution.py that reads from "
f"stdin and writes to stdout."
)
def setup_workspace(self, problem: dict[str, Any], workspace: str) -> None:
# Write test cases to workspace for evaluation
test_cases = problem.get("test_cases", [])
for i, tc in enumerate(test_cases):
input_file = os.path.join(workspace, f"test_input_{i}.txt")
expected_file = os.path.join(workspace, f"test_expected_{i}.txt")
with open(input_file, "w") as f:
f.write(tc["input"] + "\n")
with open(expected_file, "w") as f:
f.write(tc["expected_output"].strip() + "\n")
def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
pid = problem.get("id", "unknown")
rating = problem.get("rating", 1200)
solution_file = os.path.join(workspace, "solution.py")
if not os.path.exists(solution_file):
return BenchmarkResult(
problem_id=pid, passed=False,
error="solution.py not found",
metadata={"rating": rating},
)
test_cases = problem.get("test_cases", [])
all_passed = True
errors = []
for i, tc in enumerate(test_cases):
input_data = tc["input"]
expected = tc["expected_output"].strip()
rc, output = self._run_shell(
f"echo {repr(input_data)} | python3 solution.py",
cwd=workspace,
timeout=10.0,
)
actual = output.strip()
if rc != 0:
all_passed = False
errors.append(f"test {i}: runtime error: {output[:200]}")
elif actual != expected:
# Check if the output matches any valid answer
# (for problems with YES/NO + additional output)
if expected.startswith("YES") and actual.startswith("YES"):
pass # Accept if at least "YES" is correct
elif expected.startswith("NO") and actual.startswith("NO"):
pass
else:
all_passed = False
errors.append(f"test {i}: expected={expected!r}, got={actual!r}")
return BenchmarkResult(
problem_id=pid, passed=all_passed,
expected=f"all {len(test_cases)} tests",
actual=f"{'all passed' if all_passed else '; '.join(errors)}",
error="" if all_passed else "; ".join(errors),
metadata={"rating": rating},
)
def run_all(self) -> SuiteReport:
"""Override to add ELO computation to the report."""
report = super().run_all()
problems = self.load_dataset()
if self.limit is not None:
problems = problems[: self.limit]
elo = _compute_elo(report.results, problems)
print(f" Estimated Codeforces ELO: {elo}")
report.results.append(
BenchmarkResult(
problem_id="_elo_rating",
passed=True,
actual=str(elo),
metadata={"elo_rating": elo},
)
)
return report
benchmarks/suites/gpqa.py
-176
View File
@@ -1,176 +0,0 @@
"""
GPQA Diamond benchmark suite.
GPQA (Graduate-Level Google-Proof Question Answering) Diamond is a subset
of extremely difficult questions written by domain experts in biology,
physics, and chemistry. The "Diamond" subset is the hardest tier.
Reference: https://huggingface.co/datasets/Idavidrein/gpqa
"""
from __future__ import annotations
import json
import os
import re
from pathlib import Path
from typing import Any
from .base import BenchmarkResult, BenchmarkSuite
# ---------------------------------------------------------------------------
# Built-in mini dataset (10 representative GPQA Diamond problems)
# ---------------------------------------------------------------------------
_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
{
"id": "gpqa-001",
"subject": "physics",
"question": "A particle of mass m is confined to a one-dimensional box of length L. What is the energy difference between the first excited state and the ground state?",
"choices": ["3π²ℏ²/(2mL²)", "π²ℏ²/(2mL²)", "4π²ℏ²/(2mL²)", "2π²ℏ²/(mL²)"],
"answer": "A",
},
{
"id": "gpqa-002",
"subject": "chemistry",
"question": "Which of the following molecules has the highest bond dissociation energy?",
"choices": ["N₂", "O₂", "F₂", "CO"],
"answer": "D",
},
{
"id": "gpqa-003",
"subject": "biology",
"question": "In the lac operon, which component acts as the inducer that causes the repressor to release from the operator?",
"choices": ["Lactose", "Allolactose", "Glucose", "cAMP"],
"answer": "B",
},
{
"id": "gpqa-004",
"subject": "physics",
"question": "In quantum electrodynamics, what is the leading-order correction to the electron g-factor (anomalous magnetic moment)?",
"choices": ["α/(2π)", "α/π", "α²/(2π)", "2α/π"],
"answer": "A",
},
{
"id": "gpqa-005",
"subject": "chemistry",
"question": "What is the primary product when 2-methylpropene undergoes hydroboration-oxidation?",
"choices": ["2-methylpropan-2-ol", "2-methylpropan-1-ol", "2-methylpropanal", "isobutylene oxide"],
"answer": "B",
},
{
"id": "gpqa-006",
"subject": "biology",
"question": "Which of the following enzymes is responsible for adding a 5' cap to mRNA in eukaryotes?",
"choices": ["RNA polymerase II", "Guanylyltransferase", "Poly(A) polymerase", "RNA triphosphatase alone"],
"answer": "B",
},
{
"id": "gpqa-007",
"subject": "physics",
"question": "In general relativity, the Schwarzschild radius of a black hole with mass M is given by:",
"choices": ["2GM/c²", "GM/c²", "GM²/c", "2GM²/c²"],
"answer": "A",
},
{
"id": "gpqa-008",
"subject": "chemistry",
"question": "Which of the following is the correct order of acidity for the hydrogen halides in aqueous solution?",
"choices": ["HF > HCl > HBr > HI", "HI > HBr > HCl > HF", "HCl > HBr > HI > HF", "HF > HI > HBr > HCl"],
"answer": "B",
},
{
"id": "gpqa-009",
"subject": "biology",
"question": "What is the primary function of topoisomerase II during DNA replication?",
"choices": ["Unwinding the double helix", "Relieving positive supercoiling ahead of the replication fork", "Joining Okazaki fragments", "Proofreading newly synthesized DNA"],
"answer": "B",
},
{
"id": "gpqa-010",
"subject": "physics",
"question": "In the Standard Model, the Higgs mechanism gives mass to which fundamental particles?",
"choices": ["Only quarks", "Only W and Z bosons", "W bosons, Z bosons, and all fermions", "All particles including photons and gluons"],
"answer": "C",
},
]
class GPQABenchmark(BenchmarkSuite):
"""GPQA Diamond: Graduate-level science QA (physics, chemistry, biology)."""
name = "GPQA-Diamond"
description = "Graduate-level science multiple-choice (diamond difficulty)"
category = "knowledge"
def load_dataset(self) -> list[dict[str, Any]]:
jsonl_path = Path(self.data_dir) / "gpqa.jsonl"
if jsonl_path.exists():
problems: list[dict[str, Any]] = []
with open(jsonl_path) as fh:
for line in fh:
line = line.strip()
if line:
problems.append(json.loads(line))
if self.verbose:
print(f" Loaded {len(problems)} problems from {jsonl_path}")
return problems
if self.verbose:
print(f" {jsonl_path} not found — using built-in 10-problem subset")
return list(_BUILTIN_PROBLEMS)
def build_prompt(self, problem: dict[str, Any]) -> str:
question = problem["question"]
choices = problem.get("choices", [])
letters = "ABCD"
choices_text = "\n".join(
f" {letters[i]}. {c}" for i, c in enumerate(choices)
)
return (
f"Answer the following graduate-level science question.\n\n"
f"Question: {question}\n\n"
f"Choices:\n{choices_text}\n\n"
f"Think carefully and write ONLY the letter of the correct answer (A–D) "
f"to a file called answer.txt — no explanation, just the single letter."
)
def recover_output_files(
self,
problem: dict[str, Any],
workspace: str,
agent_output: str,
metadata: dict[str, Any],
) -> None:
del problem
answer_path = Path(workspace) / "answer.txt"
if answer_path.exists():
return
match = re.search(r"\b([A-D])\b", agent_output)
if match:
answer_path.write_text(match.group(1) + "\n", encoding="utf-8")
metadata["recovered_answer_from_output"] = True
def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
pid = problem.get("id", "unknown")
expected = str(problem["answer"]).strip().upper()
answer_file = os.path.join(workspace, "answer.txt")
if not os.path.exists(answer_file):
return BenchmarkResult(
problem_id=pid, passed=False, expected=expected,
error="answer.txt not found",
)
with open(answer_file) as fh:
actual_raw = fh.read().strip()
match = re.search(r"\b([A-D])\b", actual_raw.upper())
actual = match.group(1) if match else actual_raw.strip().upper()
passed = actual == expected
return BenchmarkResult(
problem_id=pid, passed=passed,
expected=expected, actual=actual_raw,
error="" if passed else f"expected={expected}, got={actual_raw}",
)
benchmarks/suites/gsm8k.py
-193
View File
@@ -1,193 +0,0 @@
"""
GSM8K benchmark suite.
GSM8K (Grade School Math 8K) is a dataset of 8,500 linguistically diverse
grade-school-level math word problems that require multi-step reasoning.
Paper: https://arxiv.org/abs/2110.14168
"""
from __future__ import annotations
import json
import os
import re
from pathlib import Path
from typing import Any
from .base import BenchmarkResult, BenchmarkSuite
# ---------------------------------------------------------------------------
# Built-in mini dataset (15 representative problems)
# ---------------------------------------------------------------------------
_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
{
"id": "gsm8k-001",
"question": "Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?",
"answer": "18",
},
{
"id": "gsm8k-002",
"question": "A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?",
"answer": "3",
},
{
"id": "gsm8k-003",
"question": "Josh decides to try flipping a house. He buys a house for $80,000 and then puts in $50,000 in repairs. This increased the value of the house by 150%. How much profit did he make?",
"answer": "70000",
},
{
"id": "gsm8k-004",
"question": "James decides to run 3 sprints 3 times a week. He runs 60 meters each sprint. How many total meters does he run a week?",
"answer": "540",
},
{
"id": "gsm8k-005",
"question": "Every day, Wendi feeds each of her chickens three cups of mixed chicken feed, containing seeds, mealworms and vegetables to help keep them healthy. She gives the chickens their feed in three separate meals. In the morning, she gives her flock of chickens 15 cups of feed. In the afternoon, she gives her chickens another 25 cups of feed. If Wendi's flock eats a combined total of 15 cups of feed in the final meal of the day, how many chickens does Wendi have?",
"answer": "20",
},
{
"id": "gsm8k-006",
"question": "Kylar went to the store to get water and some batteries. He spent $10 on the water and three times that on batteries. How much did he spend total?",
"answer": "40",
},
{
"id": "gsm8k-007",
"question": "Toulouse has twice as many sheep as Charleston. Charleston has 4 times as many sheep as Seattle. How many sheep do Toulouse, Charleston, and Seattle have together if Seattle has 20 sheep?",
"answer": "260",
},
{
"id": "gsm8k-008",
"question": "Carla is downloading a 200 GB file. Normally she can download 2 GB/minute, but 40% of the way through the download, Windows forces a restart to install updates, which takes 20 minutes. Then Carla has to restart the download from the beginning. How long does it take to download the file?",
"answer": "160",
},
{
"id": "gsm8k-009",
"question": "John drives for 3 hours at a speed of 60 mph and then turns around because he realizes he forgot something very important at home. He tries to get home in 4 hours but spends the first 2 hours in standstill traffic. He spends the rest of the time driving at a speed of 30 mph. How far is he from home?",
"answer": "120",
},
{
"id": "gsm8k-010",
"question": "Eliza's rate per hour for the first 40 hours she works each week is $10. She also receives an overtime pay of 1.2 times her regular hourly rate. If Eliza worked for 45 hours this week, how much are her earnings for this week?",
"answer": "460",
},
{
"id": "gsm8k-011",
"question": "A new program had 60 downloads in the first month. The number of downloads in the second month was three times as many as the downloads in the first month, but then reduced by 30% in the third month. How many downloads did the program have total over the three months?",
"answer": "366",
},
{
"id": "gsm8k-012",
"question": "Thomas has 6 more apples than friends. He also has twice as many friends as bags. If he has 20 bags, how many apples does he have?",
"answer": "46",
},
{
"id": "gsm8k-013",
"question": "A store sells pencils in packs of 8. If a teacher needs 120 pencils for her class, how many packs does she need to buy?",
"answer": "15",
},
{
"id": "gsm8k-014",
"question": "Sam bought a dozen boxes, each with 30 highlighter pens inside, for $10 each box. He rearranged five of these boxes into packages of six highlighters each and sold them for $3 per package. He sold the rest of the highlighters separately at the rate of three pens for $2. How much profit did Sam make in total, in dollars?",
"answer": "115",
},
{
"id": "gsm8k-015",
"question": "There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?",
"answer": "6",
},
]
def _extract_number(text: str) -> str | None:
"""Extract the last number from a text string."""
text = text.replace(",", "").replace("$", "").strip()
# Find all numbers (including decimals and negatives)
numbers = re.findall(r"-?\d+\.?\d*", text)
return numbers[-1] if numbers else None
class GSM8KBenchmark(BenchmarkSuite):
"""GSM8K: grade school math word problems."""
name = "GSM8K"
description = "Grade school math word problems requiring multi-step reasoning"
category = "math"
def load_dataset(self) -> list[dict[str, Any]]:
jsonl_path = Path(self.data_dir) / "gsm8k.jsonl"
if jsonl_path.exists():
problems: list[dict[str, Any]] = []
with open(jsonl_path) as fh:
for line in fh:
line = line.strip()
if line:
problems.append(json.loads(line))
if self.verbose:
print(f" Loaded {len(problems)} problems from {jsonl_path}")
return problems
if self.verbose:
print(f" {jsonl_path} not found — using built-in 15-problem subset")
return list(_BUILTIN_PROBLEMS)
def build_prompt(self, problem: dict[str, Any]) -> str:
question = problem["question"]
return (
f"Solve the following grade school math problem step by step. "
f"Write ONLY the final numerical answer (just the number) to a file called answer.txt.\n\n"
f"Problem: {question}\n\n"
f"Save just the number to answer.txt — no units, no explanation."
)
def recover_output_files(
self,
problem: dict[str, Any],
workspace: str,
agent_output: str,
metadata: dict[str, Any],
) -> None:
del problem
answer_path = Path(workspace) / "answer.txt"
if answer_path.exists():
return
number = _extract_number(agent_output)
if number is not None:
answer_path.write_text(number + "\n", encoding="utf-8")
metadata["recovered_answer_from_output"] = True
def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
pid = problem.get("id", "unknown")
expected = str(problem["answer"])
answer_file = os.path.join(workspace, "answer.txt")
if not os.path.exists(answer_file):
return BenchmarkResult(
problem_id=pid, passed=False, expected=expected,
error="answer.txt not found",
)
with open(answer_file) as fh:
actual_raw = fh.read().strip()
actual_num = _extract_number(actual_raw)
expected_num = _extract_number(expected)
if actual_num is None:
return BenchmarkResult(
problem_id=pid, passed=False,
expected=expected, actual=actual_raw,
error=f"Could not extract number from: {actual_raw}",
)
try:
passed = abs(float(actual_num) - float(expected_num)) < 1e-6
except (ValueError, TypeError):
passed = actual_num == expected_num
return BenchmarkResult(
problem_id=pid, passed=passed,
expected=expected, actual=actual_raw,
error="" if passed else f"expected={expected}, got={actual_raw}",
)
benchmarks/suites/hle.py
-199
View File
@@ -1,199 +0,0 @@
"""
HLE (Humanity's Last Exam) benchmark suite.
HLE is an extremely challenging benchmark designed to test the limits of
AI capabilities. It contains questions from diverse domains that are
intended to be among the hardest questions answerable by humans.
Reference: https://huggingface.co/datasets/cais/hle
"""
from __future__ import annotations
import json
import os
import re
from pathlib import Path
from typing import Any
from .base import BenchmarkResult, BenchmarkSuite
# ---------------------------------------------------------------------------
# Built-in mini dataset (10 representative HLE-style problems)
# ---------------------------------------------------------------------------
_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
{
"id": "hle-001",
"subject": "mathematics",
"question": "What is the smallest positive integer n such that n! ends with exactly 100 trailing zeros?",
"answer_type": "exact",
"answer": "405",
},
{
"id": "hle-002",
"subject": "physics",
"question": "In natural units (ℏ = c = 1), the fine-structure constant α ≈ 1/137. What is the approximate ratio of the electromagnetic force to the gravitational force between two protons?",
"answer_type": "multiple_choice",
"choices": ["10^36", "10^24", "10^42", "10^18"],
"answer": "A",
},
{
"id": "hle-003",
"subject": "computer_science",
"question": "What is the time complexity of the best known algorithm for matrix multiplication as of 2024?",
"answer_type": "multiple_choice",
"choices": ["O(n^2.371552)", "O(n^2.5)", "O(n^3)", "O(n^2 log n)"],
"answer": "A",
},
{
"id": "hle-004",
"subject": "mathematics",
"question": "How many groups of order 16 are there up to isomorphism?",
"answer_type": "exact",
"answer": "14",
},
{
"id": "hle-005",
"subject": "chemistry",
"question": "What is the maximum number of stereoisomers possible for a molecule with 3 chiral centers and no meso forms?",
"answer_type": "exact",
"answer": "8",
},
{
"id": "hle-006",
"subject": "biology",
"question": "The human genome contains approximately how many protein-coding genes?",
"answer_type": "multiple_choice",
"choices": ["~5,000", "~20,000", "~100,000", "~500,000"],
"answer": "B",
},
{
"id": "hle-007",
"subject": "mathematics",
"question": "What is the value of the Ramanujan sum c_5(3)?",
"answer_type": "exact",
"answer": "1",
},
{
"id": "hle-008",
"subject": "physics",
"question": "What is the spin of the Higgs boson?",
"answer_type": "exact",
"answer": "0",
},
{
"id": "hle-009",
"subject": "computer_science",
"question": "In computational complexity theory, which of the following containment relationships is known to be strict?",
"answer_type": "multiple_choice",
"choices": ["P ⊂ NP", "AC0 ⊂ NC1", "NP ⊂ PSPACE", "L ⊂ NL"],
"answer": "B",
},
{
"id": "hle-010",
"subject": "mathematics",
"question": "What is the chromatic number of the Petersen graph?",
"answer_type": "exact",
"answer": "3",
},
]
class HLEBenchmark(BenchmarkSuite):
"""HLE: Humanity's Last Exam — extremely challenging expert questions."""
name = "HLE"
description = "Extremely challenging expert-level questions (Humanity's Last Exam)"
category = "knowledge"
def load_dataset(self) -> list[dict[str, Any]]:
jsonl_path = Path(self.data_dir) / "hle.jsonl"
if jsonl_path.exists():
problems: list[dict[str, Any]] = []
with open(jsonl_path) as fh:
for line in fh:
line = line.strip()
if line:
problems.append(json.loads(line))
if self.verbose:
print(f" Loaded {len(problems)} problems from {jsonl_path}")
return problems
if self.verbose:
print(f" {jsonl_path} not found — using built-in 10-problem subset")
return list(_BUILTIN_PROBLEMS)
def build_prompt(self, problem: dict[str, Any]) -> str:
question = problem["question"]
answer_type = problem.get("answer_type", "exact")
if answer_type == "multiple_choice" and "choices" in problem:
choices = problem["choices"]
letters = "ABCD"
choices_text = "\n".join(
f" {letters[i]}. {c}" for i, c in enumerate(choices)
)
return (
f"Answer the following extremely challenging question.\n\n"
f"Question: {question}\n\n"
f"Choices:\n{choices_text}\n\n"
f"Think very carefully. Write ONLY the letter of the correct answer "
f"to a file called answer.txt — no explanation, just the single letter."
)
else:
return (
f"Answer the following extremely challenging question.\n\n"
f"Question: {question}\n\n"
f"Think very carefully. Write ONLY the final answer "
f"to a file called answer.txt — no explanation, just the answer."
)
def recover_output_files(
self,
problem: dict[str, Any],
workspace: str,
agent_output: str,
metadata: dict[str, Any],
) -> None:
del problem
answer_path = Path(workspace) / "answer.txt"
if answer_path.exists():
return
lines = agent_output.strip().splitlines()
if lines:
answer_path.write_text(lines[-1].strip() + "\n", encoding="utf-8")
metadata["recovered_answer_from_output"] = True
def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
pid = problem.get("id", "unknown")
expected = str(problem["answer"]).strip()
answer_type = problem.get("answer_type", "exact")
answer_file = os.path.join(workspace, "answer.txt")
if not os.path.exists(answer_file):
return BenchmarkResult(
problem_id=pid, passed=False, expected=expected,
error="answer.txt not found",
)
with open(answer_file) as fh:
actual_raw = fh.read().strip()
if answer_type == "multiple_choice":
match = re.search(r"\b([A-D])\b", actual_raw.upper())
actual = match.group(1) if match else actual_raw.strip().upper()
passed = actual == expected.upper()
else:
# Exact match — normalize numbers
actual = actual_raw.strip()
try:
passed = float(actual) == float(expected)
except (ValueError, TypeError):
passed = actual.lower() == expected.lower()
return BenchmarkResult(
problem_id=pid, passed=passed,
expected=expected, actual=actual_raw,
error="" if passed else f"expected={expected}, got={actual_raw}",
)
benchmarks/suites/humaneval.py
-264
View File
@@ -1,264 +0,0 @@
"""
HumanEval benchmark suite.
OpenAI's HumanEval evaluates code generation from docstrings.
164 hand-written Python programming problems with function signatures and
docstrings. The agent must produce a correct implementation.
Dataset: https://github.com/openai/human-eval
"""
from __future__ import annotations
import json
import os
import re
import subprocess
import sys
import textwrap
from pathlib import Path
from typing import Any
from .base import BenchmarkResult, BenchmarkSuite
# ---------------------------------------------------------------------------
# Built-in mini dataset (subset of 20 representative problems)
# ---------------------------------------------------------------------------
_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
{
"task_id": "HumanEval/0",
"prompt": "from typing import List\n\n\ndef has_close_elements(numbers: List[float], threshold: float) -> bool:\n \"\"\" Check if in given list of numbers, are any two numbers closer to each other than\n given threshold.\n >>> has_close_elements([1.0, 2.0, 3.0], 0.5)\n False\n >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)\n True\n \"\"\"\n",
"canonical_solution": " for idx, elem in enumerate(numbers):\n for idx2, elem2 in enumerate(numbers):\n if idx != idx2:\n distance = abs(elem - elem2)\n if distance < threshold:\n return True\n\n return False\n",
"test": "def check(candidate):\n assert candidate([1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.3) == True\n assert candidate([1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.05) == False\n assert candidate([1.0, 2.0, 5.9, 4.0, 5.0], 0.95) == True\n assert candidate([1.0, 2.0, 5.9, 4.0, 5.0], 0.8) == False\n assert candidate([1.0, 2.0, 3.0, 4.0, 5.0, 2.0], 0.1) == True\n assert candidate([1.1, 2.2, 3.1, 4.1, 5.1], 1.0) == True\n assert candidate([1.1, 2.2, 3.1, 4.1, 5.1], 0.5) == False\n",
"entry_point": "has_close_elements",
},
{
"task_id": "HumanEval/1",
"prompt": "from typing import List\n\n\ndef separate_paren_groups(paren_string: str) -> List[str]:\n \"\"\" Input to this function is a string containing multiple groups of nested parentheses. Your goal is to\n separate those group into separate strings and return the list of those.\n Separate groups are balanced (each open brace is properly closed) and not nested within each other\n Ignore any spaces in the input string.\n >>> separate_paren_groups('( ) (( )) (( )( ))')\n ['()', '(())', '(()())']\n \"\"\"\n",
"canonical_solution": " result = []\n current_string = []\n current_depth = 0\n\n for c in paren_string:\n if c == '(':\n current_depth += 1\n current_string.append(c)\n elif c == ')':\n current_depth -= 1\n current_string.append(c)\n\n if current_depth == 0:\n result.append(''.join(current_string))\n current_string.clear()\n\n return result\n",
"test": "def check(candidate):\n assert candidate('(()()) ((())) () ((())()())') == ['(()())', '((()))', '()', '((())()())']\n assert candidate('() (()) ((())) (((())))') == ['()', '(())', '((()))', '(((())))']\n assert candidate('(()(()))') == ['(()(()))']\n assert candidate('( ) (( )) (( )( ))') == ['()', '(())', '(()())']\n",
"entry_point": "separate_paren_groups",
},
{
"task_id": "HumanEval/2",
"prompt": "\n\ndef truncate_number(number: float) -> float:\n \"\"\" Given a positive floating point number, it can be decomposed into\n and integer part (largest integer smaller than given number) and decimals\n (leftover part always smaller than 1).\n\n Return the decimal part of the number.\n >>> truncate_number(3.5)\n 0.5\n \"\"\"\n",
"canonical_solution": " return number % 1.0\n",
"test": "def check(candidate):\n assert candidate(3.5) == 0.5\n assert abs(candidate(1.33) - 0.33) < 1e-6\n assert abs(candidate(123.456) - 0.456) < 1e-6\n",
"entry_point": "truncate_number",
},
{
"task_id": "HumanEval/3",
"prompt": "from typing import List\n\n\ndef below_zero(operations: List[int]) -> bool:\n \"\"\" You're given a list of deposit and withdrawal operations on a bank account that starts with\n zero balance. Your task is to detect if at any point the balance of account falls below zero, and\n at that point function should return True. Otherwise it should return False.\n >>> below_zero([1, 2, 3])\n False\n >>> below_zero([1, 2, -4, 5])\n True\n \"\"\"\n",
"canonical_solution": " balance = 0\n\n for op in operations:\n balance += op\n if balance < 0:\n return True\n\n return False\n",
"test": "def check(candidate):\n assert candidate([]) == False\n assert candidate([1, 2, -3, 1, 2, -3]) == False\n assert candidate([1, 2, -4, 5, 6]) == True\n assert candidate([1, -1, 2, -2, 5, -5, 4, -4]) == False\n assert candidate([1, -1, 2, -2, 5, -5, 4, -5]) == True\n assert candidate([1, -2, 2, -2, 5, -5, 4, -4]) == True\n",
"entry_point": "below_zero",
},
{
"task_id": "HumanEval/4",
"prompt": "from typing import List\n\n\ndef mean_absolute_deviation(numbers: List[float]) -> float:\n \"\"\" For a given list of input numbers, calculate Mean Absolute Deviation\n around the mean of this dataset.\n Mean Absolute Deviation is the average absolute difference between each\n element and a centerpoint (mean in this case):\n MAD = average | x - x_mean |\n >>> mean_absolute_deviation([1.0, 2.0, 3.0, 4.0])\n 1.0\n \"\"\"\n",
"canonical_solution": " mean = sum(numbers) / len(numbers)\n return sum(abs(x - mean) for x in numbers) / len(numbers)\n",
"test": "def check(candidate):\n assert abs(candidate([1.0, 2.0, 3.0]) - 2.0/3.0) < 1e-6\n assert abs(candidate([1.0, 2.0, 3.0, 4.0]) - 1.0) < 1e-6\n assert abs(candidate([1.0, 2.0, 3.0, 4.0, 5.0]) - 1.2) < 1e-6\n",
"entry_point": "mean_absolute_deviation",
},
{
"task_id": "HumanEval/5",
"prompt": "from typing import List\n\n\ndef intersperse(numbers: List[int], delimeter: int) -> List[int]:\n \"\"\" Insert a number 'delimeter' between every two consecutive elements of input list `numbers'\n >>> intersperse([], 4)\n []\n >>> intersperse([1, 2, 3], 4)\n [1, 4, 2, 4, 3]\n \"\"\"\n",
"canonical_solution": " if not numbers:\n return []\n\n result = []\n\n for n in numbers[:-1]:\n result.append(n)\n result.append(delimeter)\n\n result.append(numbers[-1])\n\n return result\n",
"test": "def check(candidate):\n assert candidate([], 7) == []\n assert candidate([5, 6, 3, 2], 8) == [5, 8, 6, 8, 3, 8, 2]\n assert candidate([2, 2, 2], 2) == [2, 2, 2, 2, 2]\n",
"entry_point": "intersperse",
},
{
"task_id": "HumanEval/6",
"prompt": "from typing import List\n\n\ndef parse_nested_parens(paren_string: str) -> List[int]:\n \"\"\" Input to this function is a string represented multiple groups of nested parentheses separated by spaces.\n For each of the group, output the deepest level of nesting of parentheses.\n E.g. (()()) has maximum two levels of nesting while ((())) has three.\n\n >>> parse_nested_parens('(()()) ((())) () ((())()())')\n [2, 3, 1, 3]\n \"\"\"\n",
"canonical_solution": " def parse_paren_group(s):\n depth = 0\n max_depth = 0\n for c in s:\n if c == '(':\n depth += 1\n max_depth = max(depth, max_depth)\n else:\n depth -= 1\n\n return max_depth\n\n return [parse_paren_group(x) for x in paren_string.split(' ') if x]\n",
"test": "def check(candidate):\n assert candidate('(()()) ((())) () ((())()())') == [2, 3, 1, 3]\n assert candidate('() (()) ((())) (((())))') == [1, 2, 3, 4]\n assert candidate('(()(())((())))') == [4]\n",
"entry_point": "parse_nested_parens",
},
{
"task_id": "HumanEval/7",
"prompt": "from typing import List\n\n\ndef filter_by_substring(strings: List[str], substring: str) -> List[str]:\n \"\"\" Filter an input list of strings only for ones that contain given substring\n >>> filter_by_substring([], 'a')\n []\n >>> filter_by_substring(['abc', 'bacd', 'cde', 'array'], 'a')\n ['abc', 'bacd', 'array']\n \"\"\"\n",
"canonical_solution": " return [x for x in strings if substring in x]\n",
"test": "def check(candidate):\n assert candidate([], 'john') == []\n assert candidate(['xxx', 'asd', 'xxy', 'john doe', 'xxxuj', 'xxx'], 'xxx') == ['xxx', 'xxxuj', 'xxx']\n assert candidate(['xxx', 'asd', 'aaber', 'john doe', 'xxxuj', 'xxx'], 'john') == ['john doe']\n assert candidate(['grunt', 'hierarchies', 'aaaxyz'], 'hi') == ['hierarchies']\n",
"entry_point": "filter_by_substring",
},
{
"task_id": "HumanEval/8",
"prompt": "from typing import List, Tuple\n\n\ndef sum_product(numbers: List[int]) -> Tuple[int, int]:\n \"\"\" For a given list of integers, return a tuple consisting of a sum and a product of all the integers in a list.\n Empty sum should be equal to 0 and empty product should be equal to 1.\n >>> sum_product([])\n (0, 1)\n >>> sum_product([1, 2, 3, 4])\n (10, 24)\n \"\"\"\n",
"canonical_solution": " sum_value = 0\n prod_value = 1\n\n for n in numbers:\n sum_value += n\n prod_value *= n\n\n return sum_value, prod_value\n",
"test": "def check(candidate):\n assert candidate([]) == (0, 1)\n assert candidate([1, 1, 1]) == (3, 1)\n assert candidate([100, 0]) == (100, 0)\n assert candidate([3, 5, 7]) == (15, 105)\n assert candidate([10]) == (10, 10)\n",
"entry_point": "sum_product",
},
{
"task_id": "HumanEval/9",
"prompt": "from typing import List, Tuple\n\n\ndef rolling_max(numbers: List[int]) -> List[int]:\n \"\"\" From a given list of integers, generate a list of rolling maximum element found until given moment\n in the sequence.\n >>> rolling_max([1, 2, 3, 2, 3, 4, 2])\n [1, 2, 3, 3, 3, 4, 4]\n \"\"\"\n",
"canonical_solution": " running_max = None\n result = []\n\n for n in numbers:\n if running_max is None:\n running_max = n\n else:\n running_max = max(running_max, n)\n\n result.append(running_max)\n\n return result\n",
"test": "def check(candidate):\n assert candidate([]) == []\n assert candidate([1, 2, 3, 4]) == [1, 2, 3, 4]\n assert candidate([4, 3, 2, 1]) == [4, 4, 4, 4]\n assert candidate([3, 3, 3, 3]) == [3, 3, 3, 3]\n",
"entry_point": "rolling_max",
},
{
"task_id": "HumanEval/10",
"prompt": "\n\ndef is_palindrome(string: str) -> bool:\n \"\"\" Test if given string is a palindrome \"\"\"\n return string == string[::-1]\n\n\ndef make_palindrome(string: str) -> str:\n \"\"\" Find the shortest palindrome that begins with a supplied string.\n Algorithm idea is simple:\n - Find the longest postfix of supplied string that is a palindrome.\n - Append to the end of the string reverse of a string prefix that comes before the palindromic suffix.\n >>> make_palindrome('')\n ''\n >>> make_palindrome('cat')\n 'catac'\n >>> make_palindrome('cata')\n 'catac'\n \"\"\"\n",
"canonical_solution": " if not string:\n return ''\n\n beginning_of_suffix = 0\n\n while not is_palindrome(string[beginning_of_suffix:]):\n beginning_of_suffix += 1\n\n return string + string[:beginning_of_suffix][::-1]\n",
"test": "def check(candidate):\n assert candidate('') == ''\n assert candidate('x') == 'x'\n assert candidate('xyz') == 'xyzyx'\n assert candidate('xyx') == 'xyx'\n assert candidate('jerry') == 'jerryrrej'\n",
"entry_point": "make_palindrome",
},
{
"task_id": "HumanEval/11",
"prompt": "from typing import List\n\n\ndef string_xor(a: str, b: str) -> str:\n \"\"\" Input are two strings a and b consisting only of 1s and 0s.\n Perform binary XOR on these inputs and return result also as a string.\n >>> string_xor('010', '110')\n '100'\n \"\"\"\n",
"canonical_solution": " def xor(i, j):\n if i == j:\n return '0'\n else:\n return '1'\n\n return ''.join(xor(x, y) for x, y in zip(a, b))\n",
"test": "def check(candidate):\n assert candidate('111000', '101010') == '010010'\n assert candidate('1', '1') == '0'\n assert candidate('0101', '0000') == '0101'\n",
"entry_point": "string_xor",
},
{
"task_id": "HumanEval/12",
"prompt": "from typing import List, Optional\n\n\ndef longest(strings: List[str]) -> Optional[str]:\n \"\"\" Out of list of strings, return the longest one. Return the first one in case of multiple\n strings of the same length. Return None in case the input list is empty.\n >>> longest([])\n\n >>> longest(['a', 'b', 'c'])\n 'a'\n >>> longest(['a', 'bb', 'ccc'])\n 'ccc'\n \"\"\"\n",
"canonical_solution": " if not strings:\n return None\n\n maxlen = max(len(x) for x in strings)\n for s in strings:\n if len(s) == maxlen:\n return s\n",
"test": "def check(candidate):\n assert candidate([]) == None\n assert candidate(['x', 'y', 'z']) == 'x'\n assert candidate(['x', 'yyy', 'zzzz', 'www', 'kkkk', 'abc']) == 'zzzz'\n",
"entry_point": "longest",
},
{
"task_id": "HumanEval/13",
"prompt": "\n\ndef greatest_common_divisor(a: int, b: int) -> int:\n \"\"\" Return a greatest common divisor of two integers a and b\n >>> greatest_common_divisor(3, 5)\n 1\n >>> greatest_common_divisor(25, 15)\n 5\n \"\"\"\n",
"canonical_solution": " while b:\n a, b = b, a % b\n return a\n",
"test": "def check(candidate):\n assert candidate(3, 7) == 1\n assert candidate(10, 15) == 5\n assert candidate(49, 14) == 7\n assert candidate(144, 60) == 12\n",
"entry_point": "greatest_common_divisor",
},
{
"task_id": "HumanEval/14",
"prompt": "from typing import List\n\n\ndef all_prefixes(string: str) -> List[str]:\n \"\"\" Return list of all prefixes from shortest to longest of the input string\n >>> all_prefixes('abc')\n ['a', 'ab', 'abc']\n \"\"\"\n",
"canonical_solution": " result = []\n\n for i in range(len(string)):\n result.append(string[:i+1])\n return result\n",
"test": "def check(candidate):\n assert candidate('') == []\n assert candidate('asdfgh') == ['a', 'as', 'asd', 'asdf', 'asdfe', 'asdfgh']\n assert candidate('WWW') == ['W', 'WW', 'WWW']\n",
"entry_point": "all_prefixes",
},
{
"task_id": "HumanEval/15",
"prompt": "\n\ndef string_sequence(n: int) -> str:\n \"\"\" Return a string containing space-delimited numbers starting from 0 upto n inclusive.\n >>> string_sequence(0)\n '0'\n >>> string_sequence(5)\n '0 1 2 3 4 5'\n \"\"\"\n",
"canonical_solution": " return ' '.join([str(x) for x in range(n + 1)])\n",
"test": "def check(candidate):\n assert candidate(0) == '0'\n assert candidate(3) == '0 1 2 3'\n assert candidate(10) == '0 1 2 3 4 5 6 7 8 9 10'\n",
"entry_point": "string_sequence",
},
{
"task_id": "HumanEval/16",
"prompt": "\n\ndef count_distinct_characters(string: str) -> int:\n \"\"\" Given a string, find out how many distinct characters (regardless of case) does it consist of\n >>> count_distinct_characters('xyzXYZ')\n 3\n >>> count_distinct_characters('Jerry')\n 4\n \"\"\"\n",
"canonical_solution": " return len(set(string.lower()))\n",
"test": "def check(candidate):\n assert candidate('') == 0\n assert candidate('abcde') == 5\n assert candidate('abcde' + 'cade' + 'CADE') == 5\n assert candidate('aaaaAAAAaaaa') == 1\n assert candidate('Jerry jbread') == 7\n",
"entry_point": "count_distinct_characters",
},
{
"task_id": "HumanEval/17",
"prompt": "from typing import List\n\n\ndef parse_music(music_string: str) -> List[int]:\n \"\"\" Input to this function is a string representing musical notes in a special ASCII format.\n Your task is to parse this string and return list of integers corresponding to how many beats does each\n not last.\n\n Here is a legend:\n 'o' - whole note, lasts four beats\n 'o|' - half note, lasts two beats\n '.|' - quarter note, lasts one beat\n\n >>> parse_music('o o| .| o| o| .| .| .| .| o o')\n [4, 2, 1, 2, 2, 1, 1, 1, 1, 4, 4]\n \"\"\"\n",
"canonical_solution": " note_map = {'o': 4, 'o|': 2, '.|': 1}\n return [note_map[x] for x in music_string.split(' ') if x]\n",
"test": "def check(candidate):\n assert candidate('') == []\n assert candidate('o o o o') == [4, 4, 4, 4]\n assert candidate('.| .| .| .|') == [1, 1, 1, 1]\n assert candidate('o| o| .| .| o o o o') == [2, 2, 1, 1, 4, 4, 4, 4]\n assert candidate('o| .| o| .| o o| o o|') == [2, 1, 2, 1, 4, 2, 4, 2]\n",
"entry_point": "parse_music",
},
{
"task_id": "HumanEval/18",
"prompt": "\n\ndef how_many_times(string: str, substring: str) -> int:\n \"\"\" Find how many times a given substring can be found in the original string. Count overlapping cases.\n >>> how_many_times('', 'a')\n 0\n >>> how_many_times('aaa', 'a')\n 3\n >>> how_many_times('aaaa', 'aa')\n 3\n \"\"\"\n",
"canonical_solution": " times = 0\n\n for i in range(len(string) - len(substring) + 1):\n if string[i:i+len(substring)] == substring:\n times += 1\n\n return times\n",
"test": "def check(candidate):\n assert candidate('', 'x') == 0\n assert candidate('xyxyxyx', 'x') == 4\n assert candidate('cacacacac', 'cac') == 4\n assert candidate('john doe', 'john') == 1\n",
"entry_point": "how_many_times",
},
{
"task_id": "HumanEval/19",
"prompt": "from typing import List\n\n\ndef sort_numbers(numbers: str) -> str:\n \"\"\" Input is a space-delimited string of numberals from 'zero' to 'nine'.\n Valid choices are 'zero', 'one', 'two', 'three', 'four', 'five', 'six', 'seven', 'eight' and 'nine'.\n Return the string with numbers sorted from smallest to largest\n >>> sort_numbers('three one five')\n 'one three five'\n \"\"\"\n",
"canonical_solution": " value_map = {\n 'zero': 0,\n 'one': 1,\n 'two': 2,\n 'three': 3,\n 'four': 4,\n 'five': 5,\n 'six': 6,\n 'seven': 7,\n 'eight': 8,\n 'nine': 9\n }\n return ' '.join(sorted([x for x in numbers.split(' ') if x], key=lambda x: value_map[x]))\n",
"test": "def check(candidate):\n assert candidate('') == ''\n assert candidate('three') == 'three'\n assert candidate('three five nine') == 'three five nine'\n assert candidate('five zero four seven nine eight') == 'zero four five seven eight nine'\n assert candidate('six five four three two one zero') == 'zero one two three four five six'\n",
"entry_point": "sort_numbers",
},
]
class HumanEvalBenchmark(BenchmarkSuite):
"""HumanEval: code generation from docstrings (164 problems)."""
name = "HumanEval"
description = "Code generation from Python function docstrings"
category = "coding"
def load_dataset(self) -> list[dict[str, Any]]:
"""Load HumanEval problems.
Checks for a JSONL file at ``data_dir/humaneval.jsonl`` first.
Falls back to the built-in 20-problem subset.
"""
jsonl_path = Path(self.data_dir) / "humaneval.jsonl"
if jsonl_path.exists():
problems: list[dict[str, Any]] = []
with open(jsonl_path) as fh:
for line in fh:
line = line.strip()
if line:
problems.append(json.loads(line))
if self.verbose:
print(f" Loaded {len(problems)} problems from {jsonl_path}")
return problems
if self.verbose:
print(
f" {jsonl_path} not found — using built-in 20-problem subset"
)
return list(_BUILTIN_PROBLEMS)
def build_prompt(self, problem: dict[str, Any]) -> str:
prompt_code = problem["prompt"]
entry = problem["entry_point"]
return (
f"Complete the following Python function. Write ONLY the function body. "
f"Save the complete file (including the signature shown below) as solution.py.\n\n"
f"```python\n{prompt_code}```\n\n"
f"The function name is `{entry}`. Make sure the file is valid Python."
)
def setup_workspace(self, problem: dict[str, Any], workspace: str) -> None:
# Write the test harness so we can verify after the agent runs
test_code = problem["test"]
entry = problem["entry_point"]
harness = (
"import sys\n"
'sys.path.insert(0, ".")\n'
f"from solution import {entry}\n\n"
f"{test_code}\n\n"
f"check({entry})\n"
'print("ALL_TESTS_PASSED")\n'
)
with open(os.path.join(workspace, "test_harness.py"), "w") as fh:
fh.write(harness)
def recover_output_files(
self,
problem: dict[str, Any],
workspace: str,
agent_output: str,
metadata: dict[str, Any],
) -> None:
del problem
solution_path = Path(workspace) / "solution.py"
if solution_path.exists():
return
blocks = re.findall(r"```(?:python)?\s*(.*?)```", agent_output, flags=re.DOTALL | re.IGNORECASE)
for block in blocks:
candidate = block.strip()
if "def " in candidate or "import " in candidate or "from " in candidate:
solution_path.write_text(candidate.rstrip() + "\n", encoding="utf-8")
metadata["recovered_solution_from_output"] = True
return
def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
pid = problem.get("task_id", problem.get("id", "unknown"))
sol = os.path.join(workspace, "solution.py")
if not os.path.exists(sol):
return BenchmarkResult(
problem_id=pid, passed=False, error="solution.py not found"
)
code, output = self._run_shell(
f"{sys.executable} test_harness.py", cwd=workspace, timeout=30.0
)
passed = code == 0 and "ALL_TESTS_PASSED" in output
return BenchmarkResult(
problem_id=pid,
passed=passed,
actual=output[:500],
error="" if passed else output[:500],
)
benchmarks/suites/ifeval.py
-248
View File
@@ -1,248 +0,0 @@
"""
IFEval benchmark suite.
IFEval (Instruction Following Evaluation) tests whether a model can
follow verifiable natural-language instructions — e.g., "write at least
200 words", "include exactly 3 bullet points", "respond in all lowercase".
Paper: https://arxiv.org/abs/2311.07911
"""
from __future__ import annotations
import json
import os
import re
from pathlib import Path
from typing import Any
from .base import BenchmarkResult, BenchmarkSuite
# ---------------------------------------------------------------------------
# Built-in mini dataset — verifiable instruction-following tasks
# ---------------------------------------------------------------------------
_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
{
"id": "ifeval-001",
"instruction": "Write a short paragraph about the benefits of exercise. Your response must be entirely in lowercase. Save it to output.txt.",
"checks": [
{"type": "file_exists", "file": "output.txt"},
{"type": "all_lowercase", "file": "output.txt"},
{"type": "min_words", "file": "output.txt", "value": 20},
],
},
{
"id": "ifeval-002",
"instruction": "Write a poem about the ocean. It must have exactly 4 lines. Save it to output.txt.",
"checks": [
{"type": "file_exists", "file": "output.txt"},
{"type": "exact_line_count", "file": "output.txt", "value": 4},
],
},
{
"id": "ifeval-003",
"instruction": "List 5 programming languages. Format each as a numbered item (1. Language). Save to output.txt.",
"checks": [
{"type": "file_exists", "file": "output.txt"},
{"type": "min_lines", "file": "output.txt", "value": 5},
{"type": "contains_pattern", "file": "output.txt", "pattern": r"^\d+\.", "min_count": 5},
],
},
{
"id": "ifeval-004",
"instruction": "Write a summary of what Python is in EXACTLY 3 sentences. Each sentence must end with a period. Save to output.txt.",
"checks": [
{"type": "file_exists", "file": "output.txt"},
{"type": "exact_sentence_count", "file": "output.txt", "value": 3},
],
},
{
"id": "ifeval-005",
"instruction": "Write a response that contains the word 'innovation' at least 3 times. The response should be about technology. Save to output.txt.",
"checks": [
{"type": "file_exists", "file": "output.txt"},
{"type": "word_frequency", "file": "output.txt", "word": "innovation", "min_count": 3},
],
},
{
"id": "ifeval-006",
"instruction": "Write a short paragraph about space exploration. It must contain exactly 2 bullet points, each starting with '- '. Save to output.txt.",
"checks": [
{"type": "file_exists", "file": "output.txt"},
{"type": "contains_pattern", "file": "output.txt", "pattern": r"^- ", "min_count": 2, "max_count": 2},
],
},
{
"id": "ifeval-007",
"instruction": "Write the numbers from 1 to 10, each on a separate line. Save to output.txt.",
"checks": [
{"type": "file_exists", "file": "output.txt"},
{"type": "exact_line_count", "file": "output.txt", "value": 10},
{"type": "contains", "file": "output.txt", "text": "1"},
{"type": "contains", "file": "output.txt", "text": "10"},
],
},
{
"id": "ifeval-008",
"instruction": "Write a haiku (3 lines: 5 syllables, 7 syllables, 5 syllables) about nature. Save to output.txt. The response must be exactly 3 lines.",
"checks": [
{"type": "file_exists", "file": "output.txt"},
{"type": "exact_line_count", "file": "output.txt", "value": 3},
],
},
{
"id": "ifeval-009",
"instruction": "Write a paragraph about artificial intelligence. Every sentence must start with the word 'AI'. Save to output.txt.",
"checks": [
{"type": "file_exists", "file": "output.txt"},
{"type": "sentences_start_with", "file": "output.txt", "prefix": "AI"},
],
},
{
"id": "ifeval-010",
"instruction": "Create a JSON file called output.txt containing an object with exactly 3 keys: 'name', 'age', and 'city'. Values can be anything.",
"checks": [
{"type": "file_exists", "file": "output.txt"},
{"type": "valid_json", "file": "output.txt"},
{"type": "json_has_keys", "file": "output.txt", "keys": ["name", "age", "city"]},
],
},
]
def _run_check(check: dict[str, Any], workspace: str) -> bool:
"""Run a single verification check. Returns True if passed."""
check_type = check["type"]
filepath = os.path.join(workspace, check.get("file", "output.txt"))
if check_type == "file_exists":
return os.path.exists(filepath)
if not os.path.exists(filepath):
return False
with open(filepath) as fh:
content = fh.read()
lines = [l for l in content.strip().split("\n") if l.strip()]
if check_type == "all_lowercase":
# Check that alphabetic characters are lowercase
alpha_chars = [c for c in content if c.isalpha()]
return all(c.islower() for c in alpha_chars) if alpha_chars else True
if check_type == "min_words":
words = content.split()
return len(words) >= check["value"]
if check_type == "exact_line_count":
return len(lines) == check["value"]
if check_type == "min_lines":
return len(lines) >= check["value"]
if check_type == "contains_pattern":
pattern = check["pattern"]
matches = sum(1 for line in lines if re.search(pattern, line))
if "min_count" in check and matches < check["min_count"]:
return False
if "max_count" in check and matches > check["max_count"]:
return False
return True
if check_type == "exact_sentence_count":
# Count sentences ending with . ! or ?
sentences = re.split(r"[.!?]+", content.strip())
sentences = [s.strip() for s in sentences if s.strip()]
return len(sentences) == check["value"]
if check_type == "word_frequency":
word = check["word"].lower()
count = content.lower().count(word)
return count >= check["min_count"]
if check_type == "contains":
return check["text"] in content
if check_type == "sentences_start_with":
prefix = check["prefix"]
sentences = re.split(r"[.!?]\s+", content.strip())
sentences = [s.strip() for s in sentences if s.strip()]
return all(s.startswith(prefix) for s in sentences) if sentences else False
if check_type == "valid_json":
try:
json.loads(content)
return True
except json.JSONDecodeError:
return False
if check_type == "json_has_keys":
try:
data = json.loads(content)
return all(k in data for k in check["keys"])
except (json.JSONDecodeError, TypeError):
return False
return False
class IFEvalBenchmark(BenchmarkSuite):
"""IFEval: instruction following evaluation."""
name = "IFEval"
description = "Verifiable instruction-following evaluation"
category = "instruction-following"
def load_dataset(self) -> list[dict[str, Any]]:
jsonl_path = Path(self.data_dir) / "ifeval.jsonl"
if jsonl_path.exists():
problems: list[dict[str, Any]] = []
with open(jsonl_path) as fh:
for line in fh:
line = line.strip()
if line:
problems.append(json.loads(line))
if self.verbose:
print(f" Loaded {len(problems)} problems from {jsonl_path}")
return problems
if self.verbose:
print(f" {jsonl_path} not found — using built-in 10-problem subset")
return list(_BUILTIN_PROBLEMS)
def build_prompt(self, problem: dict[str, Any]) -> str:
return problem["instruction"]
def recover_output_files(
self,
problem: dict[str, Any],
workspace: str,
agent_output: str,
metadata: dict[str, Any],
) -> None:
del problem
output_path = Path(workspace) / "output.txt"
if output_path.exists():
return
text = agent_output.strip()
if text:
output_path.write_text(text + "\n", encoding="utf-8")
metadata["recovered_output_from_agent_text"] = True
def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
pid = problem.get("id", "unknown")
checks = problem.get("checks", [])
failed_checks: list[str] = []
for check in checks:
if not _run_check(check, workspace):
failed_checks.append(check["type"])
passed = len(failed_checks) == 0
return BenchmarkResult(
problem_id=pid, passed=passed,
error=f"Failed checks: {', '.join(failed_checks)}" if failed_checks else "",
metadata={"total_checks": len(checks), "failed_checks": failed_checks},
)
benchmarks/suites/livecodebench.py
-186
View File
@@ -1,186 +0,0 @@
"""
LiveCodeBench benchmark suite.
LiveCodeBench evaluates coding agents on competitive-programming-style
problems that were published *after* the model's training cutoff, preventing
data contamination.
Reference: https://livecodebench.github.io/
"""
from __future__ import annotations
import json
import os
import sys
import textwrap
from pathlib import Path
from typing import Any
from .base import BenchmarkResult, BenchmarkSuite
# ---------------------------------------------------------------------------
# Built-in mini dataset — competitive-programming style problems
# ---------------------------------------------------------------------------
_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
{
"id": "lcb-001",
"title": "Two Sum Sorted",
"description": (
"Given a sorted array of integers `nums` and a target integer `target`, "
"return the indices (0-based) of the two numbers that add up to target. "
"You may assume each input has exactly one solution, and you may not use "
"the same element twice. Return them as a list [i, j] where i < j."
),
"test_cases": [
{"input": {"nums": [2, 7, 11, 15], "target": 9}, "expected": [0, 1]},
{"input": {"nums": [1, 3, 5, 7], "target": 8}, "expected": [0, 3]},
{"input": {"nums": [-1, 0, 3, 5], "target": 4}, "expected": [0, 3]},
{"input": {"nums": [1, 2], "target": 3}, "expected": [0, 1]},
],
"function_name": "two_sum_sorted",
},
{
"id": "lcb-002",
"title": "Maximum Subarray",
"description": (
"Given an array of integers `nums`, find the contiguous subarray "
"(containing at least one number) which has the largest sum and return its sum."
),
"test_cases": [
{"input": {"nums": [-2, 1, -3, 4, -1, 2, 1, -5, 4]}, "expected": 6},
{"input": {"nums": [1]}, "expected": 1},
{"input": {"nums": [-1, -2, -3]}, "expected": -1},
{"input": {"nums": [5, 4, -1, 7, 8]}, "expected": 23},
],
"function_name": "max_subarray",
},
{
"id": "lcb-003",
"title": "Valid Parentheses",
"description": (
"Given a string `s` containing just the characters '(', ')', '{', '}', "
"'[' and ']', determine if the input string is valid. An input string is "
"valid if open brackets are closed by the same type of brackets in the "
"correct order."
),
"test_cases": [
{"input": {"s": "()"}, "expected": True},
{"input": {"s": "()[]{}"}, "expected": True},
{"input": {"s": "(]"}, "expected": False},
{"input": {"s": "([)]"}, "expected": False},
{"input": {"s": "{[]}"}, "expected": True},
{"input": {"s": ""}, "expected": True},
],
"function_name": "is_valid",
},
{
"id": "lcb-004",
"title": "Longest Common Prefix",
"description": (
"Write a function that finds the longest common prefix string "
"amongst a list of strings. If there is no common prefix, return "
"an empty string."
),
"test_cases": [
{"input": {"strs": ["flower", "flow", "flight"]}, "expected": "fl"},
{"input": {"strs": ["dog", "racecar", "car"]}, "expected": ""},
{"input": {"strs": ["abc"]}, "expected": "abc"},
{"input": {"strs": ["", "b"]}, "expected": ""},
{"input": {"strs": ["ab", "ab", "ab"]}, "expected": "ab"},
],
"function_name": "longest_common_prefix",
},
{
"id": "lcb-005",
"title": "Count Inversions",
"description": (
"Given an array of integers, count the number of inversions. "
"An inversion is a pair (i, j) where i < j but nums[i] > nums[j]. "
"Return the total number of inversions."
),
"test_cases": [
{"input": {"nums": [2, 4, 1, 3, 5]}, "expected": 3},
{"input": {"nums": [1, 2, 3, 4]}, "expected": 0},
{"input": {"nums": [4, 3, 2, 1]}, "expected": 6},
{"input": {"nums": [1]}, "expected": 0},
],
"function_name": "count_inversions",
},
]
class LiveCodeBenchBenchmark(BenchmarkSuite):
"""LiveCodeBench: competitive programming problems."""
name = "LiveCodeBench"
description = "Competitive programming problems (post-training-cutoff)"
category = "coding"
def load_dataset(self) -> list[dict[str, Any]]:
jsonl_path = Path(self.data_dir) / "livecodebench.jsonl"
if jsonl_path.exists():
problems: list[dict[str, Any]] = []
with open(jsonl_path) as fh:
for line in fh:
line = line.strip()
if line:
problems.append(json.loads(line))
if self.verbose:
print(f" Loaded {len(problems)} problems from {jsonl_path}")
return problems
if self.verbose:
print(f" {jsonl_path} not found — using built-in 5-problem subset")
return list(_BUILTIN_PROBLEMS)
def build_prompt(self, problem: dict[str, Any]) -> str:
desc = problem["description"]
fname = problem["function_name"]
# Build example from first test case
tc = problem["test_cases"][0]
args = ", ".join(f"{k}={repr(v)}" for k, v in tc["input"].items())
expected = repr(tc["expected"])
return (
f"Solve the following competitive programming problem.\n\n"
f"## {problem.get('title', 'Problem')}\n\n"
f"{desc}\n\n"
f"Example: {fname}({args}) => {expected}\n\n"
f"Write a Python function called `{fname}` and save it to solution.py."
)
def setup_workspace(self, problem: dict[str, Any], workspace: str) -> None:
fname = problem["function_name"]
test_lines = [
"import sys",
'sys.path.insert(0, ".")',
f"from solution import {fname}",
"",
]
for i, tc in enumerate(problem["test_cases"]):
args = ", ".join(f"{k}={repr(v)}" for k, v in tc["input"].items())
test_lines.append(
f"assert {fname}({args}) == {repr(tc['expected'])}, "
f"'Test case {i+1} failed'"
)
test_lines.append("")
test_lines.append('print("ALL_TESTS_PASSED")')
with open(os.path.join(workspace, "test_harness.py"), "w") as fh:
fh.write("\n".join(test_lines))
def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
pid = problem.get("id", "unknown")
sol = os.path.join(workspace, "solution.py")
if not os.path.exists(sol):
return BenchmarkResult(
problem_id=pid, passed=False, error="solution.py not found"
)
code, output = self._run_shell(
f"{sys.executable} test_harness.py", cwd=workspace, timeout=30.0
)
passed = code == 0 and "ALL_TESTS_PASSED" in output
return BenchmarkResult(
problem_id=pid, passed=passed, actual=output[:500],
error="" if passed else output[:500],
)
benchmarks/suites/math_bench.py
-228
View File
@@ -1,228 +0,0 @@
"""
MATH benchmark suite.
The MATH dataset consists of 12,500 problems from mathematics competitions
(AMC, AIME, etc.) covering topics like algebra, counting, geometry, number
theory, and more. Difficulty levels 1–5.
Paper: https://arxiv.org/abs/2103.03874
"""
from __future__ import annotations
import json
import os
import re
import sys
from pathlib import Path
from typing import Any
from .base import BenchmarkResult, BenchmarkSuite
# ---------------------------------------------------------------------------
# Built-in mini dataset (15 representative problems)
# ---------------------------------------------------------------------------
_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
{
"id": "math-001",
"problem": "What is the value of $2^{10}$?",
"answer": "1024",
"subject": "algebra",
"level": 1,
},
{
"id": "math-002",
"problem": "What is $15 \\% $ of $200$?",
"answer": "30",
"subject": "algebra",
"level": 1,
},
{
"id": "math-003",
"problem": "Solve for $x$: $3x + 7 = 22$.",
"answer": "5",
"subject": "algebra",
"level": 1,
},
{
"id": "math-004",
"problem": "What is the sum of the first 10 positive integers?",
"answer": "55",
"subject": "algebra",
"level": 1,
},
{
"id": "math-005",
"problem": "What is $\\gcd(12, 18)$?",
"answer": "6",
"subject": "number_theory",
"level": 1,
},
{
"id": "math-006",
"problem": "How many ways can you choose 3 items from a set of 5 items?",
"answer": "10",
"subject": "counting",
"level": 2,
},
{
"id": "math-007",
"problem": "What is the area of a triangle with base 10 and height 6?",
"answer": "30",
"subject": "geometry",
"level": 1,
},
{
"id": "math-008",
"problem": "Evaluate $\\sum_{k=1}^{5} k^2$.",
"answer": "55",
"subject": "algebra",
"level": 2,
},
{
"id": "math-009",
"problem": "Find the remainder when $2^{20}$ is divided by $7$.",
"answer": "4",
"subject": "number_theory",
"level": 3,
},
{
"id": "math-010",
"problem": "What is the least common multiple of 12 and 18?",
"answer": "36",
"subject": "number_theory",
"level": 2,
},
{
"id": "math-011",
"problem": "A bag contains 3 red and 5 blue marbles. What is the probability of drawing a red marble? Express as a simplified fraction.",
"answer": "3/8",
"subject": "counting",
"level": 2,
},
{
"id": "math-012",
"problem": "What is the value of $\\sqrt{144}$?",
"answer": "12",
"subject": "algebra",
"level": 1,
},
{
"id": "math-013",
"problem": "If $f(x) = 2x^2 - 3x + 1$, what is $f(3)$?",
"answer": "10",
"subject": "algebra",
"level": 2,
},
{
"id": "math-014",
"problem": "How many prime numbers are there between 1 and 20?",
"answer": "8",
"subject": "number_theory",
"level": 2,
},
{
"id": "math-015",
"problem": "What is the value of $3! + 4!$?",
"answer": "30",
"subject": "counting",
"level": 1,
},
]
def _normalize_answer(text: str) -> str:
"""Normalize a math answer for comparison.
Strips whitespace, dollar signs, \\boxed{}, etc.
"""
text = text.strip()
# Remove \boxed{...}
m = re.search(r"\\boxed\{(.+?)\}", text)
if m:
text = m.group(1)
# Remove dollar signs and whitespace
text = text.replace("$", "").replace(",", "").strip()
# Try to normalize fractions
text = text.replace("\\frac{", "").replace("}{", "/").replace("}", "")
return text.strip()
class MATHBenchmark(BenchmarkSuite):
"""MATH: competition mathematics problems."""
name = "MATH"
description = "Competition mathematics problems (algebra, geometry, number theory, etc.)"
category = "math"
def load_dataset(self) -> list[dict[str, Any]]:
jsonl_path = Path(self.data_dir) / "math.jsonl"
if jsonl_path.exists():
problems: list[dict[str, Any]] = []
with open(jsonl_path) as fh:
for line in fh:
line = line.strip()
if line:
problems.append(json.loads(line))
if self.verbose:
print(f" Loaded {len(problems)} problems from {jsonl_path}")
return problems
if self.verbose:
print(f" {jsonl_path} not found — using built-in 15-problem subset")
return list(_BUILTIN_PROBLEMS)
def build_prompt(self, problem: dict[str, Any]) -> str:
prob = problem["problem"]
return (
f"Solve the following math problem. Write ONLY the final numerical "
f"answer (a single number or simple expression) to a file called answer.txt.\n\n"
f"Problem: {prob}\n\n"
f"Save just the answer to answer.txt — no explanation, no work, just the answer."
)
def recover_output_files(
self,
problem: dict[str, Any],
workspace: str,
agent_output: str,
metadata: dict[str, Any],
) -> None:
del problem
answer_path = Path(workspace) / "answer.txt"
if answer_path.exists():
return
answer = _normalize_answer(agent_output)
if answer:
answer_path.write_text(answer + "\n", encoding="utf-8")
metadata["recovered_answer_from_output"] = True
def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
pid = problem.get("id", str(problem.get("task_id", "unknown")))
expected_raw = str(problem["answer"])
answer_file = os.path.join(workspace, "answer.txt")
if not os.path.exists(answer_file):
return BenchmarkResult(
problem_id=pid, passed=False, expected=expected_raw,
error="answer.txt not found",
)
with open(answer_file) as fh:
actual_raw = fh.read().strip()
expected_norm = _normalize_answer(expected_raw)
actual_norm = _normalize_answer(actual_raw)
# Try numeric comparison
try:
passed = abs(float(actual_norm) - float(expected_norm)) < 1e-6
except (ValueError, ZeroDivisionError):
passed = actual_norm == expected_norm
return BenchmarkResult(
problem_id=pid, passed=passed,
expected=expected_raw, actual=actual_raw,
error="" if passed else f"expected={expected_raw}, got={actual_raw}",
)
benchmarks/suites/mbpp.py
-259
View File
@@ -1,259 +0,0 @@
"""
MBPP (Mostly Basic Python Problems) benchmark suite.
974 crowd-sourced Python programming problems designed to be solvable by
entry-level programmers. Each problem has a task description, code solution
and 3 automated test cases.
Paper: https://arxiv.org/abs/2108.07732
"""
from __future__ import annotations
import json
import os
import re
import sys
import textwrap
from pathlib import Path
from typing import Any
from .base import BenchmarkResult, BenchmarkSuite
# ---------------------------------------------------------------------------
# Built-in mini dataset (15 representative MBPP problems)
# ---------------------------------------------------------------------------
_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
{
"task_id": 1,
"text": "Write a function to find the minimum cost path to reach (m, n) from (0, 0) for the given cost matrix.",
"code": "R = 3\r\nC = 3\r\ndef min_cost(cost, m, n):\r\n\ttc = [[0 for x in range(C)] for x in range(R)]\r\n\ttc[0][0] = cost[0][0]\r\n\tfor i in range(1, m+1):\r\n\t\ttc[i][0] = tc[i-1][0] + cost[i][0]\r\n\tfor j in range(1, n+1):\r\n\t\ttc[0][j] = tc[0][j-1] + cost[0][j]\r\n\tfor i in range(1, m+1):\r\n\t\tfor j in range(1, n+1):\r\n\t\t\ttc[i][j] = min(tc[i-1][j-1], tc[i-1][j], tc[i][j-1]) + cost[i][j]\r\n\treturn tc[m][n]",
"test_list": [
"assert min_cost([[1, 2, 3], [4, 8, 2], [1, 5, 3]], 2, 2) == 8",
"assert min_cost([[2, 3, 4], [5, 9, 3], [2, 6, 4]], 2, 2) == 12",
"assert min_cost([[3, 4, 5], [6, 10, 4], [3, 7, 5]], 2, 2) == 16",
],
},
{
"task_id": 2,
"text": "Write a function to find the similar elements from the given two tuple lists.",
"code": "def similar_elements(test_tup1, test_tup2):\r\n res = tuple(set(test_tup1) & set(test_tup2))\r\n return (res) ",
"test_list": [
"assert similar_elements((3, 4, 5, 6),(5, 7, 4, 10)) == (4, 5)",
"assert similar_elements((1, 2, 3, 4),(5, 4, 3, 7)) == (3, 4)",
"assert similar_elements((11, 12, 14, 13),(17, 15, 14, 13)) == (13, 14)",
],
},
{
"task_id": 3,
"text": "Write a python function to identify non-prime numbers.",
"code": "import math\ndef is_not_prime(n):\n result = False\n for i in range(2,int(math.sqrt(n)) + 1):\n if n % i == 0:\n result = True\n return result",
"test_list": [
"assert is_not_prime(2) == False",
"assert is_not_prime(10) == True",
"assert is_not_prime(35) == True",
],
},
{
"task_id": 4,
"text": "Write a function to find the largest integers from a given list of numbers using heap queue algorithm.",
"code": "import heapq as hq\ndef heap_queue_largest(nums,n):\n largest_nums = hq.nlargest(n, nums)\n return largest_nums",
"test_list": [
"assert heap_queue_largest( [25, 35, 22, 85, 14, 65, 75, 22, 58],3)==[85, 75, 65]",
"assert heap_queue_largest( [25, 35, 22, 85, 14, 65, 75, 22, 58],2)==[85, 75]",
"assert heap_queue_largest( [25, 35, 22, 85, 14, 65, 75, 22, 58],5)==[85, 75, 65, 58, 35]",
],
},
{
"task_id": 5,
"text": "Write a function to find the number of ways to fill it with 2 x 1 dominoes for the given 3 x n board.",
"code": "def count_ways(n):\n\tA = [0] * (n + 1)\n\tB = [0] * (n + 1)\n\tA[0] = 1\n\tA[1] = 0\n\tB[0] = 0\n\tB[1] = 1\n\tfor i in range(2, n+1):\n\t\tA[i] = A[i - 2] + 2 * B[i - 1]\n\t\tB[i] = A[i - 1] + B[i - 2]\n\treturn A[n]",
"test_list": [
"assert count_ways(2) == 3",
"assert count_ways(8) == 153",
"assert count_ways(12) == 2131",
],
},
{
"task_id": 6,
"text": "Write a python function to check whether the two numbers differ at one bit position only or not.",
"code": "def differ_At_One_Bit_Pos(a,b):\n return ((a ^ b) & (a ^ b - 1)) == 0 and a ^ b != 0",
"test_list": [
"assert differ_At_One_Bit_Pos(13,9) == True",
"assert differ_At_One_Bit_Pos(15,8) == False",
"assert differ_At_One_Bit_Pos(2,4) == False",
],
},
{
"task_id": 7,
"text": "Write a function to find all words which are at least 4 characters long in a string.",
"code": 'import re\ndef find_char_long(text):\n return (re.findall(r"\\b\\w{4,}\\b", text))',
"test_list": [
'assert find_char_long(\'Please move back to stream\') == [\'Please\', \'move\', \'back\', \'stream\']',
'assert find_char_long(\'Joveacvber gfnbb vcdf\') == [\'Jove\', \'acvber\', \'gfnbb\', \'vcdf\']',
'assert find_char_long(\'Davvede aridge\') == [\'Davvede\', \'aridge\']',
],
},
{
"task_id": 8,
"text": "Write a function to find squares of individual elements in a list.",
"code": "def square_nums(nums):\n return list(map(lambda x: x ** 2, nums))",
"test_list": [
"assert square_nums([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])==[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]",
"assert square_nums([10,20,30])==[100,400,900]",
"assert square_nums([12,15])==[144,225]",
],
},
{
"task_id": 9,
"text": "Write a python function to find the minimum number of rotations required to get the same string.",
"code": "def find_Rotations(s):\n n = len(s)\n s2 = s + s\n for i in range(1, n + 1):\n if s2[i:i+n] == s:\n return i\n return n",
"test_list": [
'assert find_Rotations("aaaa") == 1',
'assert find_Rotations("ab") == 2',
'assert find_Rotations("abc") == 3',
],
},
{
"task_id": 10,
"text": "Write a function to get the n smallest items from a dataset.",
"code": "import heapq\ndef small_nnum(list1,n):\n smallest=heapq.nsmallest(n,list1)\n return smallest",
"test_list": [
"assert small_nnum([10, 20, 50, 70, 90, 20, 50, 40, 60, 80, 100],2)==[10,20]",
"assert small_nnum([10, 20, 50, 70, 90, 20, 50, 40, 60, 80, 100],5)==[10,20,20,40,50]",
"assert small_nnum([10, 20, 50, 70, 90, 20, 50, 40, 60, 80, 100],3)==[10,20,20]",
],
},
{
"task_id": 11,
"text": "Write a python function to remove first and last occurrence of a given character from the string.",
"code": "def remove_Occ(s,ch):\n # Remove first occurrence\n for i in range(len(s)):\n if (s[i] == ch):\n s = s[0 : i] + s[i + 1:]\n break\n # Remove last occurrence\n for i in range(len(s) - 1,-1,-1):\n if (s[i] == ch):\n s = s[0 : i] + s[i + 1:]\n break\n return s",
"test_list": [
'assert remove_Occ("hello","l") == "heo"',
'assert remove_Occ("abcda","a") == "bcd"',
'assert remove_Occ("PHP","P") == "H"',
],
},
{
"task_id": 12,
"text": "Write a function to sort a given matrix in ascending order according to the sum of its rows.",
"code": "def sort_matrix(M):\n result = sorted(M, key=sum)\n return result",
"test_list": [
"assert sort_matrix([[1, 2, 3], [2, 4, 5], [1, 1, 1]])==[[1, 1, 1], [1, 2, 3], [2, 4, 5]]",
"assert sort_matrix([[1, 2, 3], [-2, 4, -5], [1, -1, 1]])==[[-2, 4, -5], [1, -1, 1], [1, 2, 3]]",
"assert sort_matrix([[5,8,9],[6,4,3],[2,1,4]])==[[2, 1, 4], [6, 4, 3], [5, 8, 9]]",
],
},
{
"task_id": 13,
"text": "Write a function to count the most common words in a dictionary.",
"code": "from collections import Counter\ndef count_common(words):\n word_counts = Counter(words)\n top_four = word_counts.most_common(4)\n return top_four",
"test_list": [
"assert count_common(['red','green','black','pink','black','white','black','eyes','white','black','orange','pink','pink','red','red','white','orange','white','black','pink','green','green','pink','green','pink','white','orange','orange','red']) == [('pink', 6), ('black', 5), ('white', 5), ('red', 4)]",
"assert count_common(['one', 'two', 'three', 'four', 'five', 'one', 'two', 'one', 'three', 'one']) == [('one', 4), ('two', 2), ('three', 2), ('four', 1)]",
"assert count_common(['Facebook', 'Apple', 'Amazon', 'Netflix', 'Google', 'Apple', 'Netflix', 'Amazon']) == [('Apple', 2), ('Amazon', 2), ('Netflix', 2), ('Facebook', 1)]",
],
},
{
"task_id": 14,
"text": "Write a python function to find the volume of a triangular prism.",
"code": "def find_Volume(l,b,h):\n return (l * b * h) / 2",
"test_list": [
"assert find_Volume(10,8,6) == 240",
"assert find_Volume(3,2,2) == 6",
"assert find_Volume(1,2,1) == 1",
],
},
{
"task_id": 15,
"text": "Write a function to split a string at uppercase letters.",
"code": "import re\ndef split_upperstring(text):\n return re.findall('[A-Z][^A-Z]*', text)",
"test_list": [
'assert split_upperstring("PythonProgramLanguage")==[\'Python\',\'Program\',\'Language\']',
'assert split_upperstring("PythonProgram")==[\'Python\',\'Program\']',
'assert split_upperstring("ProgramLanguage")==[\'Program\',\'Language\']',
],
},
]
class MBPPBenchmark(BenchmarkSuite):
"""MBPP: Mostly Basic Python Problems (974 problems)."""
name = "MBPP"
description = "Crowd-sourced basic Python programming problems"
category = "coding"
def load_dataset(self) -> list[dict[str, Any]]:
jsonl_path = Path(self.data_dir) / "mbpp.jsonl"
if jsonl_path.exists():
problems: list[dict[str, Any]] = []
with open(jsonl_path) as fh:
for line in fh:
line = line.strip()
if line:
problems.append(json.loads(line))
if self.verbose:
print(f" Loaded {len(problems)} problems from {jsonl_path}")
return problems
if self.verbose:
print(f" {jsonl_path} not found — using built-in 15-problem subset")
return list(_BUILTIN_PROBLEMS)
def build_prompt(self, problem: dict[str, Any]) -> str:
desc = problem["text"]
tests = "\n".join(problem.get("test_list", []))
return (
f"{desc}\n\n"
f"Write the Python code and save it to solution.py.\n"
f"The following test assertions must pass:\n```python\n{tests}\n```"
)
def setup_workspace(self, problem: dict[str, Any], workspace: str) -> None:
tests = problem.get("test_list", [])
test_code = "\n".join(tests)
harness = (
"import sys\n"
'sys.path.insert(0, ".")\n'
"from solution import *\n\n"
f"{test_code}\n\n"
'print("ALL_TESTS_PASSED")\n'
)
with open(os.path.join(workspace, "test_harness.py"), "w") as fh:
fh.write(harness)
def recover_output_files(
self,
problem: dict[str, Any],
workspace: str,
agent_output: str,
metadata: dict[str, Any],
) -> None:
del problem
solution_path = Path(workspace) / "solution.py"
if solution_path.exists():
return
blocks = re.findall(r"```(?:python)?\s*(.*?)```", agent_output, flags=re.DOTALL | re.IGNORECASE)
for block in blocks:
candidate = block.strip()
if "def " in candidate or "import " in candidate or "class " in candidate:
solution_path.write_text(candidate.rstrip() + "\n", encoding="utf-8")
metadata["recovered_solution_from_output"] = True
return
def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
pid = str(problem.get("task_id", "unknown"))
sol = os.path.join(workspace, "solution.py")
if not os.path.exists(sol):
return BenchmarkResult(problem_id=pid, passed=False, error="solution.py not found")
code, output = self._run_shell(
f"{sys.executable} test_harness.py", cwd=workspace, timeout=30.0
)
passed = code == 0 and "ALL_TESTS_PASSED" in output
return BenchmarkResult(
problem_id=pid, passed=passed, actual=output[:500],
error="" if passed else output[:500],
)
benchmarks/suites/mmlu_pro.py
-176
View File
@@ -1,176 +0,0 @@
"""
MMLU-Pro benchmark suite.
MMLU-Pro (Massive Multitask Language Understanding - Professional) is an
enhanced version of MMLU with harder questions, 10 answer choices (A–J),
and chain-of-thought reasoning requirements across 14 subjects.
Reference: https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro
"""
from __future__ import annotations
import json
import os
import re
from pathlib import Path
from typing import Any
from .base import BenchmarkResult, BenchmarkSuite
# ---------------------------------------------------------------------------
# Built-in mini dataset (10 representative MMLU-Pro problems)
# ---------------------------------------------------------------------------
_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
{
"id": "mmlu-pro-001",
"subject": "computer_science",
"question": "Which of the following best describes the time complexity of binary search on a sorted array of n elements?",
"choices": ["O(1)", "O(log n)", "O(n)", "O(n log n)", "O(n^2)", "O(2^n)", "O(n!)", "O(sqrt(n))", "O(n^3)", "O(log log n)"],
"answer": "B",
},
{
"id": "mmlu-pro-002",
"subject": "physics",
"question": "A 2 kg block is placed on a frictionless inclined plane that makes a 30° angle with the horizontal. What is the acceleration of the block down the incline? (g = 9.8 m/s²)",
"choices": ["2.45 m/s²", "4.9 m/s²", "6.93 m/s²", "9.8 m/s²", "3.27 m/s²", "8.49 m/s²", "1.63 m/s²", "5.66 m/s²", "7.35 m/s²", "0.98 m/s²"],
"answer": "B",
},
{
"id": "mmlu-pro-003",
"subject": "chemistry",
"question": "What is the hybridization of the central atom in SF6?",
"choices": ["sp", "sp2", "sp3", "sp3d", "sp3d2", "dsp3", "d2sp3", "sp2d", "sp3d3", "none of the above"],
"answer": "E",
},
{
"id": "mmlu-pro-004",
"subject": "mathematics",
"question": "What is the derivative of f(x) = x³ · ln(x)?",
"choices": ["3x² · ln(x)", "x² + 3x² · ln(x)", "3x² · ln(x) + x²", "x³/x + 3x²", "3x² · ln(x) + x³", "x² · (3ln(x) + 1)", "3x · ln(x) + x²", "x³ · (1/x) + 3x² · ln(x)", "3x² / ln(x)", "ln(x) + 3x²"],
"answer": "F",
},
{
"id": "mmlu-pro-005",
"subject": "biology",
"question": "Which of the following is NOT a characteristic of the adaptive immune response?",
"choices": ["Specificity", "Memory", "Rapid initial response", "Diversity", "Self/non-self discrimination", "Clonal expansion", "Tolerance", "MHC restriction", "Somatic hypermutation", "Pattern recognition"],
"answer": "C",
},
{
"id": "mmlu-pro-006",
"subject": "history",
"question": "The Treaty of Westphalia (1648) is most significant because it:",
"choices": ["Ended the Hundred Years War", "Established the principle of state sovereignty", "Created the United Nations", "Divided Africa among European powers", "Ended World War I", "Established the Holy Roman Empire", "Created NATO", "Ended the Napoleonic Wars", "Established the European Union", "Ended the Crusades"],
"answer": "B",
},
{
"id": "mmlu-pro-007",
"subject": "economics",
"question": "In a perfectly competitive market in long-run equilibrium, which of the following is true?",
"choices": ["Price equals minimum ATC", "Firms earn positive economic profit", "Price exceeds marginal cost", "Firms produce at maximum ATC", "Barriers to entry exist", "Price equals maximum AVC", "Firms are price makers", "Marginal cost exceeds price", "Economic profit is negative", "Supply is perfectly inelastic"],
"answer": "A",
},
{
"id": "mmlu-pro-008",
"subject": "philosophy",
"question": "According to Kant's categorical imperative, an action is morally permissible if and only if:",
"choices": ["It maximizes overall happiness", "It can be universalized without contradiction", "It promotes the greatest good for the greatest number", "God commands it", "It follows natural law", "It leads to virtuous character", "It is agreed upon by rational contractors", "It satisfies one's desires", "It is consistent with social norms", "It minimizes suffering"],
"answer": "B",
},
{
"id": "mmlu-pro-009",
"subject": "law",
"question": "Under the U.S. Constitution, which amendment protects against unreasonable searches and seizures?",
"choices": ["First Amendment", "Second Amendment", "Third Amendment", "Fourth Amendment", "Fifth Amendment", "Sixth Amendment", "Seventh Amendment", "Eighth Amendment", "Ninth Amendment", "Tenth Amendment"],
"answer": "D",
},
{
"id": "mmlu-pro-010",
"subject": "psychology",
"question": "In Piaget's theory of cognitive development, the stage in which children develop the ability to think abstractly and reason hypothetically is called the:",
"choices": ["Sensorimotor stage", "Preoperational stage", "Concrete operational stage", "Formal operational stage", "Postformal stage", "Latency stage", "Genital stage", "Identity vs. role confusion", "Autonomy vs. shame", "Initiative vs. guilt"],
"answer": "D",
},
]
class MMLUProBenchmark(BenchmarkSuite):
"""MMLU-Pro: Enhanced multitask language understanding with 10-choice questions."""
name = "MMLU-Pro"
description = "Professional-level multiple-choice QA across 14 subjects (10 choices)"
category = "knowledge"
def load_dataset(self) -> list[dict[str, Any]]:
jsonl_path = Path(self.data_dir) / "mmlu_pro.jsonl"
if jsonl_path.exists():
problems: list[dict[str, Any]] = []
with open(jsonl_path) as fh:
for line in fh:
line = line.strip()
if line:
problems.append(json.loads(line))
if self.verbose:
print(f" Loaded {len(problems)} problems from {jsonl_path}")
return problems
if self.verbose:
print(f" {jsonl_path} not found — using built-in 10-problem subset")
return list(_BUILTIN_PROBLEMS)
def build_prompt(self, problem: dict[str, Any]) -> str:
question = problem["question"]
choices = problem.get("choices", [])
letters = "ABCDEFGHIJ"
choices_text = "\n".join(
f" {letters[i]}. {c}" for i, c in enumerate(choices)
)
return (
f"Answer the following multiple-choice question.\n\n"
f"Question: {question}\n\n"
f"Choices:\n{choices_text}\n\n"
f"Write ONLY the letter of the correct answer (A–J) to a file called answer.txt — "
f"no explanation, just the single letter."
)
def recover_output_files(
self,
problem: dict[str, Any],
workspace: str,
agent_output: str,
metadata: dict[str, Any],
) -> None:
del problem
answer_path = Path(workspace) / "answer.txt"
if answer_path.exists():
return
match = re.search(r"\b([A-J])\b", agent_output)
if match:
answer_path.write_text(match.group(1) + "\n", encoding="utf-8")
metadata["recovered_answer_from_output"] = True
def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
pid = problem.get("id", "unknown")
expected = str(problem["answer"]).strip().upper()
answer_file = os.path.join(workspace, "answer.txt")
if not os.path.exists(answer_file):
return BenchmarkResult(
problem_id=pid, passed=False, expected=expected,
error="answer.txt not found",
)
with open(answer_file) as fh:
actual_raw = fh.read().strip()
match = re.search(r"\b([A-J])\b", actual_raw.upper())
actual = match.group(1) if match else actual_raw.strip().upper()
passed = actual == expected
return BenchmarkResult(
problem_id=pid, passed=passed,
expected=expected, actual=actual_raw,
error="" if passed else f"expected={expected}, got={actual_raw}",
)
benchmarks/suites/mmmlu.py
-191
View File
@@ -1,191 +0,0 @@
"""
MMMLU (Multilingual Massive Multitask Language Understanding) benchmark suite.
MMMLU extends MMLU to multiple languages, testing language understanding
across diverse cultural and linguistic contexts.
Reference: https://huggingface.co/datasets/openai/MMMLU
"""
from __future__ import annotations
import json
import os
import re
from pathlib import Path
from typing import Any
from .base import BenchmarkResult, BenchmarkSuite
# ---------------------------------------------------------------------------
# Built-in mini dataset (10 representative MMMLU problems — multilingual)
# ---------------------------------------------------------------------------
_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
{
"id": "mmmlu-001",
"language": "en",
"subject": "abstract_algebra",
"question": "Find the degree of the extension Q(sqrt(2), sqrt(3), sqrt(18)) over Q.",
"choices": ["0", "4", "2", "6"],
"answer": "B",
},
{
"id": "mmmlu-002",
"language": "es",
"subject": "anatomy",
"question": "¿Cuál de los siguientes músculos es el principal responsable de la abducción del brazo?",
"choices": ["Deltoides", "Trapecio", "Pectoral mayor", "Dorsal ancho"],
"answer": "A",
},
{
"id": "mmmlu-003",
"language": "fr",
"subject": "astronomy",
"question": "Quelle est la planète la plus proche du Soleil?",
"choices": ["Vénus", "Terre", "Mercure", "Mars"],
"answer": "C",
},
{
"id": "mmmlu-004",
"language": "de",
"subject": "business_ethics",
"question": "Welcher der folgenden Begriffe beschreibt die Verantwortung eines Unternehmens gegenüber der Gesellschaft am besten?",
"choices": ["Corporate Social Responsibility", "Shareholder Theory", "Laissez-faire", "Monopol"],
"answer": "A",
},
{
"id": "mmmlu-005",
"language": "zh",
"subject": "computer_science",
"question": "在计算机科学中，二叉搜索树的平均时间复杂度是多少？",
"choices": ["O(n)", "O(log n)", "O(n²)", "O(1)"],
"answer": "B",
},
{
"id": "mmmlu-006",
"language": "ja",
"subject": "world_history",
"question": "フランス革命が始まったのは何年ですか？",
"choices": ["1776年", "1789年", "1804年", "1815年"],
"answer": "B",
},
{
"id": "mmmlu-007",
"language": "pt",
"subject": "geography",
"question": "Qual é o rio mais longo do mundo?",
"choices": ["Amazonas", "Nilo", "Mississipi", "Yangtzé"],
"answer": "B",
},
{
"id": "mmmlu-008",
"language": "it",
"subject": "philosophy",
"question": "Chi ha scritto 'La Repubblica'?",
"choices": ["Aristotele", "Platone", "Socrate", "Epicuro"],
"answer": "B",
},
{
"id": "mmmlu-009",
"language": "ko",
"subject": "physics",
"question": "뉴턴의 제2법칙에서 힘의 공식은?",
"choices": ["F = mv", "F = ma", "F = mg", "F = mc²"],
"answer": "B",
},
{
"id": "mmmlu-010",
"language": "ar",
"subject": "chemistry",
"question": "ما هو العنصر الأكثر وفرة في القشرة الأرضية؟",
"choices": ["الحديد", "الأكسجين", "السيليكون", "الألمنيوم"],
"answer": "B",
},
]
class MMMMLUBenchmark(BenchmarkSuite):
"""MMMLU: Multilingual MMLU across diverse languages."""
name = "MMMLU"
description = "Multilingual multiple-choice QA across languages and subjects"
category = "knowledge"
def load_dataset(self) -> list[dict[str, Any]]:
jsonl_path = Path(self.data_dir) / "mmmlu.jsonl"
if jsonl_path.exists():
problems: list[dict[str, Any]] = []
with open(jsonl_path) as fh:
for line in fh:
line = line.strip()
if line:
problems.append(json.loads(line))
if self.verbose:
print(f" Loaded {len(problems)} problems from {jsonl_path}")
return problems
if self.verbose:
print(f" {jsonl_path} not found — using built-in 10-problem subset")
return list(_BUILTIN_PROBLEMS)
def build_prompt(self, problem: dict[str, Any]) -> str:
question = problem["question"]
choices = problem.get("choices", [])
language = problem.get("language", "en")
letters = "ABCD"
choices_text = "\n".join(
f" {letters[i]}. {c}" for i, c in enumerate(choices)
)
lang_instruction = ""
if language != "en":
lang_instruction = f"The question is in {language}. "
return (
f"Answer the following multiple-choice question. {lang_instruction}\n\n"
f"Question: {question}\n\n"
f"Choices:\n{choices_text}\n\n"
f"Write ONLY the letter of the correct answer (A–D) to a file called answer.txt — "
f"no explanation, just the single letter."
)
def recover_output_files(
self,
problem: dict[str, Any],
workspace: str,
agent_output: str,
metadata: dict[str, Any],
) -> None:
del problem
answer_path = Path(workspace) / "answer.txt"
if answer_path.exists():
return
match = re.search(r"\b([A-D])\b", agent_output)
if match:
answer_path.write_text(match.group(1) + "\n", encoding="utf-8")
metadata["recovered_answer_from_output"] = True
def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
pid = problem.get("id", "unknown")
expected = str(problem["answer"]).strip().upper()
answer_file = os.path.join(workspace, "answer.txt")
if not os.path.exists(answer_file):
return BenchmarkResult(
problem_id=pid, passed=False, expected=expected,
error="answer.txt not found",
)
with open(answer_file) as fh:
actual_raw = fh.read().strip()
match = re.search(r"\b([A-D])\b", actual_raw.upper())
actual = match.group(1) if match else actual_raw.strip().upper()
passed = actual == expected
return BenchmarkResult(
problem_id=pid, passed=passed,
expected=expected, actual=actual_raw,
error="" if passed else f"expected={expected}, got={actual_raw}",
)
benchmarks/suites/swe_bench.py
-211
View File
@@ -1,211 +0,0 @@
"""
SWE-Bench benchmark suite.
SWE-Bench tests whether an agent can resolve real GitHub issues.
Each problem provides a repository, an issue description, and a patch
that should make the failing tests pass.
Paper: https://arxiv.org/abs/2310.06770
Dataset: https://huggingface.co/datasets/princeton-nlp/SWE-bench_Lite
"""
from __future__ import annotations
import json
import os
import sys
import textwrap
from pathlib import Path
from typing import Any
from .base import BenchmarkResult, BenchmarkSuite
# ---------------------------------------------------------------------------
# Built-in mini dataset — simplified SWE-Bench-style problems
# These are self-contained (no actual repo cloning needed).
# ---------------------------------------------------------------------------
_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
{
"instance_id": "swe-mini-001",
"problem_statement": (
"The `StringFormatter.format_title` method should capitalize the first letter "
"of each word in the input string. Currently it lowercases everything instead.\n\n"
"Expected: format_title('hello world') => 'Hello World'\n"
"Actual: format_title('hello world') => 'hello world'"
),
"setup_code": textwrap.dedent("""\
cat > formatter.py << 'PYEOF'
class StringFormatter:
def format_title(self, text):
return text.lower()
PYEOF
cat > test_formatter.py << 'PYEOF'
from formatter import StringFormatter
f = StringFormatter()
assert f.format_title('hello world') == 'Hello World'
assert f.format_title('PYTHON IS FUN') == 'Python Is Fun'
assert f.format_title('already Good') == 'Already Good'
print("ALL_TESTS_PASSED")
PYEOF
"""),
"test_cmd": "python3 test_formatter.py",
},
{
"instance_id": "swe-mini-002",
"problem_statement": (
"The `DataValidator.validate_email` function returns True for emails "
"that don't have a domain extension (e.g., 'user@domain' without .com). "
"It should require at least one dot after the @ sign.\n\n"
"Expected: validate_email('user@domain') => False\n"
"Actual: validate_email('user@domain') => True"
),
"setup_code": textwrap.dedent("""\
cat > validator.py << 'PYEOF'
class DataValidator:
def validate_email(self, email):
return '@' in email
PYEOF
cat > test_validator.py << 'PYEOF'
from validator import DataValidator
v = DataValidator()
assert v.validate_email('user@example.com') == True
assert v.validate_email('bad') == False
assert v.validate_email('user@domain') == False
assert v.validate_email('a@b.c') == True
print("ALL_TESTS_PASSED")
PYEOF
"""),
"test_cmd": "python3 test_validator.py",
},
{
"instance_id": "swe-mini-003",
"problem_statement": (
"The `FileProcessor.count_lines` method crashes with an unhandled "
"exception when the file doesn't exist. It should return -1 for missing files.\n\n"
"Expected: count_lines('nonexistent.txt') => -1\n"
"Actual: FileNotFoundError is raised"
),
"setup_code": textwrap.dedent("""\
cat > processor.py << 'PYEOF'
class FileProcessor:
def count_lines(self, filepath):
with open(filepath) as f:
return len(f.readlines())
PYEOF
echo -e "line1\\nline2\\nline3" > sample.txt
cat > test_processor.py << 'PYEOF'
from processor import FileProcessor
p = FileProcessor()
assert p.count_lines('sample.txt') == 3
assert p.count_lines('nonexistent.txt') == -1
print("ALL_TESTS_PASSED")
PYEOF
"""),
"test_cmd": "python3 test_processor.py",
},
{
"instance_id": "swe-mini-004",
"problem_statement": (
"The `MathHelper.factorial` function gives wrong results for n=0. "
"factorial(0) should return 1, but it currently returns 0.\n\n"
"Expected: factorial(0) => 1\n"
"Actual: factorial(0) => 0"
),
"setup_code": textwrap.dedent("""\
cat > mathhelper.py << 'PYEOF'
class MathHelper:
def factorial(self, n):
result = 0
for i in range(1, n + 1):
result *= i
return result
PYEOF
cat > test_math.py << 'PYEOF'
from mathhelper import MathHelper
m = MathHelper()
assert m.factorial(0) == 1
assert m.factorial(1) == 1
assert m.factorial(5) == 120
assert m.factorial(10) == 3628800
print("ALL_TESTS_PASSED")
PYEOF
"""),
"test_cmd": "python3 test_math.py",
},
{
"instance_id": "swe-mini-005",
"problem_statement": (
"The `ListUtils.unique` method should return unique elements in the "
"order they first appear, but it currently sorts them alphabetically.\n\n"
"Expected: unique(['b', 'a', 'b', 'c', 'a']) => ['b', 'a', 'c']\n"
"Actual: unique(['b', 'a', 'b', 'c', 'a']) => ['a', 'b', 'c']"
),
"setup_code": textwrap.dedent("""\
cat > listutils.py << 'PYEOF'
class ListUtils:
def unique(self, items):
return sorted(set(items))
PYEOF
cat > test_list.py << 'PYEOF'
from listutils import ListUtils
u = ListUtils()
assert u.unique(['b', 'a', 'b', 'c', 'a']) == ['b', 'a', 'c']
assert u.unique([3, 1, 2, 1, 3]) == [3, 1, 2]
assert u.unique([]) == []
assert u.unique([1]) == [1]
print("ALL_TESTS_PASSED")
PYEOF
"""),
"test_cmd": "python3 test_list.py",
},
]
class SWEBenchBenchmark(BenchmarkSuite):
"""SWE-Bench: resolve real-world GitHub issues."""
name = "SWE-Bench"
description = "Resolve GitHub issues by editing source code"
category = "coding"
def load_dataset(self) -> list[dict[str, Any]]:
jsonl_path = Path(self.data_dir) / "swe_bench.jsonl"
if jsonl_path.exists():
problems: list[dict[str, Any]] = []
with open(jsonl_path) as fh:
for line in fh:
line = line.strip()
if line:
problems.append(json.loads(line))
if self.verbose:
print(f" Loaded {len(problems)} problems from {jsonl_path}")
return problems
if self.verbose:
print(f" {jsonl_path} not found — using built-in 5-problem subset")
return list(_BUILTIN_PROBLEMS)
def build_prompt(self, problem: dict[str, Any]) -> str:
stmt = problem["problem_statement"]
return (
f"You are a software engineer fixing a bug. Here is the issue:\n\n"
f"{stmt}\n\n"
f"Read the code in the workspace, find the bug, and fix it. "
f"Do not modify the test files."
)
def setup_workspace(self, problem: dict[str, Any], workspace: str) -> None:
setup = problem.get("setup_code", "")
if setup:
self._run_shell(setup, cwd=workspace, timeout=30.0)
def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
pid = problem.get("instance_id", "unknown")
test_cmd = problem.get("test_cmd", "echo FAIL && exit 1")
code, output = self._run_shell(test_cmd, cwd=workspace, timeout=30.0)
passed = code == 0 and "ALL_TESTS_PASSED" in output
return BenchmarkResult(
problem_id=pid, passed=passed, actual=output[:500],
error="" if passed else output[:500],
)
benchmarks/suites/tau2.py
-181
View File
@@ -1,181 +0,0 @@
"""
Tau2 benchmark suite.
Tau2 (TAU-bench v2) evaluates language models on tool-augmented tasks
requiring multi-step reasoning with tool use. It measures the ability
to plan, select, and chain tool calls to complete complex tasks.
The benchmark reports an average score across 3 task domains:
retail, airline, and finance.
Reference: https://github.com/sierra-research/tau-bench
"""
from __future__ import annotations
import json
import os
import re
from pathlib import Path
from typing import Any
from .base import BenchmarkResult, BenchmarkSuite
# ---------------------------------------------------------------------------
# Built-in mini dataset (10 representative Tau2-style problems)
# ---------------------------------------------------------------------------
_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
{
"id": "tau2-001",
"domain": "retail",
"question": "A customer wants to return a laptop purchased 20 days ago. The store policy allows returns within 30 days with receipt. The customer has the receipt. The laptop is in original packaging. What is the correct action?",
"choices": ["Process full refund", "Deny return - past return window", "Offer store credit only", "Process exchange only"],
"answer": "A",
},
{
"id": "tau2-002",
"domain": "airline",
"question": "A passenger has a connecting flight with a 45-minute layover. Their first flight is delayed by 30 minutes. The minimum connection time at the hub airport is 60 minutes. What should the agent recommend?",
"choices": ["Rebook on next available flight", "Keep current booking, passenger will make it", "Cancel entire itinerary", "Upgrade to first class on delayed flight"],
"answer": "A",
},
{
"id": "tau2-003",
"domain": "finance",
"question": "A client wants to transfer $50,000 between accounts. The daily transfer limit is $25,000. The client has proper identification. What is the correct procedure?",
"choices": ["Split into two $25,000 transfers over two days", "Override limit and process single transfer", "Deny the transfer entirely", "Process as a wire transfer instead"],
"answer": "A",
},
{
"id": "tau2-004",
"domain": "retail",
"question": "An item shows a price of $29.99 on the shelf but rings up as $34.99 at checkout. Store policy states the customer gets the lower price if there's a discrepancy. The shelf tag is verified to be for this item. What action should be taken?",
"choices": ["Honor the shelf price of $29.99", "Charge the register price of $34.99", "Split the difference", "Give the item for free"],
"answer": "A",
},
{
"id": "tau2-005",
"domain": "airline",
"question": "A passenger with a non-refundable ticket wants to change their travel date. The fare difference is +$150 and there is a $75 change fee. The passenger is a Gold status member (change fees waived). What is the total additional cost?",
"choices": ["$150", "$225", "$75", "$0"],
"answer": "A",
},
{
"id": "tau2-006",
"domain": "finance",
"question": "A customer's account shows 3 pending transactions totaling $500, but their available balance is $400. One pending transaction of $200 is a pre-authorization from a gas station from 5 days ago. What should the agent do?",
"choices": ["Release the gas station pre-authorization as it exceeds the standard hold period", "Decline all pending transactions", "Suggest the customer deposit more funds", "Freeze the account for suspicious activity"],
"answer": "A",
},
{
"id": "tau2-007",
"domain": "retail",
"question": "A customer purchased an appliance with a 2-year manufacturer warranty 18 months ago. The appliance has stopped working due to a manufacturing defect. The customer does not have an extended warranty. What should the agent do?",
"choices": ["Process warranty claim with manufacturer", "Offer store repair at customer's cost", "Deny any assistance", "Offer replacement at discounted price"],
"answer": "A",
},
{
"id": "tau2-008",
"domain": "airline",
"question": "A flight is overbooked by 2 passengers. 3 passengers volunteer to give up their seats. Compensation offered is $400 voucher + next flight (2 hours later). Airline policy requires bumping in reverse check-in order if volunteers insufficient. What should happen?",
"choices": ["Accept 2 of the 3 volunteers, compensate them", "Accept all 3 volunteers", "Involuntarily deny 2 passengers boarding", "Cancel the flight"],
"answer": "A",
},
{
"id": "tau2-009",
"domain": "finance",
"question": "A customer reports their credit card stolen. There are 3 unauthorized transactions in the last 24 hours totaling $1,200. Federal regulation limits customer liability for unauthorized transactions reported within 2 business days. What steps should be taken first?",
"choices": ["Block the card immediately and initiate fraud investigation", "Wait for the monthly statement to dispute charges", "Transfer balance to new card without blocking old one", "Ask customer to contact merchants directly"],
"answer": "A",
},
{
"id": "tau2-010",
"domain": "retail",
"question": "A customer wants to price match an identical item found cheaper at a competitor. Store policy allows price matching for identical items at local competitors within 14 days of purchase. The customer purchased 10 days ago and has proof of the competitor's current price. The competitor is an online-only store. Store policy explicitly excludes online-only retailers. What should the agent do?",
"choices": ["Deny price match - online-only retailers excluded", "Honor the price match anyway", "Offer partial price adjustment", "Contact competitor to verify price"],
"answer": "A",
},
]
class Tau2Benchmark(BenchmarkSuite):
"""Tau2: Tool-augmented task evaluation across retail, airline, and finance."""
name = "Tau2"
description = "Tool-augmented multi-step reasoning (retail/airline/finance)"
category = "reasoning"
def load_dataset(self) -> list[dict[str, Any]]:
jsonl_path = Path(self.data_dir) / "tau2.jsonl"
if jsonl_path.exists():
problems: list[dict[str, Any]] = []
with open(jsonl_path) as fh:
for line in fh:
line = line.strip()
if line:
problems.append(json.loads(line))
if self.verbose:
print(f" Loaded {len(problems)} problems from {jsonl_path}")
return problems
if self.verbose:
print(f" {jsonl_path} not found — using built-in 10-problem subset")
return list(_BUILTIN_PROBLEMS)
def build_prompt(self, problem: dict[str, Any]) -> str:
question = problem["question"]
choices = problem.get("choices", [])
domain = problem.get("domain", "general")
letters = "ABCD"
choices_text = "\n".join(
f" {letters[i]}. {c}" for i, c in enumerate(choices)
)
return (
f"You are a {domain} service agent. Answer the following question "
f"about the correct action to take.\n\n"
f"Scenario: {question}\n\n"
f"Options:\n{choices_text}\n\n"
f"Write ONLY the letter of the correct answer (A–D) to a file called answer.txt — "
f"no explanation, just the single letter."
)
def recover_output_files(
self,
problem: dict[str, Any],
workspace: str,
agent_output: str,
metadata: dict[str, Any],
) -> None:
del problem
answer_path = Path(workspace) / "answer.txt"
if answer_path.exists():
return
match = re.search(r"\b([A-D])\b", agent_output)
if match:
answer_path.write_text(match.group(1) + "\n", encoding="utf-8")
metadata["recovered_answer_from_output"] = True
def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
pid = problem.get("id", "unknown")
expected = str(problem["answer"]).strip().upper()
answer_file = os.path.join(workspace, "answer.txt")
if not os.path.exists(answer_file):
return BenchmarkResult(
problem_id=pid, passed=False, expected=expected,
error="answer.txt not found",
)
with open(answer_file) as fh:
actual_raw = fh.read().strip()
match = re.search(r"\b([A-D])\b", actual_raw.upper())
actual = match.group(1) if match else actual_raw.strip().upper()
passed = actual == expected
return BenchmarkResult(
problem_id=pid, passed=passed,
expected=expected, actual=actual_raw,
error="" if passed else f"expected={expected}, got={actual_raw}",
)
benchmarks/tasks/__init__.py
View File
benchmarks/tasks/definitions.py
-654
View File
@@ -1,654 +0,0 @@
"""
Benchmark task definitions for claw-code-agent.
Each task has:
- id: unique identifier
- category: what skill is being tested
- difficulty: easy / medium / hard
- instruction: what the agent is told to do
- setup: shell commands to prepare the workspace (run BEFORE the agent)
- verify: shell commands that return exit 0 on success (run AFTER the agent)
"""
from __future__ import annotations
from dataclasses import dataclass
@dataclass(frozen=True)
class BenchmarkTask:
id: str
category: str
difficulty: str
instruction: str
setup: str
verify: str
TASKS: tuple[BenchmarkTask, ...] = (
# ------------------------------------------------------------------
# 1. FILE CREATION
# ------------------------------------------------------------------
BenchmarkTask(
id="file-create-basic",
category="file-ops",
difficulty="easy",
instruction="Create a file called hello.txt containing exactly the text: Hello, World!",
setup="",
verify='[ -f hello.txt ] && grep -qx "Hello, World!" hello.txt',
),
BenchmarkTask(
id="file-create-nested",
category="file-ops",
difficulty="easy",
instruction="Create the directory structure src/utils/ and inside it create a file called helpers.py containing a Python function called greet that takes a name parameter and returns the string 'Hello, <name>!'.",
setup="",
verify=(
'[ -f src/utils/helpers.py ] && '
'python3 -c "from src.utils.helpers import greet; assert greet(\'World\') == \'Hello, World!\'"'
),
),
# ------------------------------------------------------------------
# 2. FILE EDITING
# ------------------------------------------------------------------
BenchmarkTask(
id="edit-replace-string",
category="file-edit",
difficulty="easy",
instruction="In the file config.txt, replace every occurrence of 'localhost' with '0.0.0.0'.",
setup=(
'echo "host=localhost\nport=8080\ndb_host=localhost\nbackup=localhost:3000" > config.txt'
),
verify=(
'! grep -q "localhost" config.txt && '
'grep -q "0.0.0.0" config.txt && '
'[ "$(grep -c "0.0.0.0" config.txt)" = "3" ]'
),
),
BenchmarkTask(
id="edit-add-function",
category="file-edit",
difficulty="medium",
instruction="The file math_utils.py has an add function. Add a new function called multiply that takes two arguments a and b and returns a * b. Do not change the existing add function.",
setup=(
'cat > math_utils.py << \'PYEOF\'\n'
'def add(a, b):\n'
' return a + b\n'
'PYEOF'
),
verify=(
'python3 -c "'
'from math_utils import add, multiply; '
'assert add(2, 3) == 5; '
'assert multiply(4, 5) == 20; '
'assert multiply(0, 10) == 0; '
'assert multiply(-2, 3) == -6'
'"'
),
),
# ------------------------------------------------------------------
# 3. BUG FIXING
# ------------------------------------------------------------------
BenchmarkTask(
id="bugfix-off-by-one",
category="bugfix",
difficulty="medium",
instruction="The file fibonacci.py has a function that should return the first n Fibonacci numbers as a list. For example, fibonacci(5) should return [0, 1, 1, 2, 3]. But it has a bug. Find and fix it.",
setup=(
'cat > fibonacci.py << \'PYEOF\'\n'
'def fibonacci(n):\n'
' if n <= 0:\n'
' return []\n'
' if n == 1:\n'
' return [0]\n'
' fibs = [0, 1]\n'
' for i in range(2, n + 1):\n'
' fibs.append(fibs[i-1] + fibs[i-2])\n'
' return fibs\n'
'PYEOF'
),
verify=(
'python3 -c "'
'from fibonacci import fibonacci; '
'assert fibonacci(0) == []; '
'assert fibonacci(1) == [0]; '
'assert fibonacci(2) == [0, 1]; '
'assert fibonacci(5) == [0, 1, 1, 2, 3]; '
'assert fibonacci(8) == [0, 1, 1, 2, 3, 5, 8, 13]'
'"'
),
),
BenchmarkTask(
id="bugfix-syntax-error",
category="bugfix",
difficulty="easy",
instruction="The file broken.py has syntax errors that prevent it from running. Fix all syntax errors so that running 'python3 broken.py' prints 'All tests passed'.",
setup=(
'cat > broken.py << \'PYEOF\'\n'
'def calculate(x, y)\n'
' result = x + y\n'
' return result\n'
'\n'
'def main():\n'
' total = calculate(10, 20\n'
' if total == 30:\n'
' print("All tests passed")\n'
' else\n'
' print("Failed")\n'
'\n'
'if __name__ == "__main__":\n'
' main()\n'
'PYEOF'
),
verify='python3 broken.py 2>&1 | grep -qx "All tests passed"',
),
BenchmarkTask(
id="bugfix-logic-error",
category="bugfix",
difficulty="medium",
instruction="The file sorter.py has a function called bubble_sort that should sort a list in ascending order, but it produces wrong results. Find the bug and fix it. Do not replace the algorithm with a different one — fix the existing bubble sort logic.",
setup=(
'cat > sorter.py << \'PYEOF\'\n'
'def bubble_sort(arr):\n'
' n = len(arr)\n'
' for i in range(n):\n'
' for j in range(0, n - 1):\n'
' if arr[j] > arr[j + 1]:\n'
' arr[j] = arr[j + 1]\n'
' arr[j + 1] = arr[j]\n'
' return arr\n'
'PYEOF'
),
verify=(
'python3 -c "'
'from sorter import bubble_sort; '
'assert bubble_sort([3,1,2]) == [1,2,3]; '
'assert bubble_sort([5,4,3,2,1]) == [1,2,3,4,5]; '
'assert bubble_sort([]) == []; '
'assert bubble_sort([1]) == [1]; '
'assert bubble_sort([2,2,1]) == [1,2,2]'
'"'
),
),
# ------------------------------------------------------------------
# 4. CODE GENERATION
# ------------------------------------------------------------------
BenchmarkTask(
id="codegen-csv-parser",
category="codegen",
difficulty="medium",
instruction=(
"Create a file called csv_parser.py with a function called parse_csv that takes a "
"filename (string) and returns a list of dictionaries. The first row of the CSV is "
"the header. Each subsequent row becomes a dict mapping header names to values. "
"Use only the Python standard library."
),
setup=(
'cat > data.csv << \'CSVEOF\'\n'
'name,age,city\n'
'Alice,30,NYC\n'
'Bob,25,LA\n'
'CSVEOF'
),
verify=(
'python3 -c "'
'from csv_parser import parse_csv; '
'rows = parse_csv(\"data.csv\"); '
'assert len(rows) == 2; '
'assert rows[0][\"name\"] == \"Alice\"; '
'assert rows[0][\"age\"] == \"30\"; '
'assert rows[0][\"city\"] == \"NYC\"; '
'assert rows[1][\"name\"] == \"Bob\"'
'"'
),
),
BenchmarkTask(
id="codegen-stack",
category="codegen",
difficulty="medium",
instruction=(
"Create a file called stack.py with a class called Stack. "
"It should support: push(item), pop() which returns the item (raises IndexError if empty), "
"peek() which returns the top item without removing it (raises IndexError if empty), "
"is_empty() which returns True/False, and size() which returns the count."
),
setup="",
verify=(
'python3 -c "'
'from stack import Stack; '
's = Stack(); '
'assert s.is_empty() == True; '
'assert s.size() == 0; '
's.push(10); s.push(20); s.push(30); '
'assert s.size() == 3; '
'assert s.peek() == 30; '
'assert s.pop() == 30; '
'assert s.pop() == 20; '
'assert s.size() == 1; '
'assert s.is_empty() == False; '
's.pop(); '
'try:\n'
' s.pop()\n'
' assert False\n'
'except IndexError:\n'
' pass; '
'try:\n'
' s.peek()\n'
' assert False\n'
'except IndexError:\n'
' pass'
'"'
),
),
BenchmarkTask(
id="codegen-rest-api",
category="codegen",
difficulty="hard",
instruction=(
"Create a file called todo_api.py that implements a minimal TODO API using only "
"the Python standard library (http.server). It should handle:\n"
" GET /todos -> return JSON list of all todos\n"
" POST /todos -> create a todo from JSON body {\"title\": \"...\"}, return it with an auto-increment id and done=false\n"
" GET /todos/<id> -> return a single todo by id, or 404\n"
"Todos are stored in memory (no database). Each todo has: id (int), title (str), done (bool)."
),
setup="",
verify=(
'python3 << \'TESTEOF\'\n'
'import subprocess, time, json, urllib.request, urllib.error, sys, signal, os\n'
'proc = subprocess.Popen([sys.executable, "todo_api.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n'
'time.sleep(2)\n'
'try:\n'
' # GET empty\n'
' r = urllib.request.urlopen("http://127.0.0.1:8080/todos")\n'
' assert json.loads(r.read()) == [], "GET /todos should be empty"\n'
' # POST\n'
' data = json.dumps({"title": "Buy milk"}).encode()\n'
' req = urllib.request.Request("http://127.0.0.1:8080/todos", data=data, headers={"Content-Type": "application/json"}, method="POST")\n'
' r = urllib.request.urlopen(req)\n'
' todo = json.loads(r.read())\n'
' assert todo["id"] == 1\n'
' assert todo["title"] == "Buy milk"\n'
' assert todo["done"] == False\n'
' # GET by id\n'
' r = urllib.request.urlopen("http://127.0.0.1:8080/todos/1")\n'
' assert json.loads(r.read())["title"] == "Buy milk"\n'
' # 404\n'
' try:\n'
' urllib.request.urlopen("http://127.0.0.1:8080/todos/999")\n'
' assert False\n'
' except urllib.error.HTTPError as e:\n'
' assert e.code == 404\n'
' # GET all\n'
' r = urllib.request.urlopen("http://127.0.0.1:8080/todos")\n'
' assert len(json.loads(r.read())) == 1\n'
' print("ALL_TESTS_PASSED")\n'
'finally:\n'
' proc.terminate()\n'
' proc.wait()\n'
'TESTEOF\n'
'[ $? -eq 0 ]'
),
),
# ------------------------------------------------------------------
# 5. SHELL / SYSTEM TASKS
# ------------------------------------------------------------------
BenchmarkTask(
id="shell-find-largest",
category="shell",
difficulty="easy",
instruction=(
"There are several .txt files in the workspace. Find which .txt file has the most lines "
"and write its filename (just the name, e.g. 'data3.txt') into a file called answer.txt."
),
setup=(
'for i in 1 2 3 4 5; do\n'
' head -c $((i * 50)) /dev/urandom | base64 | head -n $((i * 3)) > "data${i}.txt"\n'
'done'
),
verify=(
'EXPECTED=$(wc -l data*.txt | sort -n | tail -2 | head -1 | awk \'{print $2}\') && '
'[ -f answer.txt ] && '
'ANSWER=$(cat answer.txt | tr -d "[:space:]") && '
'[ "$ANSWER" = "$EXPECTED" ]'
),
),
BenchmarkTask(
id="shell-count-python-funcs",
category="shell",
difficulty="medium",
instruction=(
"Count the total number of Python function definitions (lines starting with 'def ') "
"across ALL .py files in the project/ directory (recursively). Write just the number "
"into a file called answer.txt."
),
setup=(
'mkdir -p project/sub\n'
'cat > project/a.py << \'PY\'\n'
'def foo():\n'
' pass\n'
'def bar():\n'
' pass\n'
'PY\n'
'cat > project/b.py << \'PY\'\n'
'def baz():\n'
' pass\n'
'PY\n'
'cat > project/sub/c.py << \'PY\'\n'
'def one():\n'
' pass\n'
'def two():\n'
' pass\n'
'def three():\n'
' pass\n'
'PY'
),
verify=(
'[ -f answer.txt ] && '
'ANSWER=$(cat answer.txt | tr -d "[:space:]") && '
'[ "$ANSWER" = "6" ]'
),
),
# ------------------------------------------------------------------
# 6. REFACTORING
# ------------------------------------------------------------------
BenchmarkTask(
id="refactor-extract-function",
category="refactor",
difficulty="medium",
instruction=(
"The file process.py has duplicated logic for validating emails in two places. "
"Extract the common validation logic into a single function called is_valid_email "
"and make both register_user and update_email call it. All existing behavior must "
"remain the same."
),
setup=(
'cat > process.py << \'PYEOF\'\n'
'def register_user(name, email):\n'
' if "@" not in email or "." not in email.split("@")[-1]:\n'
' return {"error": "invalid email"}\n'
' return {"name": name, "email": email, "status": "registered"}\n'
'\n'
'def update_email(user, new_email):\n'
' if "@" not in new_email or "." not in new_email.split("@")[-1]:\n'
' return {"error": "invalid email"}\n'
' user["email"] = new_email\n'
' return user\n'
'PYEOF'
),
verify=(
'python3 -c "'
'from process import register_user, update_email, is_valid_email; '
'assert is_valid_email(\"test@example.com\") == True; '
'assert is_valid_email(\"bad\") == False; '
'assert is_valid_email(\"no@dot\") == False; '
'r = register_user(\"Alice\", \"a@b.c\"); assert r[\"status\"] == \"registered\"; '
'r = register_user(\"Bob\", \"bad\"); assert r[\"error\"] == \"invalid email\"; '
'u = {\"name\": \"X\", \"email\": \"old@o.com\"}; '
'r = update_email(u, \"new@n.com\"); assert r[\"email\"] == \"new@n.com\"; '
'r = update_email(u, \"bad\"); assert r[\"error\"] == \"invalid email\"'
'"'
),
),
# ------------------------------------------------------------------
# 7. TESTING / TEST WRITING
# ------------------------------------------------------------------
BenchmarkTask(
id="testgen-write-tests",
category="testing",
difficulty="hard",
instruction=(
"The file calculator.py has a Calculator class with add, subtract, multiply, and "
"divide methods. Write a test file called test_calculator.py using unittest that "
"has at least 8 test cases covering normal usage AND edge cases (division by zero "
"should raise ValueError). All tests must pass when run with 'python3 -m unittest test_calculator -v'."
),
setup=(
'cat > calculator.py << \'PYEOF\'\n'
'class Calculator:\n'
' def add(self, a, b):\n'
' return a + b\n'
'\n'
' def subtract(self, a, b):\n'
' return a - b\n'
'\n'
' def multiply(self, a, b):\n'
' return a * b\n'
'\n'
' def divide(self, a, b):\n'
' if b == 0:\n'
' raise ValueError("Cannot divide by zero")\n'
' return a / b\n'
'PYEOF'
),
verify=(
'python3 -m unittest test_calculator -v 2>&1 | tail -1 | grep -q "OK" && '
'TESTS=$(python3 -m unittest test_calculator -v 2>&1 | grep -c "\\.\\.\\. ok") && '
'[ "$TESTS" -ge 8 ]'
),
),
# ------------------------------------------------------------------
# 8. DATA PROCESSING
# ------------------------------------------------------------------
BenchmarkTask(
id="data-json-transform",
category="data",
difficulty="medium",
instruction=(
"The file users.json contains an array of user objects with fields: name, age, city. "
"Create a Python script called transform.py that reads users.json and writes a new "
"file called summary.json containing: {\"total\": <count>, \"average_age\": <float rounded to 1 decimal>, "
"\"cities\": [<sorted unique city list>]}."
),
setup=(
'cat > users.json << \'JSONEOF\'\n'
'[\n'
' {"name": "Alice", "age": 30, "city": "NYC"},\n'
' {"name": "Bob", "age": 25, "city": "LA"},\n'
' {"name": "Carol", "age": 35, "city": "NYC"},\n'
' {"name": "Dave", "age": 28, "city": "Chicago"},\n'
' {"name": "Eve", "age": 22, "city": "LA"}\n'
']\n'
'JSONEOF'
),
verify=(
'python3 transform.py && '
'python3 -c "'
'import json; '
'd = json.load(open(\"summary.json\")); '
'assert d[\"total\"] == 5; '
'assert d[\"average_age\"] == 28.0; '
'assert d[\"cities\"] == [\"Chicago\", \"LA\", \"NYC\"]'
'"'
),
),
# ------------------------------------------------------------------
# 9. MULTI-FILE PROJECT
# ------------------------------------------------------------------
BenchmarkTask(
id="project-fix-imports",
category="project",
difficulty="hard",
instruction=(
"The project has three files: app/main.py, app/models.py, and app/utils.py. "
"The main.py tries to import from models and utils but has broken imports. "
"Also, app/__init__.py is missing. Fix everything so that running "
"'python3 -m app.main' prints 'App running: User(admin) validated'."
),
setup=(
'mkdir -p app\n'
'cat > app/models.py << \'PY\'\n'
'class User:\n'
' def __init__(self, name):\n'
' self.name = name\n'
' def __repr__(self):\n'
' return f"User({self.name})"\n'
'PY\n'
'cat > app/utils.py << \'PY\'\n'
'def validate(user):\n'
' return user.name is not None and len(user.name) > 0\n'
'PY\n'
'cat > app/main.py << \'PY\'\n'
'from models import User\n'
'from utils import validate\n'
'\n'
'def run():\n'
' u = User("admin")\n'
' v = validate(u)\n'
' status = "validated" if v else "invalid"\n'
' print(f"App running: {u} {status}")\n'
'\n'
'if __name__ == "__main__":\n'
' run()\n'
'PY'
),
verify='python3 -m app.main 2>&1 | grep -qx "App running: User(admin) validated"',
),
# ------------------------------------------------------------------
# 10. ALGORITHM IMPLEMENTATION
# ------------------------------------------------------------------
BenchmarkTask(
id="algo-binary-search",
category="algorithm",
difficulty="medium",
instruction=(
"Create a file called search.py with a function called binary_search that takes "
"a sorted list and a target value. It should return the index of the target if found, "
"or -1 if not found. Implement it using actual binary search (not list.index or linear scan)."
),
setup="",
verify=(
'python3 -c "'
'from search import binary_search; '
'assert binary_search([1,2,3,4,5], 3) == 2; '
'assert binary_search([1,2,3,4,5], 1) == 0; '
'assert binary_search([1,2,3,4,5], 5) == 4; '
'assert binary_search([1,2,3,4,5], 6) == -1; '
'assert binary_search([], 1) == -1; '
'assert binary_search([10], 10) == 0; '
'assert binary_search([10], 5) == -1; '
'assert binary_search(list(range(1000)), 500) == 500'
'"'
),
),
# ------------------------------------------------------------------
# 11. DEBUGGING WITH READING
# ------------------------------------------------------------------
BenchmarkTask(
id="debug-read-and-fix",
category="debug",
difficulty="hard",
instruction=(
"The file server_config.py has a function called parse_config that reads a .ini "
"style config file and returns a dictionary. But it crashes on the provided "
"settings.ini file. Read both files, find the bug, and fix parse_config so it "
"works correctly. Do not modify settings.ini."
),
setup=(
'cat > settings.ini << \'INI\'\n'
'[database]\n'
'host = localhost\n'
'port = 5432\n'
'\n'
'# This is a comment\n'
'[server]\n'
'debug = true\n'
'workers = 4\n'
'\n'
'[logging]\n'
'level = info\n'
'INI\n'
'cat > server_config.py << \'PYEOF\'\n'
'def parse_config(filename):\n'
' result = {}\n'
' current_section = None\n'
' with open(filename) as f:\n'
' for line in f:\n'
' line = line.strip()\n'
' if line.startswith("["):\n'
' current_section = line[1:-1]\n'
' result[current_section] = {}\n'
' elif "=" in line:\n'
' key, value = line.split("=")\n'
' result[current_section][key.strip()] = value.strip()\n'
'PYEOF'
),
verify=(
'python3 -c "'
'from server_config import parse_config; '
'c = parse_config(\"settings.ini\"); '
'assert c[\"database\"][\"host\"] == \"localhost\"; '
'assert c[\"database\"][\"port\"] == \"5432\"; '
'assert c[\"server\"][\"debug\"] == \"true\"; '
'assert c[\"server\"][\"workers\"] == \"4\"; '
'assert c[\"logging\"][\"level\"] == \"info\"'
'"'
),
),
# ------------------------------------------------------------------
# 12. GREP + ANALYSIS
# ------------------------------------------------------------------
BenchmarkTask(
id="analysis-find-todos",
category="analysis",
difficulty="easy",
instruction=(
"Search all .py files in the codebase/ directory recursively for lines containing "
"'TODO'. Create a file called todos.txt where each line has the format: "
"'<filename>:<line_number>: <the TODO text>'. Sort by filename then line number."
),
setup=(
'mkdir -p codebase/sub\n'
'cat > codebase/alpha.py << \'PY\'\n'
'# TODO: add logging\n'
'def alpha():\n'
' pass # TODO: implement\n'
'PY\n'
'cat > codebase/sub/beta.py << \'PY\'\n'
'def beta():\n'
' # TODO: handle errors\n'
' return 42\n'
'PY'
),
verify=(
'[ -f todos.txt ] && '
'[ "$(wc -l < todos.txt | tr -d " ")" = "3" ] && '
'grep -q "alpha.py" todos.txt && '
'grep -q "beta.py" todos.txt && '
'grep -q "TODO" todos.txt'
),
),
)
def get_task(task_id: str) -> BenchmarkTask | None:
for t in TASKS:
if t.id == task_id:
return t
return None
def list_tasks() -> list[dict[str, str]]:
return [
{"id": t.id, "category": t.category, "difficulty": t.difficulty}
for t in TASKS
]
docs/data_agent_implementation_tasks.md
-41
View File
@@ -1,41 +0,0 @@
# 数据 Agent 实现任务表
范围：先在现有 agent loop 上开发和调试 data-agent 的 tools 与 skills，暂时不改主运行链路。
## 极简任务表
| 顺序 | 任务 | 产物 | 验证方式 |
|---|---|---|---|
| 1 | 定义 canonical record v1 | `src/data_agent_schema.py` 或 `src/data_agent/records.py` | 单测覆盖合法和非法样本 |
| 2 | 实现 `validate_dataset_records` 工具 | 新增 `AgentTool` 注册和 handler | 用 JSONL 样例跑工具，检查结构化错误 |
| 3 | 实现 `render_dataset_preview` 工具 | 生成面向人工 review 的 Markdown 预览 | 用 3 条样本生成预览 |
| 4 | 实现 `convert_dataset_format` 工具 | 至少支持 `training_jsonl`、`eval_prompt_csv`、`review_table_csv` | 输入 canonical JSONL，检查导出文件 |
| 5 | 实现 `export_dataset` 工具 | 多格式导出和 manifest | 校验失败时拒绝导出 |
| 6 | 增加最小 data skill | 新增类似 `dataset-construction` 的目录化 skill | Web UI 中可见，并指导 validate/export 循环 |
| 7 | 准备调试样例 | `test_data/data_agent/*.jsonl` | 通过 Web UI 跑 skill + tools |
| 8 | 实现 `normalize_dataset_records` 工具 | 支持简单 CSV/JSONL 字段映射到 canonical JSONL | 输入 CSV，输出 canonical JSONL |
| 9 | 实现 `dedup_dataset_records` 工具 | 按 query + label 做精确去重 | 输出去重后 JSONL 和重复样本报告 |
| 10 | 按需增强 Web UI trace | 更清晰展示 tool 参数和结果 | 人工 UI 验证 |
## 第一批实现切片
先做任务 1、2、6、7：
```text
skill 指导
-> Agent 生成 canonical JSONL
-> validate_dataset_records 校验
-> Web UI 展示工具调用参数和校验错误
```
## 预计第一批文件
```text
src/data_agent_schema.py
src/agent_tools.py
src/bundled_skills.py
tests/test_data_agent_schema.py
tests/test_data_agent_tools.py
test_data/data_agent/valid_records.jsonl
test_data/data_agent/invalid_records.jsonl
```
docs/data_agent_new_tools.md
-730
View File
@@ -1,730 +0,0 @@
下面是中文翻译：
# Data Agent 新工具提案
本文档列出了 data-agent 改造中建议新增的工具。范围刻意聚焦在第一阶段实现：数据 schema、生成、校验、格式转换、评审，以及后续在线挖掘。
## 设计原则
* 优先使用窄口径的数据工具，而不是通用 shell、SQL 或临时代码。
* 工具应该稳定数据格式和可重复操作。
* 开放式推理、聚类解释和生成策略可以保留在 Agent loop 中；工具负责提供结构化输入、校验和可重复转换。
* 每个工具都应该返回机器可读的 metadata，以及人类可读的摘要。
* 生成的数据集必须在导出或持久化前完成校验。
## 优先级
* `P0`：短期立即实现所必需。
* `P1`：很快需要，但可以在第一条 schema / 生成链路跑通后再加。
* `P2`：用于在线挖掘或生产级加固。
# P0：数据集 Schema 与格式工具
这些是基础工具，避免 Agent 手写脆弱的格式转换逻辑。
数据格式设计分两层：
1. **标准数据记录层**：验证数据要素是否完整。该层回答一个 record 是否具备所需的对话轮次、`query`、`tts` 和最终 label。数据生成和在线挖掘工具应该直接产出这一标准层，或者在进一步使用前先 normalize 到这一层。
2. **导出格式层**：把标准 records 转换成固定的下游格式。一旦标准 metadata 格式稳定，转换到 training JSONL、prompt/eval CSV、review tables、display markdown 和其他消费格式，就应该成为确定性的工具逻辑。
实际规则是：
```text
generation/mining output
-> canonical records
-> validation
-> dedup/review
-> deterministic exports
```
Agent 应该负责理解数据意图和编写计划，但 schema 校验和格式转换应该由工具负责。
## 标准 Record 层
这一层定义生成或挖掘数据的内部事实源。第一版应该足够严格，避免缺失关键元素；同时也要足够灵活，以支持单轮和多轮对话。
建议的标准 record 结构：
```json
{
"record_id": "optional-stable-id",
"conversation": [
{
"query": "...",
"tts": "...",
"metadata": {}
}
],
"label": {
"type": "function_or_agent",
"name": "...",
"arguments": {}
},
"source": {
"type": "generated|mined|eval_error|manual",
"task_id": "...",
"notes": "..."
},
"metadata": {}
}
```
待确认的 schema 问题：
* `tts` 是否应该每一轮都必填？还是当源数据没有 TTS 时可以为空？
* label type 是否应该标准化为 `function` 和 `agent` 这样的 enum？
* label arguments 应该是必填、可选，还是由目标格式决定？
* record-level metadata 是否应该包含 model version、device、date、source table、mining strategy、reviewer decision 等信息？
## `validate_dataset_records`
用途：校验 records 是否符合标准数据 schema。
建议输入：
```json
{
"input_path": "artifacts/candidates.jsonl",
"schema_name": "router_conversation_v1",
"max_errors": 50
}
```
建议输出：
```json
{
"valid": true,
"record_count": 120,
"error_count": 0,
"errors": []
}
```
说明：
* 检查多轮对话、`query`、`tts`、最终 `label` 等必填字段。
* 应同时支持单轮和多轮 records。
* 应在生成 / 挖掘之后运行，并在 dedup、export 或 persistence 之前运行。
* 这是完整性和结构校验，不是下游格式校验。
## `normalize_dataset_records`
用途：把历史数据、生成数据、挖掘数据或 eval result 中略有差异的 record 形态转换成标准 record schema。
建议输入：
```json
{
"input_path": "context/raw_cases.csv",
"output_path": "artifacts/normalized_cases.jsonl",
"source_format": "auto",
"schema_name": "router_conversation_v1",
"field_mapping": {
"query": "query",
"tts": "tts",
"label": "correct_label"
}
}
```
说明：
* 现有 eval 文件、文档和表格可能存在格式差异，因此这个工具有用。
* 不应该过度猜测。如果必填字段含义不明确，应返回 mapping error，让 Agent 向用户确认。
* 对于在线挖掘，挖掘工具最好直接输出标准 records。这个 normalizer 主要适合 legacy 文件和用户提供的表格。
## `generate_record_ids`
用途：为标准 records 填充稳定的 `record_id`。
建议输入：
```json
{
"input_path": "artifacts/normalized_cases.jsonl",
"output_path": "artifacts/normalized_cases.with_ids.jsonl",
"id_strategy": "hash"
}
```
说明：
* 稳定 ID 会让 review、dedup、export manifest 和 git diff 更容易处理。
* 默认策略可以对标准化后的 conversation 和 label 做 hash。
# 数据集质量层
这些工具在导出前作用于标准 records。
## `dedup_dataset_records`
用途：导出前删除或标记重复、近重复 records。
建议输入：
```json
{
"input_path": "artifacts/eval_candidates.jsonl",
"output_path": "artifacts/eval_candidates_deduped.jsonl",
"keys": ["conversation.query", "label.name"],
"mode": "flag"
}
```
说明：
* 在 eval set 中，重复样本会抬高指标，或让专项集看起来比实际更大。
* 在 training set 中，重复样本可能会无意中加重某种 pattern 的权重。
* MVP 可以先从精确重复检测开始。近重复检测可以放到 P1 / P2。
* `mode="flag"` 比直接删除更安全，因为 reviewer 可以检查哪些内容被移除了。
# 导出格式层
## `convert_dataset_format`
用途：把已校验的标准 records 转换成一个批准的下游格式。
建议格式：
```text
training_jsonl
eval_prompt_csv
review_table_csv
markdown_preview
```
建议输入：
```json
{
"input_path": "artifacts/eval_candidates.jsonl",
"output_path": "artifacts/eval_candidates.csv",
"target_format": "eval_prompt_csv",
"schema_name": "router_conversation_v1"
}
```
说明：
* 这个工具和 validation 分开，因为不同转换目标可能有不同的列要求和布局要求。
* 它不应该改变语义内容。如果目标格式无法表达某个字段，工具应该在 metadata 中说明哪些字段被省略。
## `render_dataset_preview`
用途：把 records 渲染成人类评审用的格式。
建议输入：
```json
{
"input_path": "artifacts/eval_candidates.jsonl",
"output_path": "artifacts/eval_candidates_preview.md",
"max_records": 50,
"group_by": "label"
}
```
说明：
* Review 格式应该对人友好，但不应被当作训练 / 评测的标准格式。
## `export_dataset`
用途：把已校验的标准 records 导出为一个或多个批准的 artifact 文件，并写入 export manifest。
建议输入：
```json
{
"input_path": "artifacts/eval_candidates_deduped.jsonl",
"exports": [
{
"target_format": "training_jsonl",
"output_path": "artifacts/export/train.jsonl"
},
{
"target_format": "eval_prompt_csv",
"output_path": "artifacts/export/eval.csv"
},
{
"target_format": "markdown_preview",
"output_path": "artifacts/export/review.md"
}
],
"require_valid": true
}
```
说明：
* 这个工具可以在写入最终输出前内部调用 validation。
* 应生成 manifest，包含 record 数量、schema version、export paths 和 validation status。
* 这是基于确定性转换的便利性 / 编排工具。调试时，仍然可以直接调用 `convert_dataset_format`。
# P1：Eval 输入与错误分析工具
这些工具支撑任务 1：利用现有 eval errors 规划并生成针对性数据。
## `load_eval_results`
用途：从 CSV、JSONL 或类 spreadsheet 导出文件中加载 eval result，并转换成标准内部表。
建议输入：
```json
{
"input_path": "context/eval_errors.csv",
"output_path": "artifacts/eval_results.normalized.jsonl",
"field_mapping": {
"query": "query",
"expected_label": "correct_label",
"predicted_label": "model_label"
}
}
```
说明：
* 应支持显式 field mapping。
* 可以支持 `field_mapping="auto"`，但应返回低置信度 mapping warning，而不是静默猜测。
## `profile_error_cases`
用途：从 eval errors 中产出结构化统计和切片分析。
这是比全自动聚类更安全的第一版。
建议输出：
```json
{
"total_errors": 328,
"by_expected_label": {},
"by_predicted_label": {},
"confusion_pairs": [],
"top_terms": [],
"sample_records": []
}
```
说明：
* 这个工具给 Agent 提供分析证据。
* 随后 Agent 可以写 `artifacts/error_analysis.md`。
## `cluster_error_cases`
用途：把相似错误样本分组，便于分析。
处置：`P1`，但应被视为辅助聚类，而不是最终事实。
难点：
* 输入文件格式不一致。
* 好的 cluster 经常需要人工解释。
* 之前的聚类是交互式的，这一点应该保留。
建议设计：
```json
{
"input_path": "artifacts/eval_results.normalized.jsonl",
"output_path": "artifacts/error_clusters.json",
"features": ["query", "expected_label", "predicted_label"],
"method": "heuristic",
"max_clusters": 20,
"include_examples": 10
}
```
推荐 MVP：
* 从 `profile_error_cases` 开始，再基于 confusion pair、label、keywords 或显式字段做简单分组。
* 由 Agent 在 markdown 中产出 cluster 名称和假设。
* 由人工评审、合并、拆分 clusters。
## `summarize_error_clusters`
用途：把 cluster 数据转换成人类可评审的报告。
与 `cluster_error_cases` 的关系：
* `cluster_error_cases` 创建结构化分组和代表样例。
* `summarize_error_clusters` 把这些分组转换成报告：cluster 标题、疑似根因、示例、建议的数据生成方向。
建议输入：
```json
{
"clusters_path": "artifacts/error_clusters.json",
"output_path": "artifacts/error_cluster_summary.md"
}
```
说明：
* 这个工具可能部分由 Agent 编写。工具可以渲染基础确定性报告；Agent 可以在其上补充推理。
## `create_review_packet`
用途：把分析结果打包成一个紧凑的人类评审 artifact。
建议输入：
```json
{
"inputs": [
"artifacts/error_analysis.md",
"artifacts/error_cluster_summary.md"
],
"output_path": "artifacts/review_packet.md",
"questions_path": "memory/open_questions.md"
}
```
说明：
* 在调用 `ask_user_question` 前有用。
* 可以让交互式评审始终基于文件内容展开。
# P1：数据规划与生成工具
这些工具支撑任务 1 和任务 2。
## `generate_data_plan`
用途：基于已评审的错误 clusters 或标签定义创建结构化生成计划。
建议输入：
```json
{
"analysis_path": "artifacts/error_cluster_summary.md",
"constraints_path": "context/data_requirements.md",
"output_path": "artifacts/generation_plan.json"
}
```
建议输出形态：
```json
{
"batches": [
{
"name": "hard_negative_fast_direct",
"target_label": "SLOW_FILTER_RANK",
"count": 50,
"scenario": "...",
"requirements": [],
"negative_constraints": []
}
]
}
```
说明：
* 可以由 Agent 生成，再由工具校验。
* 短期内，这个工具可以先负责校验和规范化计划，而不是完整生成计划。
## `validate_data_plan`
用途：校验 generation plan 是否完整、可执行。
说明：
* 检查数量、标签、必填字段、schema names 和不支持的指令。
* 有助于保持交互式 plan review 稳定。
## `generate_dataset_records`
用途：根据已校验的 plan 和 data schema 生成 records。
建议输入：
```json
{
"plan_path": "artifacts/generation_plan.json",
"output_path": "artifacts/generated_candidates.jsonl",
"schema_name": "router_conversation_v1",
"generation_mode": "llm_assisted"
}
```
说明：
* 如果内部使用 LLM，工具输出仍然需要校验。
* 更简单的 MVP 是：Agent 生成候选 JSONL，然后调用 `validate_dataset_records`。
* 长期看，这个工具可以负责生成，减少格式漂移。
## `validate_label_coverage`
用途：检查生成数据是否覆盖了计划要求的 labels、scenarios 和 boundary types。
建议输入：
```json
{
"dataset_path": "artifacts/generated_candidates.jsonl",
"plan_path": "artifacts/generation_plan.json",
"output_path": "artifacts/coverage_report.md"
}
```
说明：
* 这和 schema validation 不同。
* Schema validation 问的是：“record 结构是否合法？”
* Coverage validation 问的是：“是否生成了计划中想要的数据？”
# P1：产品或标签定义工具
这些工具支撑任务 2。
## `load_definition_document`
用途：把产品或标签定义文档加载为标准 markdown / text。
建议输入：
```json
{
"input_path": "context/product_definition.md",
"output_path": "artifacts/definition.normalized.md"
}
```
说明：
* 可以支持 markdown、txt、csv，之后按需支持 docx / pdf。
## `extract_label_definition`
用途：抽取候选标签定义、正例、反例和模糊边界。
建议输出：
```json
{
"labels": [],
"positive_rules": [],
"negative_rules": [],
"ambiguous_boundaries": [],
"examples": []
}
```
说明：
* 可以由 LLM 辅助，但应输出结构化 JSON 和 markdown summary。
## `render_label_boundary_summary`
用途：基于抽取出的定义创建可评审的标签边界摘要。
建议输出：
```text
artifacts/label_boundary_summary.md
```
说明：
* 应接入 `ask_user_question` 或人工评审流程。
# P2：在线挖掘工具
这些工具支撑任务 3。应在 schema / generation 工具稳定后再添加。
## `analyze_badcases`
用途：总结输入 badcases 的共性特征。
建议输出：
* 常见 query patterns
* 涉及 labels
* 候选过滤条件
* 高风险 / 模糊字段
* 代表样例
## `build_mining_strategy`
用途：把 badcase 分析转换成结构化在线数据搜索策略。
建议输出形态：
```json
{
"filters": {
"date_range": {},
"device": [],
"agent_type": [],
"keywords": [],
"regex": [],
"domain": []
},
"sampling": {
"limit": 100,
"method": "diverse"
}
}
```
说明：
* 大规模查询前，应支持人工评审。
## `search_online_sessions`
用途：通过受控、可审计的 filters 查询线上 sessions。
说明：
* Agent 不应该写原始 SQL。
* 工具 schema 应只暴露被批准的字段和过滤操作符。
* 敏感字段应在工具边界完成脱敏。
## `sample_online_candidates`
用途：从检索到的线上 sessions 中采样候选样本，供评审使用。
说明：
* 支持“先标注 100 条”的循环。
* 应包含确定性 sampling metadata，保证可复现。
## `create_annotation_batch`
用途：从采样候选中创建 review / annotation packet。
说明：
* MVP 可以是本地 markdown / CSV。
* 之后可以接入标注系统。
## `read_annotation_result`
用途：把人工评审结果读回 task workspace。
说明：
* 应规范化 review labels 和 comments。
## `evaluate_mining_precision`
用途：评估当前 mining strategy 的精度是否足够。
建议输出：
* 已评审数量
* 正样本数量
* Precision
* 主要误召 pattern
* 推荐的下一步过滤条件调整
## `refine_mining_strategy`
用途：根据评估结果生成修订版 mining strategy。
说明：
* 初期可以保留为 Agent 编写；工具负责校验 strategy JSON。
## `export_mined_dataset`
用途：把挖掘数据导出成 badcase set、eval set 或 training set 格式。
说明：
* 应调用或复用 validation、dedup、conversion 和 export 逻辑。
# 持久化与 Git 工具
## `persist_dataset_artifact`
用途：持久化已批准的数据集导出，并写入 manifest。
建议输入：
```json
{
"artifact_paths": [
"artifacts/export/train.jsonl",
"artifacts/export/eval.csv"
],
"manifest_path": "artifacts/export/manifest.json",
"destination": "workspace"
}
```
说明：
* `destination="workspace"` 可作为 MVP。
* 之后 destination 可以包括对象存储、dataset registry 或内部平台。
## `prepare_dataset_git_commit`
用途：stage 已批准的数据集 artifacts，并生成 commit summary。
处置：`P1/P2`。
说明：
* 实际 git push 应需要人工确认。
* 这个工具应和 dataset export 分开。Export 负责创建文件；git persistence 负责发布或版本化。
# 建议的首批实现集合
当前短期工作建议先实现这些：
```text
validate_dataset_records
normalize_dataset_records
generate_record_ids
convert_dataset_format
render_dataset_preview
dedup_dataset_records
export_dataset
load_eval_results
profile_error_cases
validate_data_plan
persist_dataset_artifact
```
然后再添加：
```text
cluster_error_cases
summarize_error_clusters
generate_data_plan
generate_dataset_records
validate_label_coverage
load_definition_document
extract_label_definition
render_label_boundary_summary
```
最后添加在线挖掘工具：
```text
analyze_badcases
build_mining_strategy
search_online_sessions
sample_online_candidates
create_annotation_batch
read_annotation_result
evaluate_mining_precision
refine_mining_strategy
export_mined_dataset
```
# 开放问题
* 对话 records 的 canonical schema name 和 version 是什么？
* 确切接受哪些 export formats？需要哪些必填列？
* 生成 records 中的 `tts` 应该是必填、可选，还是派生字段？
* 哪些 labels 是合法的？label registry 存在哪里？
* 在线挖掘时，哪些字段允许用于过滤和导出？
* 哪些动作在持久化或 git push 前需要人工确认？
docs/data_agent_skill_map.md
-26
View File
@@ -1,26 +0,0 @@
# 数据 Agent Skill 地图
这是一份第一批 data-agent 目录化 skill 的工作索引。
## Skill 列表
| Skill | 主要用途 | 当前状态 |
|---|---|---|
| `eval-error-data-generation` | 分析评测错误，并规划针对性训练/评测数据生成 | 草案；数据工具仍待实现 |
| `product-definition-data-generation` | 从产品/标签定义、手写规则或示例 query 生成边界感知的数据计划、dataset draft text 和 canonical records | 草案；已合并数据格式协议，定义抽取工具仍待实现 |
| `online-badcase-mining` | 分析 badcase，并起草可迭代的线上挖掘策略 | 草案；线上挖掘工具仍待实现 |
| `tool-smoke-test` | 验证目录化 skill 加载和基础工具 trace | 可用 smoke test |
## 示例触发 Query
```text
Use the eval-error-data-generation skill. 输入文件：tasks/debug/context/eval_errors.csv。请先起草错误分析和数据生成计划。
```
```text
Use the product-definition-data-generation skill. 输入文件：tasks/debug/context/product_definition.md。请总结标签边界并起草数据生成计划，确认后生成 dataset draft text 并转成 canonical records。
```
```text
Use the online-badcase-mining skill. 输入文件：tasks/debug/context/badcases.jsonl。请分析共性模式并起草挖掘策略。
```
docs/data_agent_tool_inventory.md
-189
View File
@@ -1,189 +0,0 @@
# Data Agent 工具清单
本文档是当前 `claw-code-agent` 工具能力面的工作清单，用于未来的数据 Agent 改造。
范围：
* 源注册表：`src.agent_tools.default_tool_registry()`
* 工具形态：`AgentTool(name, description, parameters, handler)`
* 运行时执行：模型输出 `tool_calls`，随后 `LocalCodingAgent` 执行匹配的 handler，并把结果作为 tool message 写回。
## 当前工具生命周期
1. `default_tool_registry()` 构建基础注册表。
2. 在 `LocalCodingAgent.__post_init__` 阶段，可能会把插件别名和虚拟工具合并进注册表。
3. 每个 `AgentTool` 会通过 `to_openai_tool()` 转换成 OpenAI 兼容的 function schema。
4. 模型接收 `messages` 和 `tool_specs`。
5. 如果模型返回 `tool_calls`，运行时会执行每个指定名称的工具。
6. 工具 handler 接收 `(arguments, ToolExecutionContext)`。
7. handler 返回字符串，或返回 `(content, metadata)`。
8. 运行时序列化结果，并把它作为 `role="tool"` 的消息追加进去，供下一轮模型调用使用。
## Data-Agent 处置标记说明
* `Keep`：适合首版 data-agent MVP 使用。
* `Candidate`：可能有用，但应只在具体数据工作流需要时启用。
* `Wrap`：能力有用，但应该通过数据专用工具名或更窄的 schema 暴露。
* `Defer`：首版 data-agent MVP 暂不需要。
* `Disable`：对首版 data-agent MVP 来说太宽泛，或过于面向代码。
* `Special`：由 agent loop 特殊处理，不是普通业务工具。
## 工作区与文件工具
| 工具 | 当前描述 | Data-Agent 处置 | 备注 |
| --------------- | ------------------------------------------------------- | ------------- | ----------------------------------------------- |
| `list_dir` | 列出 workspace 路径下的文件和目录。 | Keep | 用于任务工作区检查。 |
| `read_file` | 读取 workspace 内 UTF-8 文本文件内容。 | Keep | Markdown 优先流程和 artifact 读取的核心工具。 |
| `write_file` | 在 workspace 内完整写入文件；必要时创建父目录。 | Keep | 用于 `goal.md`、memory 文件、报告、JSONL artifact。需要写权限。 |
| `edit_file` | 使用精确字符串匹配替换 workspace 文件中的文本。 | Keep | 适合增量更新 memory / artifact。 |
| `notebook_edit` | 通过替换或追加 source 来编辑 `.ipynb` 文件中的 Jupyter notebook cell。 | Defer | data-agent MVP 应优先使用 Markdown、JSONL、CSV 和报告。 |
| `glob_search` | 在 workspace 内查找匹配 glob pattern 的文件。 | Keep | 用于发现任务文件和 skill 文件。 |
| `grep_search` | 在 workspace 文件中搜索字符串或正则表达式。 | Keep | 用于本地标签定义、历史笔记和 skill 查找。 |
## Shell 与代码智能工具
| 工具 | 当前描述 | Data-Agent 处置 | 备注 |
| ------ | ------------------------- | ------------- | --------------------------------------- |
| `bash` | 在 workspace 内运行 shell 命令。 | Disable | 对 data-agent MVP 来说过于宽泛。优先使用专用数据工具和校验器。 |
| `LSP` | 使用本地 LSP 风格的代码智能能力。 | Disable | 面向代码的能力；数据任务不需要。 |
## Web 与搜索工具
| 工具 | 当前描述 | Data-Agent 处置 | 备注 |
| -------------------------- | ------------------------------- | ------------- | ---------------------------------------- |
| `web_fetch` | 从 HTTP、HTTPS 或 file URL 获取文本资源。 | Defer | 可能用于文档读取，但不是核心数据工作流。 |
| `search_status` | 显示本地搜索运行时摘要。 | Candidate | 仅当在线 session / 搜索 provider 被建模为搜索运行时时有用。 |
| `search_list_providers` | 列出已配置的本地搜索 provider。 | Candidate | 同上。 |
| `search_activate_provider` | 设置当前激活的本地搜索 provider。 | Defer | 属于配置动作，不应是普通 agent 任务行为。 |
| `web_search` | 通过已配置后端执行真实 Web 搜索。 | Defer | 外部 Web 搜索不同于内部数据挖掘。 |
| `tool_search` | 搜索当前激活的工具注册表。 | Keep | 在工具面持续演进时有用。之后进入严格生产模式可以移除。 |
| `sleep` | 暂停执行一小段时间。 | Defer | 适合轮询，但在存在异步任务之前不需要。 |
## 人工评审与决策工具
| 工具 | 当前描述 | Data-Agent 处置 | 备注 |
| ------------------- | ---------------------------- | ------------- | -------------- |
| `ask_user_question` | 向本地 ask-user runtime 请求用户回答。 | Keep | 对边界澄清和人工裁决很重要。 |
## 账号与配置工具
| 工具 | 当前描述 | Data-Agent 处置 | 备注 |
| ----------------------- | ------------------------------------ | ------------- | --------------------------------- |
| `account_status` | 显示本地账号运行时摘要。 | Candidate | 如果数据系统需要账号 profile，会有用。 |
| `account_list_profiles` | 列出已配置的本地账号 profile。 | Candidate | 用于调试访问配置。 |
| `account_login` | 激活本地账号 profile 或临时身份。 | Defer | 不应出现在常规自主数据任务循环里。 |
| `account_logout` | 清空当前激活账号 session 状态。 | Defer | 运维动作。 |
| `config_list` | 列出合并后的或特定来源的 workspace 配置 key。 | Candidate | 用于检查 data-agent 配置。 |
| `config_get` | 通过 dotted key path 读取 workspace 配置值。 | Candidate | 用于 data-agent policy / config 查询。 |
| `config_set` | 写入 workspace 配置值。 | Disable | 配置变更应由人控制。 |
## MCP 工具
| 工具 | 当前描述 | Data-Agent 处置 | 备注 |
| -------------------- | ------------------------------- | ------------- | ----------------------------------------------------------------------- |
| `mcp_list_resources` | 列出本地 MCP resources。 | Candidate | 适合 skill / resource 发现或内部文档读取。 |
| `mcp_read_resource` | 通过 URI 读取本地 MCP resource。 | Candidate | 如果标签、策略或历史决策以 resource 形式暴露，会有用。 |
| `mcp_list_tools` | 列出已配置 MCP server 暴露的 MCP tools。 | Candidate | 用于发现数据系统工具。 |
| `mcp_call_tool` | 调用已配置 MCP server 暴露的 MCP tool。 | Wrap | 这是强大的通用桥接能力。常规 agent 使用时，应优先使用 `search_online_sessions` 这类数据专用 wrapper。 |
## Remote、Worktree、Workflow 与 Trigger 工具
| 工具 | 当前描述 | Data-Agent 处置 | 备注 |
| ---------------------- | -------------------------------- | ------------- | --------------------------------- |
| `remote_status` | 显示本地 remote 运行时摘要。 | Defer | 运维相关。 |
| `remote_list_profiles` | 列出已配置的本地 remote profile。 | Defer | 运维相关。 |
| `remote_connect` | 激活本地 remote target / profile。 | Disable | 对 data-agent 自主模式来说过于宽泛。 |
| `remote_disconnect` | 清空当前激活 remote connection。 | Disable | 运维变更。 |
| `worktree_status` | 显示当前受管 git worktree session 状态。 | Disable | 面向代码。 |
| `worktree_enter` | 创建并进入隔离 git worktree。 | Disable | 面向代码。 |
| `worktree_exit` | 离开当前受管 worktree session。 | Disable | 面向代码。 |
| `workflow_list` | 列出本地 workflow 定义。 | Candidate | 如果数据 pipeline 以 workflow 表示，可能有用。 |
| `workflow_get` | 显示某个本地 workflow 定义。 | Candidate | 同上。 |
| `workflow_run` | 记录并渲染一次 workflow 执行请求。 | Wrap | 应变成明确的数据动作，而不是任意 workflow 执行。 |
| `remote_trigger` | 列出、查看、创建、更新或运行本地 remote trigger。 | Defer | 更偏自动化，首版 data-agent MVP 暂不需要。 |
## Planning、Task、Team 与后台工具
| 工具 | 当前描述 | Data-Agent 处置 | 备注 |
| --------------- | ----------------------------- | ------------- | -------------------------------------- |
| `plan_get` | 显示当前本地运行时 plan。 | Candidate | 对结构化长周期数据任务有用。 |
| `update_plan` | 替换当前运行时 plan。 | Candidate | 有用，但不能替代任务 workspace memory。 |
| `plan_clear` | 清空当前运行时 plan。 | Defer | 运维动作。 |
| `task_next` | 显示下一批可执行 runtime task。 | Candidate | 之后可映射到数据任务 backlog。 |
| `task_list` | 列出本地存储的 runtime task。 | Candidate | 可支持数据任务状态。 |
| `task_get` | 通过 id 显示本地存储的 runtime task。 | Candidate | 同上。 |
| `task_create` | 创建本地存储的 runtime task。 | Defer | 优先使用 task workspace。 |
| `task_update` | 更新本地存储的 runtime task。 | Defer | 优先使用 task workspace。 |
| `task_start` | 标记任务为进行中。 | Defer | 优先使用 task workspace。 |
| `task_complete` | 标记任务为完成。 | Defer | 优先使用 task workspace。 |
| `task_block` | 标记任务被阻塞。 | Defer | 优先使用 `open_questions.md`。 |
| `task_cancel` | 标记任务被取消。 | Defer | 运维动作。 |
| `team_list` | 列出本地配置的协作团队。 | Defer | 非核心 MVP。 |
| `team_get` | 显示某个本地配置的协作团队。 | Defer | 非核心 MVP。 |
| `team_create` | 创建本地存储的协作团队。 | Disable | 非核心 MVP；属于变更动作。 |
| `team_delete` | 删除本地存储的协作团队。 | Disable | 破坏性运维动作。 |
| `send_message` | 发送本地协作消息。 | Defer | 之后在评审路由中可能有用。 |
| `team_messages` | 显示已记录的协作消息。 | Defer | 之后在评审路由中可能有用。 |
| `todo_write` | 用结构化 todo list 替换当前本地运行时任务列表。 | Candidate | 可用于 agent 自组织；但 task workspace 仍应是事实源。 |
| `TaskOutput` | 通过 ID 获取后台任务输出。 | Defer | 等异步数据任务存在后有用。 |
| `TaskStop` | 通过 ID 停止运行中的后台任务。 | Defer | 等异步数据任务存在后有用。 |
## Loop 特殊工具
| 工具 | 当前描述 | Data-Agent 处置 | 备注 |
| ---------------- | ------------------- | ------------- | ----------------------------------------- |
| `EnterPlanMode` | 进入 plan mode。 | Special | coding-agent 行为；可能会被 data-agent 任务规划指导替代。 |
| `ExitPlanMode` | 退出 plan mode。 | Special | 同上。 |
| `Agent` | 为复杂任务启动一个新 agent。 | Defer | 之后可用于并行挖掘 / 评审，但首版 MVP 应保持单 agent loop。 |
| `delegate_agent` | 旧版：把子任务委托给嵌套 agent。 | Disable | 如果之后需要委托，优先使用 `Agent`。 |
| `Skill` | 在主对话中执行 skill。 | Special | 作为概念入口保留，但基于目录的数据 skills 需要单独设计。 |
## 建议的 MVP Data Tool Registry
首版 data-agent 实验建议从这个最小注册表开始：
```text
list_dir
read_file
write_file
edit_file
glob_search
grep_search
ask_user_question
tool_search
```
原型阶段可选 MCP 桥接：
```text
mcp_list_resources
mcp_read_resource
mcp_list_tools
mcp_call_tool
```
当内部数据工具准备好后，应优先添加窄口径的数据专用工具，而不是暴露宽泛工具：
```text
search_online_sessions
sample_top_queries
find_similar_cases
mine_by_pattern
mine_by_model_disagreement
cluster_and_dedup
generate_eval_candidates
validate_eval_jsonl
run_router_eval
run_online_replay
create_annotation_queue
read_human_review_result
persist_dataset_artifact
```
## Data-Agent 改造实现说明
* 新增 `default_data_tool_registry()`，不要直接修改 `default_tool_registry()`。
* 第一阶段保留文件工具、人工评审工具，以及可选的 MCP discovery。
* 内部数据系统应通过稳定、可审计的数据工具暴露，而不是通过 `bash`、原始 SQL 或通用 `mcp_call_tool` 暴露。
* 将 `mcp_call_tool` 视为开发桥接；生产 data-agent 流程应使用面向目的构建的 wrapper tools。
* 工具权限应与数据治理动作对齐，而不只是与底层执行机制对齐。
docs/task-001-directory-skills.md
-43
View File
@@ -1,43 +0,0 @@
# Task 001：目录化 Skills
## 目标
把 skill 内容从 Python 内置字符串迁移到文件目录中，让 data-agent 工作流可以用文件形式开发、review 和维护。
## 范围
- 保持当前 `Skill` 工具和 agent loop 不变。
- 支持从 `src/skills/bundled/*/SKILL.md` 加载内置目录化 skill。
- 保持 Python 动态 skill 可用。
- 使用已有内置 skill 作为迁移样例。
## 第一批切片
- `simplify` 和 `debug` 继续保留 Python 实现，因为它们需要读取动态运行时数据。
- `verify` 和 `update-config` 迁移到目录化 skill。
- `SKILL.md` 支持简单 front matter：
```text
---
name: verify
description: Verify a code change works by running the app and tests.
when_to_use: When the user asks to verify, test, or check that recent changes work.
allowed_tools: read_file, bash, grep_search, glob_search
---
Skill prompt body...
```
## 成功标准
- `/skills` 仍然能列出迁移后的 skills。
- Web UI `/api/skills` 仍然能返回迁移后的 skills。
- `Skill({"skill": "verify"})` 返回 `SKILL.md` 中的 prompt 正文。
- 现有 Python 动态 skills 仍然可用。
## 当前状态
- 目录化 skill loader 已在 `src/bundled_skills.py` 实现。
- `verify` 已迁移到 `src/skills/bundled/verify/SKILL.md`。
- `update-config` 已迁移到 `src/skills/bundled/update-config/SKILL.md`。
- 动态 skills `simplify` 和 `debug` 继续保持 Python 实现。
docs/task-002-project-skills.md
-33
View File
@@ -1,33 +0,0 @@
# Task 002：项目级 Skills
## 目标
允许项目维护者直接在 workspace 根目录添加 skills：
```text
skills/
my-skill/
SKILL.md
```
## 查找顺序
当存在 workspace `cwd` 时：
1. `cwd/skills/*/SKILL.md`
2. `src/skills/bundled/*/SKILL.md`
3. Python 动态 skills
项目级 skills 优先于同名的内置目录 skill 或 Python 动态 skill。
## 接入点
- `Skill` 工具执行时，会通过当前 runtime `cwd` 解析项目级 skills。
- `/skills` 会列出当前 workspace 下的项目级 skills。
- Web UI `/api/skills` 会列出当前 Working dir 下的项目级 skills。
## 说明
- 项目级 skill 与内置目录化 skill 共用同一套 `SKILL.md` front matter 格式。
- `name`、`aliases`、`allowed_tools` 保持面向机器的英文字段。
- skill 正文可以使用中文或英文维护。
images/demo1.gif
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 4.7 MiB

images/demo_2.gif
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 5.1 MiB

images/logo.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 5.8 MiB