Add 10 standard evaluation benchmark suites with CLI runner and README

Implements HumanEval, MBPP, SWE-Bench, Aider, LiveCodeBench (coding),
MATH, GSM8K, AIME (math), and IFEval, BFCL (instruction following).

Each suite includes built-in problem subsets (108 total) and supports
loading full datasets from JSONL files. Includes comprehensive README
with all commands.

Agent-Logs-Url: https://github.com/HarnessLab/claw-code-agent/sessions/6890e3d0-3058-4b1f-b7e5-27171c079c62

Co-authored-by: abdoelsayed2016 <27821589+abdoelsayed2016@users.noreply.github.com>

benchmarks/README.md

@@ -0,0 +1,333 @@
+# Benchmarks — claw-code-agent
+
+This directory contains two benchmark systems:
+
+1. **Local task benchmarks** (`benchmarks/run.py`) — custom tasks that test agent capabilities directly
+2. **Standard evaluation suites** (`benchmarks/run_suite.py`) — implementations of well-known AI evaluation benchmarks
+
+---
+
+## Quick Start
+
+```bash
+# From the repository root:
+
+# Set up your model endpoint
+export OPENAI_API_KEY="your-key"
+export OPENAI_MODEL="gpt-4"                        # or your model name
+export OPENAI_BASE_URL="http://localhost:8000/v1"   # if using local vLLM/ollama
+
+# Run a quick smoke test (5 problems from HumanEval)
+python3 -m benchmarks.run_suite --suite humaneval --limit 5 -v
+
+# Run all benchmarks
+python3 -m benchmarks.run_suite --all -o results.json
+```
+
+---
+
+## Standard Evaluation Suites
+
+### Available Benchmarks
+
+| Suite | Category | # Problems (built-in) | Description |
+|-------|----------|----------------------|-------------|
+| **HumanEval** | Coding | 20 | Code generation from Python docstrings |
+| **MBPP** | Coding | 15 | Basic Python programming problems |
+| **SWE-Bench** | Coding | 5 | Resolve real-world GitHub issues |
+| **Aider** | Coding | 6 | Code editing and refactoring tasks |
+| **LiveCodeBench** | Coding | 5 | Competitive programming problems |
+| **MATH** | Math | 15 | Competition mathematics problems |
+| **GSM8K** | Math | 15 | Grade school math word problems |
+| **AIME** | Math | 10 | Challenging competition math (integers 0–999) |
+| **IFEval** | Instruction Following | 10 | Verifiable instruction-following evaluation |
+| **BFCL** | Instruction Following | 7 | Function/tool calling evaluation |
+
+### Commands
+
+#### List All Suites
+
+```bash
+python3 -m benchmarks.run_suite --list
+```
+
+#### Run a Specific Suite
+
+```bash
+# Coding benchmarks
+python3 -m benchmarks.run_suite --suite humaneval
+python3 -m benchmarks.run_suite --suite mbpp
+python3 -m benchmarks.run_suite --suite swe-bench
+python3 -m benchmarks.run_suite --suite aider
+python3 -m benchmarks.run_suite --suite livecodebench
+
+# Math benchmarks
+python3 -m benchmarks.run_suite --suite math
+python3 -m benchmarks.run_suite --suite gsm8k
+python3 -m benchmarks.run_suite --suite aime
+
+# Instruction following benchmarks
+python3 -m benchmarks.run_suite --suite ifeval
+python3 -m benchmarks.run_suite --suite bfcl
+```
+
+#### Run by Category
+
+```bash
+# All coding benchmarks (~51 problems)
+python3 -m benchmarks.run_suite --category coding
+
+# All math benchmarks (~40 problems)
+python3 -m benchmarks.run_suite --category math
+
+# All instruction following benchmarks (~17 problems)
+python3 -m benchmarks.run_suite --category instruction-following
+```
+
+#### Run ALL Suites
+
+```bash
+python3 -m benchmarks.run_suite --all
+```
+
+#### Limit Problems (Quick Testing)
+
+```bash
+# Run only first 3 problems from each suite
+python3 -m benchmarks.run_suite --all --limit 3
+
+# Quick HumanEval smoke test
+python3 -m benchmarks.run_suite --suite humaneval --limit 5 -v
+```
+
+#### Save Results to JSON
+
+```bash
+python3 -m benchmarks.run_suite --all -o results.json
+python3 -m benchmarks.run_suite --suite humaneval -o humaneval_results.json
+```
+
+#### Verbose Output
+
+```bash
+python3 -m benchmarks.run_suite --suite humaneval -v
+```
+
+#### Custom Timeout
+
+```bash
+# 10 minutes per problem
+python3 -m benchmarks.run_suite --suite swe-bench --timeout 600
+```
+
+#### Use Full Datasets (JSONL)
+
+```bash
+# Download or place your datasets in benchmarks/data/
+python3 -m benchmarks.run_suite --suite humaneval --data-dir ./benchmarks/data/
+
+# Or specify a custom directory
+python3 -m benchmarks.run_suite --suite humaneval --data-dir /path/to/datasets/
+```
+
+---
+
+## Using Full Datasets
+
+Each suite looks for a JSONL file in the data directory. If the file isn't found, it falls back to the built-in subset.
+
+| Suite | Expected File | Format |
+|-------|--------------|--------|
+| HumanEval | `humaneval.jsonl` | `{"task_id", "prompt", "canonical_solution", "test", "entry_point"}` |
+| MBPP | `mbpp.jsonl` | `{"task_id", "text", "code", "test_list"}` |
+| SWE-Bench | `swe_bench.jsonl` | `{"instance_id", "problem_statement", "setup_code", "test_cmd"}` |
+| Aider | `aider.jsonl` | `{"id", "instruction", "setup_code", "test_code"}` |
+| LiveCodeBench | `livecodebench.jsonl` | `{"id", "title", "description", "test_cases", "function_name"}` |
+| MATH | `math.jsonl` | `{"id", "problem", "answer", "subject", "level"}` |
+| GSM8K | `gsm8k.jsonl` | `{"id", "question", "answer"}` |
+| AIME | `aime.jsonl` | `{"id", "problem", "answer"}` |
+| IFEval | `ifeval.jsonl` | `{"id", "instruction", "checks"}` |
+| BFCL | `bfcl.jsonl` | `{"id", "instruction", "expected_function", "setup_code", "test_code"}` |
+
+### Downloading Full Datasets
+
+```bash
+# HumanEval (from OpenAI)
+wget -O benchmarks/data/humaneval.jsonl \
+  https://raw.githubusercontent.com/openai/human-eval/master/data/HumanEval.jsonl.gz
+gunzip benchmarks/data/humaneval.jsonl.gz
+
+# GSM8K (from HuggingFace — requires datasets library)
+pip install datasets
+python3 -c "
+from datasets import load_dataset
+import json
+ds = load_dataset('gsm8k', 'main', split='test')
+with open('benchmarks/data/gsm8k.jsonl', 'w') as f:
+    for i, item in enumerate(ds):
+        f.write(json.dumps({'id': f'gsm8k-{i}', 'question': item['question'], 'answer': item['answer'].split('####')[-1].strip()}) + '\n')
+"
+
+# MATH (from HuggingFace)
+python3 -c "
+from datasets import load_dataset
+import json
+ds = load_dataset('hendrycks/competition_math', split='test')
+with open('benchmarks/data/math.jsonl', 'w') as f:
+    for i, item in enumerate(ds):
+        f.write(json.dumps({'id': f'math-{i}', 'problem': item['problem'], 'answer': item['solution'], 'subject': item['type'], 'level': item['level']}) + '\n')
+"
+```
+
+---
+
+## Local Task Benchmarks
+
+The original local benchmark system tests the agent on custom tasks.
+
+```bash
+# Run all local tasks
+python3 -m benchmarks.run
+
+# Run a single task
+python3 -m benchmarks.run --task file-create-basic
+
+# Run a category
+python3 -m benchmarks.run --category bugfix
+
+# Run by difficulty
+python3 -m benchmarks.run --difficulty easy
+
+# List available tasks
+python3 -m benchmarks.run --list
+
+# Verbose + save results
+python3 -m benchmarks.run -v -o local_results.json
+```
+
+---
+
+## All Commands Reference
+
+```bash
+# ─── STANDARD EVALUATION SUITES ────────────────────────────────────────
+
+# List suites
+python3 -m benchmarks.run_suite --list
+
+# Individual suites
+python3 -m benchmarks.run_suite --suite humaneval          # Coding: docstring → code
+python3 -m benchmarks.run_suite --suite mbpp               # Coding: basic Python
+python3 -m benchmarks.run_suite --suite swe-bench          # Coding: GitHub issues
+python3 -m benchmarks.run_suite --suite aider              # Coding: code editing
+python3 -m benchmarks.run_suite --suite livecodebench      # Coding: competitive programming
+python3 -m benchmarks.run_suite --suite math               # Math: competition math
+python3 -m benchmarks.run_suite --suite gsm8k              # Math: grade school
+python3 -m benchmarks.run_suite --suite aime               # Math: AIME competition
+python3 -m benchmarks.run_suite --suite ifeval             # Instruction: format following
+python3 -m benchmarks.run_suite --suite bfcl               # Instruction: function calling
+
+# Category runs
+python3 -m benchmarks.run_suite --category coding                  # All coding (~51 problems)
+python3 -m benchmarks.run_suite --category math                    # All math (~40 problems)
+python3 -m benchmarks.run_suite --category instruction-following   # All IF (~17 problems)
+
+# Full run
+python3 -m benchmarks.run_suite --all                              # All suites (~108 problems)
+python3 -m benchmarks.run_suite --all --limit 3                    # Quick: 3 per suite
+python3 -m benchmarks.run_suite --all -v -o results.json           # Full + verbose + save
+
+# Options
+python3 -m benchmarks.run_suite --suite humaneval --limit 5        # Limit problems
+python3 -m benchmarks.run_suite --suite humaneval --timeout 600    # Custom timeout (seconds)
+python3 -m benchmarks.run_suite --suite humaneval --data-dir ./data # Custom data directory
+python3 -m benchmarks.run_suite --suite humaneval -v               # Verbose mode
+python3 -m benchmarks.run_suite --suite humaneval -o out.json      # Save to JSON
+
+# ─── LOCAL TASK BENCHMARKS ─────────────────────────────────────────────
+
+python3 -m benchmarks.run --list                            # List local tasks
+python3 -m benchmarks.run                                   # Run all local tasks
+python3 -m benchmarks.run --task file-create-basic          # Single task
+python3 -m benchmarks.run --category bugfix                 # Category filter
+python3 -m benchmarks.run --difficulty easy                 # Difficulty filter
+python3 -m benchmarks.run -v -o local_results.json          # Verbose + save
+```
+
+---
+
+## Environment Variables
+
+| Variable | Description | Example |
+|----------|-------------|---------|
+| `OPENAI_API_KEY` | API key for the model provider | `sk-...` |
+| `OPENAI_MODEL` | Model name to use | `gpt-4`, `qwen2.5-coder-32b` |
+| `OPENAI_BASE_URL` | API base URL (for local models) | `http://localhost:8000/v1` |
+
+---
+
+## Output Format
+
+Results are saved as JSON with this structure:
+
+```json
+{
+  "benchmark_run": "claw-code-agent-evaluation",
+  "timestamp": "2025-01-15T10:30:00",
+  "suites": [
+    {
+      "suite_name": "HumanEval",
+      "total": 20,
+      "passed": 15,
+      "failed": 5,
+      "score_pct": 75.0,
+      "duration_sec": 450.5,
+      "model": "gpt-4",
+      "results": [
+        {
+          "problem_id": "HumanEval/0",
+          "passed": true,
+          "duration_sec": 12.3,
+          "error": ""
+        }
+      ]
+    }
+  ],
+  "summary": {
+    "total_suites": 10,
+    "total_problems": 108,
+    "total_passed": 85,
+    "overall_score_pct": 78.7
+  }
+}
+```
+
+---
+
+## Architecture
+
+```
+benchmarks/
+├── __init__.py
+├── run.py                  # Local task benchmark runner
+├── run_suite.py            # Standard evaluation suite runner (CLI)
+├── README.md               # This file
+├── data/                   # Dataset files (JSONL) — not committed
+│   └── .gitkeep
+├── tasks/
+│   ├── __init__.py
+│   └── definitions.py      # Local task definitions
+└── suites/
+    ├── __init__.py
+    ├── base.py             # Base class for all suites
+    ├── humaneval.py        # HumanEval benchmark
+    ├── mbpp.py             # MBPP benchmark
+    ├── swe_bench.py        # SWE-Bench benchmark
+    ├── aider.py            # Aider benchmark
+    ├── livecodebench.py    # LiveCodeBench benchmark
+    ├── math_bench.py       # MATH benchmark
+    ├── gsm8k.py            # GSM8K benchmark
+    ├── aime.py             # AIME benchmark
+    ├── ifeval.py           # IFEval benchmark
+    └── bfcl.py             # BFCL benchmark
+```

benchmarks/run_suite.py

@@ -0,0 +1,252 @@
+#!/usr/bin/env python3
+"""
+Standard evaluation benchmark runner for claw-code-agent.
+
+Runs the agent against well-known evaluation suites and reports scores.
+
+Usage:
+    # List available benchmark suites
+    python3 -m benchmarks.run_suite --list
+
+    # Run a specific suite (built-in subset)
+    python3 -m benchmarks.run_suite --suite humaneval
+    python3 -m benchmarks.run_suite --suite mbpp
+    python3 -m benchmarks.run_suite --suite swe-bench
+    python3 -m benchmarks.run_suite --suite aider
+    python3 -m benchmarks.run_suite --suite livecodebench
+    python3 -m benchmarks.run_suite --suite math
+    python3 -m benchmarks.run_suite --suite gsm8k
+    python3 -m benchmarks.run_suite --suite aime
+    python3 -m benchmarks.run_suite --suite ifeval
+    python3 -m benchmarks.run_suite --suite bfcl
+
+    # Run ALL suites
+    python3 -m benchmarks.run_suite --all
+
+    # Run by category
+    python3 -m benchmarks.run_suite --category coding
+    python3 -m benchmarks.run_suite --category math
+    python3 -m benchmarks.run_suite --category instruction-following
+
+    # Limit problems per suite (for quick testing)
+    python3 -m benchmarks.run_suite --suite humaneval --limit 5
+
+    # Verbose output + save results
+    python3 -m benchmarks.run_suite --suite humaneval -v -o results.json
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import time
+
+from benchmarks.suites.base import BenchmarkSuite, SuiteReport
+
+# Import all suites
+from benchmarks.suites.humaneval import HumanEvalBenchmark
+from benchmarks.suites.mbpp import MBPPBenchmark
+from benchmarks.suites.swe_bench import SWEBenchBenchmark
+from benchmarks.suites.aider import AiderBenchmark
+from benchmarks.suites.livecodebench import LiveCodeBenchBenchmark
+from benchmarks.suites.math_bench import MATHBenchmark
+from benchmarks.suites.gsm8k import GSM8KBenchmark
+from benchmarks.suites.aime import AIMEBenchmark
+from benchmarks.suites.ifeval import IFEvalBenchmark
+from benchmarks.suites.bfcl import BFCLBenchmark
+
+
+# ---------------------------------------------------------------------------
+# Registry
+# ---------------------------------------------------------------------------
+
+SUITE_REGISTRY: dict[str, type[BenchmarkSuite]] = {
+    "humaneval": HumanEvalBenchmark,
+    "mbpp": MBPPBenchmark,
+    "swe-bench": SWEBenchBenchmark,
+    "aider": AiderBenchmark,
+    "livecodebench": LiveCodeBenchBenchmark,
+    "math": MATHBenchmark,
+    "gsm8k": GSM8KBenchmark,
+    "aime": AIMEBenchmark,
+    "ifeval": IFEvalBenchmark,
+    "bfcl": BFCLBenchmark,
+}
+
+CATEGORY_MAP: dict[str, list[str]] = {
+    "coding": ["humaneval", "mbpp", "swe-bench", "aider", "livecodebench"],
+    "math": ["math", "gsm8k", "aime"],
+    "instruction-following": ["ifeval", "bfcl"],
+}
+
+
+# ---------------------------------------------------------------------------
+# Multi-suite reporting
+# ---------------------------------------------------------------------------
+
+def print_combined_report(reports: list[SuiteReport]) -> None:
+    """Print a combined summary of all suite runs."""
+    print()
+    print("=" * 80)
+    print("  COMBINED BENCHMARK REPORT")
+    print("=" * 80)
+    print()
+    print(f"  {'Suite':<20} {'Category':<22} {'Passed':>8} {'Total':>8} {'Score':>8} {'Time':>8}")
+    print(f"  {'─' * 20} {'─' * 22} {'─' * 8} {'─' * 8} {'─' * 8} {'─' * 8}")
+
+    total_passed = 0
+    total_problems = 0
+    total_time = 0.0
+
+    for r in reports:
+        suite_cls = SUITE_REGISTRY.get(r.suite_name.lower().replace("-", "").replace("_", ""))
+        cat = suite_cls.category if suite_cls else "unknown"
+        print(
+            f"  {r.suite_name:<20} {cat:<22} "
+            f"{r.passed:>8} {r.total:>8} "
+            f"{r.score_pct:>7.1f}% {r.duration_sec:>7.1f}s"
+        )
+        total_passed += r.passed
+        total_problems += r.total
+        total_time += r.duration_sec
+
+    print()
+    print("─" * 80)
+    overall_pct = round(100.0 * total_passed / total_problems, 1) if total_problems else 0.0
+    print(
+        f"  OVERALL: {total_passed}/{total_problems} ({overall_pct}%) "
+        f"in {total_time:.1f}s"
+    )
+    print("─" * 80)
+
+    # Category breakdown
+    print()
+    print("  Category Breakdown:")
+    for cat_name, suite_names in CATEGORY_MAP.items():
+        cat_reports = [r for r in reports if r.suite_name.lower().replace("-", "").replace("_", "") in
+                       [s.replace("-", "").replace("_", "") for s in suite_names]]
+        if cat_reports:
+            cp = sum(r.passed for r in cat_reports)
+            ct = sum(r.total for r in cat_reports)
+            cpct = round(100.0 * cp / ct, 1) if ct else 0.0
+            bar_len = 20
+            filled = round(bar_len * cp / ct) if ct else 0
+            bar = "█" * filled + "░" * (bar_len - filled)
+            print(f"    {cat_name:<22} {bar}  {cp}/{ct} ({cpct}%)")
+
+    print()
+
+
+def save_combined_report(reports: list[SuiteReport], path: str) -> None:
+    data = {
+        "benchmark_run": "claw-code-agent-evaluation",
+        "timestamp": time.strftime("%Y-%m-%dT%H:%M:%S"),
+        "suites": [r.to_dict() for r in reports],
+        "summary": {
+            "total_suites": len(reports),
+            "total_problems": sum(r.total for r in reports),
+            "total_passed": sum(r.passed for r in reports),
+            "overall_score_pct": round(
+                100.0 * sum(r.passed for r in reports) / sum(r.total for r in reports), 1
+            ) if any(r.total for r in reports) else 0.0,
+        },
+    }
+    with open(path, "w") as fh:
+        json.dump(data, fh, indent=2)
+    print(f"  Combined results saved to {path}")
+
+
+# ---------------------------------------------------------------------------
+# CLI
+# ---------------------------------------------------------------------------
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Run standard evaluation benchmarks against claw-code-agent",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog=(
+            "Examples:\n"
+            "  python3 -m benchmarks.run_suite --list\n"
+            "  python3 -m benchmarks.run_suite --suite humaneval\n"
+            "  python3 -m benchmarks.run_suite --suite humaneval --limit 5 -v\n"
+            "  python3 -m benchmarks.run_suite --category coding\n"
+            "  python3 -m benchmarks.run_suite --all\n"
+            "  python3 -m benchmarks.run_suite --all -o results.json\n"
+        ),
+    )
+    parser.add_argument("--suite", choices=list(SUITE_REGISTRY.keys()),
+                        help="Run a specific benchmark suite")
+    parser.add_argument("--category", choices=list(CATEGORY_MAP.keys()),
+                        help="Run all suites in a category")
+    parser.add_argument("--all", action="store_true",
+                        help="Run ALL benchmark suites")
+    parser.add_argument("--list", action="store_true",
+                        help="List available benchmark suites")
+    parser.add_argument("--limit", type=int, default=None,
+                        help="Max problems per suite (for quick testing)")
+    parser.add_argument("--timeout", type=float, default=300.0,
+                        help="Agent timeout per problem in seconds (default: 300)")
+    parser.add_argument("--verbose", "-v", action="store_true",
+                        help="Verbose output")
+    parser.add_argument("--output", "-o",
+                        help="Save results to JSON file")
+    parser.add_argument("--data-dir",
+                        help="Directory containing dataset files (JSONL)")
+    args = parser.parse_args()
+
+    if args.list:
+        print()
+        print("  Available Benchmark Suites:")
+        print("  " + "─" * 68)
+        print(f"  {'Name':<18} {'Category':<22} {'Description'}")
+        print("  " + "─" * 68)
+        for name, cls in SUITE_REGISTRY.items():
+            print(f"  {name:<18} {cls.category:<22} {cls.description}")
+        print()
+        print("  Categories:")
+        for cat, suites in CATEGORY_MAP.items():
+            print(f"    {cat}: {', '.join(suites)}")
+        print()
+        return
+
+    # Determine which suites to run
+    suite_names: list[str] = []
+    if args.all:
+        suite_names = list(SUITE_REGISTRY.keys())
+    elif args.category:
+        suite_names = CATEGORY_MAP.get(args.category, [])
+    elif args.suite:
+        suite_names = [args.suite]
+    else:
+        parser.print_help()
+        print("\nError: specify --suite, --category, or --all")
+        sys.exit(1)
+
+    if not suite_names:
+        print("No suites matched.")
+        sys.exit(1)
+
+    # Run suites
+    reports: list[SuiteReport] = []
+    for name in suite_names:
+        cls = SUITE_REGISTRY[name]
+        suite = cls(
+            data_dir=args.data_dir,
+            limit=args.limit,
+            agent_timeout=args.timeout,
+            verbose=args.verbose,
+        )
+        report = suite.run_all()
+        reports.append(report)
+
+    # Combined report
+    if len(reports) > 1:
+        print_combined_report(reports)
+
+    if args.output:
+        save_combined_report(reports, args.output)
+
+
+if __name__ == "__main__":
+    main()

benchmarks/suites/__init__.py

@@ -0,0 +1 @@
+"""Standard evaluation benchmark suites for claw-code-agent."""

benchmarks/suites/aider.py

@@ -0,0 +1,300 @@
+"""
+Aider benchmark suite.
+
+Aider's code editing benchmark measures how well an agent can apply
+specified edits to existing codebases — refactoring, adding features,
+and fixing bugs based on natural language instructions.
+
+Reference: https://aider.chat/docs/leaderboards/
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import sys
+import textwrap
+from pathlib import Path
+from typing import Any
+
+from .base import BenchmarkResult, BenchmarkSuite
+
+# ---------------------------------------------------------------------------
+# Built-in mini dataset — Aider-style edit tasks
+# ---------------------------------------------------------------------------
+
+_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
+    {
+        "id": "aider-001",
+        "instruction": "Add a `__len__` method to the `TaskList` class that returns the number of tasks.",
+        "setup_code": textwrap.dedent("""\
+            cat > tasks.py << 'PYEOF'
+            class TaskList:
+                def __init__(self):
+                    self._tasks = []
+
+                def add(self, task):
+                    self._tasks.append(task)
+
+                def get_all(self):
+                    return list(self._tasks)
+            PYEOF
+        """),
+        "test_code": textwrap.dedent("""\
+            from tasks import TaskList
+            t = TaskList()
+            assert len(t) == 0
+            t.add("task1")
+            t.add("task2")
+            assert len(t) == 2
+            # Ensure original methods still work
+            assert t.get_all() == ["task1", "task2"]
+            print("ALL_TESTS_PASSED")
+        """),
+    },
+    {
+        "id": "aider-002",
+        "instruction": (
+            "Refactor the `process_data` function to use a list comprehension "
+            "instead of the for loop. The behavior must remain identical."
+        ),
+        "setup_code": textwrap.dedent("""\
+            cat > processor.py << 'PYEOF'
+            def process_data(items):
+                result = []
+                for item in items:
+                    if item > 0:
+                        result.append(item * 2)
+                return result
+            PYEOF
+        """),
+        "test_code": textwrap.dedent("""\
+            import inspect
+            from processor import process_data
+            assert process_data([1, -2, 3, -4, 5]) == [2, 6, 10]
+            assert process_data([]) == []
+            assert process_data([-1, -2]) == []
+            # Check that a list comprehension is used
+            src = inspect.getsource(process_data)
+            assert 'for' in src and '[' in src, "Should use list comprehension"
+            assert src.count('for') <= 2, "Should not have a separate for loop"
+            print("ALL_TESTS_PASSED")
+        """),
+    },
+    {
+        "id": "aider-003",
+        "instruction": (
+            "Add error handling to the `divide` function so it raises a "
+            "`ValueError` with message 'Division by zero' when b is 0, "
+            "instead of letting ZeroDivisionError propagate."
+        ),
+        "setup_code": textwrap.dedent("""\
+            cat > mathops.py << 'PYEOF'
+            def divide(a, b):
+                return a / b
+            PYEOF
+        """),
+        "test_code": textwrap.dedent("""\
+            from mathops import divide
+            assert divide(10, 2) == 5.0
+            assert divide(7, 2) == 3.5
+            try:
+                divide(1, 0)
+                assert False, "Should have raised ValueError"
+            except ValueError as e:
+                assert str(e) == 'Division by zero'
+            except ZeroDivisionError:
+                assert False, "Should raise ValueError, not ZeroDivisionError"
+            print("ALL_TESTS_PASSED")
+        """),
+    },
+    {
+        "id": "aider-004",
+        "instruction": (
+            "Add a `to_dict` method to the `Config` class that returns a dictionary "
+            "of all configuration key-value pairs."
+        ),
+        "setup_code": textwrap.dedent("""\
+            cat > config.py << 'PYEOF'
+            class Config:
+                def __init__(self):
+                    self._data = {}
+
+                def set(self, key, value):
+                    self._data[key] = value
+
+                def get(self, key, default=None):
+                    return self._data.get(key, default)
+            PYEOF
+        """),
+        "test_code": textwrap.dedent("""\
+            from config import Config
+            c = Config()
+            assert c.to_dict() == {}
+            c.set('host', 'localhost')
+            c.set('port', 8080)
+            d = c.to_dict()
+            assert d == {'host': 'localhost', 'port': 8080}
+            # Ensure it's a copy, not a reference
+            d['new'] = 'value'
+            assert 'new' not in c.to_dict()
+            print("ALL_TESTS_PASSED")
+        """),
+    },
+    {
+        "id": "aider-005",
+        "instruction": (
+            "Add a `reverse` method to the `LinkedList` class that reverses the "
+            "list in-place."
+        ),
+        "setup_code": textwrap.dedent("""\
+            cat > linkedlist.py << 'PYEOF'
+            class Node:
+                def __init__(self, val, next=None):
+                    self.val = val
+                    self.next = next
+
+            class LinkedList:
+                def __init__(self):
+                    self.head = None
+
+                def append(self, val):
+                    if not self.head:
+                        self.head = Node(val)
+                        return
+                    curr = self.head
+                    while curr.next:
+                        curr = curr.next
+                    curr.next = Node(val)
+
+                def to_list(self):
+                    result = []
+                    curr = self.head
+                    while curr:
+                        result.append(curr.val)
+                        curr = curr.next
+                    return result
+            PYEOF
+        """),
+        "test_code": textwrap.dedent("""\
+            from linkedlist import LinkedList
+            ll = LinkedList()
+            ll.append(1)
+            ll.append(2)
+            ll.append(3)
+            ll.reverse()
+            assert ll.to_list() == [3, 2, 1]
+            # Test single element
+            ll2 = LinkedList()
+            ll2.append(42)
+            ll2.reverse()
+            assert ll2.to_list() == [42]
+            # Test empty
+            ll3 = LinkedList()
+            ll3.reverse()
+            assert ll3.to_list() == []
+            print("ALL_TESTS_PASSED")
+        """),
+    },
+    {
+        "id": "aider-006",
+        "instruction": (
+            "Convert the `UserStore` class to use a context manager pattern. "
+            "Add `__enter__` and `__exit__` methods. `__enter__` should return self, "
+            "and `__exit__` should call the existing `close` method."
+        ),
+        "setup_code": textwrap.dedent("""\
+            cat > userstore.py << 'PYEOF'
+            class UserStore:
+                def __init__(self):
+                    self._users = {}
+                    self._closed = False
+
+                def add(self, name, email):
+                    self._users[name] = email
+
+                def get(self, name):
+                    return self._users.get(name)
+
+                def close(self):
+                    self._closed = True
+
+                @property
+                def is_closed(self):
+                    return self._closed
+            PYEOF
+        """),
+        "test_code": textwrap.dedent("""\
+            from userstore import UserStore
+            with UserStore() as store:
+                store.add("alice", "alice@example.com")
+                assert store.get("alice") == "alice@example.com"
+                assert not store.is_closed
+            assert store.is_closed
+            # Test exception in with block still calls close
+            try:
+                with UserStore() as store2:
+                    store2.add("bob", "bob@example.com")
+                    raise ValueError("test error")
+            except ValueError:
+                pass
+            assert store2.is_closed
+            print("ALL_TESTS_PASSED")
+        """),
+    },
+]
+
+
+class AiderBenchmark(BenchmarkSuite):
+    """Aider: code editing benchmark."""
+
+    name = "Aider"
+    description = "Code editing and refactoring tasks"
+    category = "coding"
+
+    def load_dataset(self) -> list[dict[str, Any]]:
+        jsonl_path = Path(self.data_dir) / "aider.jsonl"
+        if jsonl_path.exists():
+            problems: list[dict[str, Any]] = []
+            with open(jsonl_path) as fh:
+                for line in fh:
+                    line = line.strip()
+                    if line:
+                        problems.append(json.loads(line))
+            if self.verbose:
+                print(f"  Loaded {len(problems)} problems from {jsonl_path}")
+            return problems
+
+        if self.verbose:
+            print(f"  {jsonl_path} not found — using built-in 6-problem subset")
+        return list(_BUILTIN_PROBLEMS)
+
+    def build_prompt(self, problem: dict[str, Any]) -> str:
+        return (
+            f"Edit the code in the workspace to satisfy this requirement:\n\n"
+            f"{problem['instruction']}\n\n"
+            f"Read the existing file(s), make the minimal changes needed, "
+            f"and save the updated file(s)."
+        )
+
+    def setup_workspace(self, problem: dict[str, Any], workspace: str) -> None:
+        setup = problem.get("setup_code", "")
+        if setup:
+            self._run_shell(setup, cwd=workspace, timeout=30.0)
+
+        # Write test file
+        test_code = problem.get("test_code", "")
+        if test_code:
+            with open(os.path.join(workspace, "test_harness.py"), "w") as fh:
+                fh.write(test_code)
+
+    def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
+        pid = problem.get("id", "unknown")
+        code, output = self._run_shell(
+            f"{sys.executable} test_harness.py", cwd=workspace, timeout=30.0
+        )
+        passed = code == 0 and "ALL_TESTS_PASSED" in output
+        return BenchmarkResult(
+            problem_id=pid, passed=passed, actual=output[:500],
+            error="" if passed else output[:500],
+        )

benchmarks/suites/aime.py

@@ -0,0 +1,140 @@
+"""
+AIME benchmark suite.
+
+AIME (American Invitational Mathematics Examination) problems are
+challenging high-school competition problems.  All answers are integers
+from 000 to 999.
+
+Reference: https://artofproblemsolving.com/wiki/index.php/AIME
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import re
+from pathlib import Path
+from typing import Any
+
+from .base import BenchmarkResult, BenchmarkSuite
+
+# ---------------------------------------------------------------------------
+# Built-in mini dataset (10 representative AIME-style problems)
+# ---------------------------------------------------------------------------
+
+_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
+    {
+        "id": "aime-001",
+        "problem": "Find the sum of all positive integers $n$ such that $n^2 - 19n + 99$ is a perfect square.",
+        "answer": "38",
+    },
+    {
+        "id": "aime-002",
+        "problem": "The number $2^{1993} + 3^{1993}$ is divisible by 5. What is the units digit of the quotient $\\frac{2^{1993}+3^{1993}}{5}$?",
+        "answer": "3",
+    },
+    {
+        "id": "aime-003",
+        "problem": "Let $S$ be the set of integers between 1 and $2^{40}$ whose binary expansions have exactly two 1's. If a number is chosen at random from $S$, what is the probability that it is divisible by 9? Express as $p/q$ where $p$ and $q$ are coprime, and find $p+q$.",
+        "answer": "913",
+    },
+    {
+        "id": "aime-004",
+        "problem": "What is the largest prime factor of $1{,}000{,}027$?",
+        "answer": "103",
+    },
+    {
+        "id": "aime-005",
+        "problem": "Compute the remainder when $3^{1000}$ is divided by $1000$.",
+        "answer": "1",
+    },
+    {
+        "id": "aime-006",
+        "problem": "How many four-digit numbers have the property that the digits sum to 10?",
+        "answer": "219",
+    },
+    {
+        "id": "aime-007",
+        "problem": "Find the number of ordered triples $(a, b, c)$ of positive integers satisfying $a + b + c = 20$ and $a \\leq b \\leq c$.",
+        "answer": "33",
+    },
+    {
+        "id": "aime-008",
+        "problem": "A fair coin is tossed 10 times. What is the probability that no two consecutive tosses are both heads? If the probability is $\\frac{m}{n}$ in lowest terms, find $m + n$.",
+        "answer": "73",
+    },
+    {
+        "id": "aime-009",
+        "problem": "Let $f(n) = n^2 + n + 1$. Find the remainder when $f(1) \\cdot f(2) \\cdot f(3) \\cdots f(10)$ is divided by 100.",
+        "answer": "10",
+    },
+    {
+        "id": "aime-010",
+        "problem": "What is the last three digits of $7^{999}$?",
+        "answer": "343",
+    },
+]
+
+
+class AIMEBenchmark(BenchmarkSuite):
+    """AIME: American Invitational Mathematics Examination problems."""
+
+    name = "AIME"
+    description = "Challenging competition math problems (answers are integers 000–999)"
+    category = "math"
+
+    def load_dataset(self) -> list[dict[str, Any]]:
+        jsonl_path = Path(self.data_dir) / "aime.jsonl"
+        if jsonl_path.exists():
+            problems: list[dict[str, Any]] = []
+            with open(jsonl_path) as fh:
+                for line in fh:
+                    line = line.strip()
+                    if line:
+                        problems.append(json.loads(line))
+            if self.verbose:
+                print(f"  Loaded {len(problems)} problems from {jsonl_path}")
+            return problems
+
+        if self.verbose:
+            print(f"  {jsonl_path} not found — using built-in 10-problem subset")
+        return list(_BUILTIN_PROBLEMS)
+
+    def build_prompt(self, problem: dict[str, Any]) -> str:
+        prob = problem["problem"]
+        return (
+            f"Solve the following AIME-style competition math problem.\n\n"
+            f"Problem: {prob}\n\n"
+            f"The answer is an integer between 0 and 999. "
+            f"Write ONLY the integer answer to a file called answer.txt — "
+            f"no explanation, no work, just the number."
+        )
+
+    def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
+        pid = problem.get("id", "unknown")
+        expected = str(problem["answer"]).strip()
+        answer_file = os.path.join(workspace, "answer.txt")
+
+        if not os.path.exists(answer_file):
+            return BenchmarkResult(
+                problem_id=pid, passed=False, expected=expected,
+                error="answer.txt not found",
+            )
+
+        with open(answer_file) as fh:
+            actual_raw = fh.read().strip()
+
+        # Extract last integer from the answer
+        numbers = re.findall(r"-?\d+", actual_raw.replace(",", ""))
+        actual = numbers[-1] if numbers else actual_raw
+
+        try:
+            passed = int(actual) == int(expected)
+        except (ValueError, TypeError):
+            passed = actual == expected
+
+        return BenchmarkResult(
+            problem_id=pid, passed=passed,
+            expected=expected, actual=actual_raw,
+            error="" if passed else f"expected={expected}, got={actual_raw}",
+        )

benchmarks/suites/base.py

@@ -0,0 +1,273 @@
+"""
+Base class for all benchmark suites.
+
+Every suite implements:
+  - load_dataset()   -> download/load the evaluation dataset
+  - run_single()     -> run the agent on one problem
+  - evaluate()       -> score agent output for one problem
+  - run_all()        -> orchestrate the full benchmark
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+from abc import ABC, abstractmethod
+from dataclasses import asdict, dataclass, field
+from pathlib import Path
+from typing import Any
+
+
+@dataclass
+class BenchmarkResult:
+    """Result for a single problem in a benchmark suite."""
+
+    problem_id: str
+    passed: bool
+    expected: str = ""
+    actual: str = ""
+    duration_sec: float = 0.0
+    error: str = ""
+    metadata: dict[str, Any] = field(default_factory=dict)
+
+
+@dataclass
+class SuiteReport:
+    """Aggregated report for a full benchmark suite run."""
+
+    suite_name: str
+    total: int
+    passed: int
+    failed: int
+    score_pct: float
+    duration_sec: float
+    model: str
+    results: list[BenchmarkResult]
+    timestamp: str = ""
+
+    def to_dict(self) -> dict[str, Any]:
+        return {
+            "suite_name": self.suite_name,
+            "total": self.total,
+            "passed": self.passed,
+            "failed": self.failed,
+            "score_pct": self.score_pct,
+            "duration_sec": round(self.duration_sec, 2),
+            "model": self.model,
+            "timestamp": self.timestamp,
+            "results": [asdict(r) for r in self.results],
+        }
+
+
+class BenchmarkSuite(ABC):
+    """Abstract base for a benchmark suite."""
+
+    # Subclasses set these
+    name: str = "base"
+    description: str = ""
+    category: str = "general"  # coding | math | instruction-following
+
+    def __init__(
+        self,
+        *,
+        data_dir: str | None = None,
+        limit: int | None = None,
+        agent_timeout: float = 300.0,
+        verbose: bool = False,
+    ) -> None:
+        self.data_dir = data_dir or str(
+            Path(__file__).resolve().parent.parent / "data"
+        )
+        self.limit = limit
+        self.agent_timeout = agent_timeout
+        self.verbose = verbose
+        self.project_root = str(Path(__file__).resolve().parent.parent.parent)
+
+    # ------------------------------------------------------------------
+    # Abstract interface
+    # ------------------------------------------------------------------
+
+    @abstractmethod
+    def load_dataset(self) -> list[dict[str, Any]]:
+        """Return a list of problem dicts. Each must have at least an 'id' key."""
+        ...
+
+    @abstractmethod
+    def build_prompt(self, problem: dict[str, Any]) -> str:
+        """Convert a problem dict into the instruction string sent to the agent."""
+        ...
+
+    @abstractmethod
+    def evaluate(
+        self, problem: dict[str, Any], workspace: str
+    ) -> BenchmarkResult:
+        """Score the agent's output for one problem.  Return a BenchmarkResult."""
+        ...
+
+    # ------------------------------------------------------------------
+    # Agent execution helpers
+    # ------------------------------------------------------------------
+
+    def _run_shell(
+        self, cmd: str, cwd: str, timeout: float = 30.0
+    ) -> tuple[int, str]:
+        """Run a shell command, return (exit_code, combined_output)."""
+        try:
+            proc = subprocess.run(
+                cmd,
+                shell=True,
+                cwd=cwd,
+                capture_output=True,
+                text=True,
+                timeout=timeout,
+            )
+            return proc.returncode, (proc.stdout + proc.stderr).strip()
+        except subprocess.TimeoutExpired:
+            return 1, f"[TIMEOUT after {timeout}s]"
+        except Exception as exc:
+            return 1, str(exc)
+
+    def run_agent(self, instruction: str, workspace: str) -> tuple[int, str, float]:
+        """Run the claw-code-agent on *instruction* inside *workspace*.
+
+        Returns (exit_code, output, duration_sec).
+        """
+        import shlex
+
+        agent_cmd = (
+            f"{sys.executable} -m src.main agent "
+            f"{shlex.quote(instruction)} "
+            f"--cwd {shlex.quote(workspace)} "
+            f"--allow-write "
+            f"--allow-shell"
+        )
+        if self.verbose:
+            print(f"  agent cmd: {agent_cmd[:120]}...")
+
+        start = time.time()
+        code, output = self._run_shell(
+            agent_cmd, cwd=self.project_root, timeout=self.agent_timeout
+        )
+        duration = time.time() - start
+
+        if self.verbose:
+            print(f"  agent exit={code}  duration={duration:.1f}s")
+
+        return code, output, duration
+
+    # ------------------------------------------------------------------
+    # Orchestration
+    # ------------------------------------------------------------------
+
+    def run_all(self) -> SuiteReport:
+        """Run the full benchmark suite and return a SuiteReport."""
+        problems = self.load_dataset()
+        if self.limit is not None:
+            problems = problems[: self.limit]
+
+        print()
+        print("=" * 72)
+        print(f"  {self.name} BENCHMARK")
+        print(f"  {self.description}")
+        print("=" * 72)
+        model = os.environ.get("OPENAI_MODEL", "unknown")
+        print(f"  Model:    {model}")
+        print(f"  Problems: {len(problems)}")
+        print(f"  Timeout:  {self.agent_timeout}s per problem")
+        print("=" * 72)
+        print()
+
+        all_results: list[BenchmarkResult] = []
+        suite_start = time.time()
+
+        for i, problem in enumerate(problems, 1):
+            pid = problem.get("id", problem.get("task_id", f"problem-{i}"))
+            print(f"[{i}/{len(problems)}] {pid}")
+
+            workspace = tempfile.mkdtemp(prefix=f"claw_{self.name}_{pid}_")
+            try:
+                # Prepare workspace
+                self.setup_workspace(problem, workspace)
+
+                # Build prompt and run agent
+                prompt = self.build_prompt(problem)
+                _code, _output, duration = self.run_agent(prompt, workspace)
+
+                # Evaluate
+                result = self.evaluate(problem, workspace)
+                result.duration_sec = duration
+
+                status = "PASS ✅" if result.passed else "FAIL ❌"
+                print(f"  -> {status}  ({duration:.1f}s)")
+            except Exception as exc:
+                result = BenchmarkResult(
+                    problem_id=pid,
+                    passed=False,
+                    error=str(exc),
+                )
+                print(f"  -> ERROR ❌  {exc}")
+            finally:
+                shutil.rmtree(workspace, ignore_errors=True)
+
+            all_results.append(result)
+            print()
+
+        suite_duration = time.time() - suite_start
+        passed = sum(1 for r in all_results if r.passed)
+        total = len(all_results)
+
+        report = SuiteReport(
+            suite_name=self.name,
+            total=total,
+            passed=passed,
+            failed=total - passed,
+            score_pct=round(100.0 * passed / total, 1) if total else 0.0,
+            duration_sec=suite_duration,
+            model=model,
+            results=all_results,
+            timestamp=time.strftime("%Y-%m-%dT%H:%M:%S"),
+        )
+
+        self._print_report(report)
+        return report
+
+    def setup_workspace(
+        self, problem: dict[str, Any], workspace: str
+    ) -> None:
+        """Optional: prepare files in workspace before the agent runs."""
+
+    # ------------------------------------------------------------------
+    # Reporting
+    # ------------------------------------------------------------------
+
+    @staticmethod
+    def _print_report(report: SuiteReport) -> None:
+        print()
+        print("=" * 72)
+        print(f"  {report.suite_name} — RESULTS")
+        print("=" * 72)
+        print()
+        for r in report.results:
+            icon = "✅" if r.passed else "❌"
+            print(f"  {icon} {r.problem_id:<40} {r.duration_sec:.1f}s")
+        print()
+        print("─" * 72)
+        print(
+            f"  Total: {report.total}  |  Passed: {report.passed}  "
+            f"|  Failed: {report.failed}  |  Score: {report.score_pct:.1f}%"
+        )
+        print(f"  Total time: {report.duration_sec:.1f}s")
+        print("─" * 72)
+        print()
+
+    @staticmethod
+    def save_report(report: SuiteReport, path: str) -> None:
+        """Persist a SuiteReport to JSON."""
+        with open(path, "w") as fh:
+            json.dump(report.to_dict(), fh, indent=2)
+        print(f"  Report saved to {path}")

benchmarks/suites/bfcl.py

@@ -0,0 +1,340 @@
+"""
+BFCL (Berkeley Function Calling Leaderboard) benchmark suite.
+
+BFCL tests whether an agent can correctly identify and call functions/tools
+based on natural language instructions. This is directly relevant for
+coding agents that use tool calling.
+
+Reference: https://gorilla.cs.berkeley.edu/leaderboard.html
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import re
+import sys
+from pathlib import Path
+from typing import Any
+
+from .base import BenchmarkResult, BenchmarkSuite
+
+# ---------------------------------------------------------------------------
+# Built-in mini dataset — function-calling tasks
+# ---------------------------------------------------------------------------
+
+_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
+    {
+        "id": "bfcl-001",
+        "instruction": (
+            "You have access to a function `get_weather(city: str, unit: str = 'celsius') -> dict` "
+            "that returns weather info. Write a Python script called solution.py that calls "
+            "get_weather for 'San Francisco' with unit 'fahrenheit' and prints the result."
+        ),
+        "expected_function": "get_weather",
+        "expected_args": {"city": "San Francisco", "unit": "fahrenheit"},
+        "setup_code": (
+            "cat > weather_api.py << 'PYEOF'\n"
+            "def get_weather(city: str, unit: str = 'celsius') -> dict:\n"
+            "    return {'city': city, 'unit': unit, 'temp': 72 if unit == 'fahrenheit' else 22}\n"
+            "PYEOF"
+        ),
+        "test_code": (
+            "import sys\n"
+            "sys.path.insert(0, '.')\n"
+            "# Check that solution.py exists and calls get_weather correctly\n"
+            "with open('solution.py') as f:\n"
+            "    code = f.read()\n"
+            "assert 'get_weather' in code, 'Must call get_weather'\n"
+            "assert 'San Francisco' in code, 'Must use San Francisco'\n"
+            "assert 'fahrenheit' in code, 'Must use fahrenheit'\n"
+            "exec(open('solution.py').read())\n"
+            "print('ALL_TESTS_PASSED')\n"
+        ),
+    },
+    {
+        "id": "bfcl-002",
+        "instruction": (
+            "You have access to these functions:\n"
+            "- `search_files(directory: str, pattern: str) -> list[str]`\n"
+            "- `read_file(path: str) -> str`\n"
+            "- `write_file(path: str, content: str) -> None`\n\n"
+            "Write a Python script called solution.py that:\n"
+            "1. Calls search_files('.', '*.txt') to find all text files\n"
+            "2. Reads each file using read_file\n"
+            "3. Writes a combined output using write_file to 'combined.txt'"
+        ),
+        "expected_function": "search_files",
+        "expected_args": {"directory": ".", "pattern": "*.txt"},
+        "setup_code": (
+            "cat > file_api.py << 'PYEOF'\n"
+            "import glob\n"
+            "def search_files(directory: str, pattern: str) -> list:\n"
+            "    import os\n"
+            "    return [os.path.join(directory, f) for f in os.listdir(directory) if f.endswith('.txt') and f != 'combined.txt']\n"
+            "def read_file(path: str) -> str:\n"
+            "    with open(path) as f:\n"
+            "        return f.read()\n"
+            "def write_file(path: str, content: str) -> None:\n"
+            "    with open(path, 'w') as f:\n"
+            "        f.write(content)\n"
+            "PYEOF\n"
+            "echo 'hello' > a.txt\n"
+            "echo 'world' > b.txt\n"
+        ),
+        "test_code": (
+            "import sys\n"
+            "sys.path.insert(0, '.')\n"
+            "with open('solution.py') as f:\n"
+            "    code = f.read()\n"
+            "assert 'search_files' in code, 'Must call search_files'\n"
+            "assert 'read_file' in code, 'Must call read_file'\n"
+            "assert 'write_file' in code, 'Must call write_file'\n"
+            "exec(open('solution.py').read())\n"
+            "import os\n"
+            "assert os.path.exists('combined.txt'), 'combined.txt must exist'\n"
+            "content = open('combined.txt').read()\n"
+            "assert 'hello' in content and 'world' in content\n"
+            "print('ALL_TESTS_PASSED')\n"
+        ),
+    },
+    {
+        "id": "bfcl-003",
+        "instruction": (
+            "You have a function `calculate(expression: str) -> float` that evaluates "
+            "a mathematical expression string. Write solution.py that:\n"
+            "1. Calculates '(15 + 25) * 3'\n"
+            "2. Calculates '100 / 4 - 5'\n"
+            "3. Prints both results"
+        ),
+        "expected_function": "calculate",
+        "expected_args": {"expression": "(15 + 25) * 3"},
+        "setup_code": (
+            "cat > calc_api.py << 'PYEOF'\n"
+            "def calculate(expression: str) -> float:\n"
+            "    return float(eval(expression))\n"
+            "PYEOF"
+        ),
+        "test_code": (
+            "import sys\n"
+            "sys.path.insert(0, '.')\n"
+            "with open('solution.py') as f:\n"
+            "    code = f.read()\n"
+            "assert 'calculate' in code, 'Must call calculate'\n"
+            "exec(open('solution.py').read())\n"
+            "print('ALL_TESTS_PASSED')\n"
+        ),
+    },
+    {
+        "id": "bfcl-004",
+        "instruction": (
+            "You have these functions:\n"
+            "- `create_user(name: str, email: str, role: str = 'user') -> dict`\n"
+            "- `delete_user(user_id: int) -> bool`\n"
+            "- `list_users() -> list[dict]`\n\n"
+            "Write solution.py that creates a user named 'Alice' with email "
+            "'alice@example.com' and role 'admin', then lists all users."
+        ),
+        "expected_function": "create_user",
+        "expected_args": {"name": "Alice", "email": "alice@example.com", "role": "admin"},
+        "setup_code": (
+            "cat > user_api.py << 'PYEOF'\n"
+            "_users = []\n"
+            "_next_id = 1\n"
+            "def create_user(name: str, email: str, role: str = 'user') -> dict:\n"
+            "    global _next_id\n"
+            "    user = {'id': _next_id, 'name': name, 'email': email, 'role': role}\n"
+            "    _users.append(user)\n"
+            "    _next_id += 1\n"
+            "    return user\n"
+            "def delete_user(user_id: int) -> bool:\n"
+            "    global _users\n"
+            "    _users = [u for u in _users if u['id'] != user_id]\n"
+            "    return True\n"
+            "def list_users() -> list:\n"
+            "    return list(_users)\n"
+            "PYEOF"
+        ),
+        "test_code": (
+            "import sys\n"
+            "sys.path.insert(0, '.')\n"
+            "with open('solution.py') as f:\n"
+            "    code = f.read()\n"
+            "assert 'create_user' in code, 'Must call create_user'\n"
+            "assert 'Alice' in code, 'Must use Alice'\n"
+            "assert 'alice@example.com' in code, 'Must use correct email'\n"
+            "assert 'admin' in code, 'Must use admin role'\n"
+            "assert 'list_users' in code, 'Must call list_users'\n"
+            "print('ALL_TESTS_PASSED')\n"
+        ),
+    },
+    {
+        "id": "bfcl-005",
+        "instruction": (
+            "You have a function `send_email(to: str, subject: str, body: str, "
+            "cc: list[str] = None) -> bool`. Write solution.py that sends an email "
+            "to 'boss@company.com' with subject 'Weekly Report', body 'Please see attached report.', "
+            "and cc=['team@company.com']."
+        ),
+        "expected_function": "send_email",
+        "expected_args": {
+            "to": "boss@company.com",
+            "subject": "Weekly Report",
+            "body": "Please see attached report.",
+            "cc": ["team@company.com"],
+        },
+        "setup_code": (
+            "cat > email_api.py << 'PYEOF'\n"
+            "def send_email(to: str, subject: str, body: str, cc: list = None) -> bool:\n"
+            "    print(f'Email sent to {to}, subject: {subject}')\n"
+            "    return True\n"
+            "PYEOF"
+        ),
+        "test_code": (
+            "import sys\n"
+            "sys.path.insert(0, '.')\n"
+            "with open('solution.py') as f:\n"
+            "    code = f.read()\n"
+            "assert 'send_email' in code, 'Must call send_email'\n"
+            "assert 'boss@company.com' in code, 'Must use correct recipient'\n"
+            "assert 'Weekly Report' in code, 'Must use correct subject'\n"
+            "assert 'team@company.com' in code, 'Must include cc'\n"
+            "print('ALL_TESTS_PASSED')\n"
+        ),
+    },
+    {
+        "id": "bfcl-006",
+        "instruction": (
+            "You have these functions:\n"
+            "- `connect_db(host: str, port: int, database: str) -> object`\n"
+            "- `execute_query(connection: object, query: str) -> list`\n"
+            "- `close_db(connection: object) -> None`\n\n"
+            "Write solution.py that connects to host='localhost', port=5432, database='mydb', "
+            "executes 'SELECT * FROM users', and closes the connection."
+        ),
+        "expected_function": "connect_db",
+        "expected_args": {"host": "localhost", "port": 5432, "database": "mydb"},
+        "setup_code": (
+            "cat > db_api.py << 'PYEOF'\n"
+            "class FakeConnection:\n"
+            "    def __init__(self, host, port, db):\n"
+            "        self.host = host\n"
+            "        self.port = port\n"
+            "        self.db = db\n"
+            "        self.closed = False\n"
+            "def connect_db(host: str, port: int, database: str) -> object:\n"
+            "    return FakeConnection(host, port, database)\n"
+            "def execute_query(connection: object, query: str) -> list:\n"
+            "    return [{'id': 1, 'name': 'test'}]\n"
+            "def close_db(connection: object) -> None:\n"
+            "    connection.closed = True\n"
+            "PYEOF"
+        ),
+        "test_code": (
+            "import sys\n"
+            "sys.path.insert(0, '.')\n"
+            "with open('solution.py') as f:\n"
+            "    code = f.read()\n"
+            "assert 'connect_db' in code, 'Must call connect_db'\n"
+            "assert 'execute_query' in code, 'Must call execute_query'\n"
+            "assert 'close_db' in code, 'Must call close_db'\n"
+            "assert 'localhost' in code, 'Must use localhost'\n"
+            "assert '5432' in code, 'Must use port 5432'\n"
+            "assert 'mydb' in code, 'Must use mydb database'\n"
+            "print('ALL_TESTS_PASSED')\n"
+        ),
+    },
+    {
+        "id": "bfcl-007",
+        "instruction": (
+            "You have a function `sort_data(data: list, key: str, reverse: bool = False) -> list`. "
+            "Write solution.py that sorts the list [{'name': 'Charlie', 'age': 30}, "
+            "{'name': 'Alice', 'age': 25}, {'name': 'Bob', 'age': 35}] by 'age' in "
+            "descending order (reverse=True) and prints the result."
+        ),
+        "expected_function": "sort_data",
+        "expected_args": {"key": "age", "reverse": True},
+        "setup_code": (
+            "cat > sort_api.py << 'PYEOF'\n"
+            "def sort_data(data: list, key: str, reverse: bool = False) -> list:\n"
+            "    return sorted(data, key=lambda x: x[key], reverse=reverse)\n"
+            "PYEOF"
+        ),
+        "test_code": (
+            "import sys\n"
+            "sys.path.insert(0, '.')\n"
+            "with open('solution.py') as f:\n"
+            "    code = f.read()\n"
+            "assert 'sort_data' in code, 'Must call sort_data'\n"
+            "assert 'age' in code, 'Must sort by age'\n"
+            "assert 'reverse' in code or 'True' in code, 'Must use reverse order'\n"
+            "print('ALL_TESTS_PASSED')\n"
+        ),
+    },
+]
+
+
+class BFCLBenchmark(BenchmarkSuite):
+    """BFCL: Berkeley Function Calling Leaderboard."""
+
+    name = "BFCL"
+    description = "Function/tool calling evaluation"
+    category = "instruction-following"
+
+    def load_dataset(self) -> list[dict[str, Any]]:
+        jsonl_path = Path(self.data_dir) / "bfcl.jsonl"
+        if jsonl_path.exists():
+            problems: list[dict[str, Any]] = []
+            with open(jsonl_path) as fh:
+                for line in fh:
+                    line = line.strip()
+                    if line:
+                        problems.append(json.loads(line))
+            if self.verbose:
+                print(f"  Loaded {len(problems)} problems from {jsonl_path}")
+            return problems
+
+        if self.verbose:
+            print(f"  {jsonl_path} not found — using built-in 7-problem subset")
+        return list(_BUILTIN_PROBLEMS)
+
+    def build_prompt(self, problem: dict[str, Any]) -> str:
+        return problem["instruction"]
+
+    def setup_workspace(self, problem: dict[str, Any], workspace: str) -> None:
+        setup = problem.get("setup_code", "")
+        if setup:
+            self._run_shell(setup, cwd=workspace, timeout=30.0)
+
+        test_code = problem.get("test_code", "")
+        if test_code:
+            with open(os.path.join(workspace, "test_harness.py"), "w") as fh:
+                fh.write(test_code)
+
+    def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
+        pid = problem.get("id", "unknown")
+
+        sol = os.path.join(workspace, "solution.py")
+        if not os.path.exists(sol):
+            return BenchmarkResult(
+                problem_id=pid, passed=False, error="solution.py not found"
+            )
+
+        test_harness = os.path.join(workspace, "test_harness.py")
+        if os.path.exists(test_harness):
+            code, output = self._run_shell(
+                f"{sys.executable} test_harness.py", cwd=workspace, timeout=30.0
+            )
+            passed = code == 0 and "ALL_TESTS_PASSED" in output
+        else:
+            # Basic check: verify the expected function is called
+            with open(sol) as fh:
+                code_content = fh.read()
+            expected_fn = problem.get("expected_function", "")
+            passed = expected_fn in code_content if expected_fn else True
+
+        return BenchmarkResult(
+            problem_id=pid, passed=passed,
+            actual=output[:500] if "output" in dir() else "",
+            error="" if passed else (output[:500] if "output" in dir() else "function call not found"),
+        )

benchmarks/suites/gsm8k.py

@@ -0,0 +1,177 @@
+"""
+GSM8K benchmark suite.
+
+GSM8K (Grade School Math 8K) is a dataset of 8,500 linguistically diverse
+grade-school-level math word problems that require multi-step reasoning.
+
+Paper: https://arxiv.org/abs/2110.14168
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import re
+from pathlib import Path
+from typing import Any
+
+from .base import BenchmarkResult, BenchmarkSuite
+
+# ---------------------------------------------------------------------------
+# Built-in mini dataset (15 representative problems)
+# ---------------------------------------------------------------------------
+
+_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
+    {
+        "id": "gsm8k-001",
+        "question": "Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?",
+        "answer": "18",
+    },
+    {
+        "id": "gsm8k-002",
+        "question": "A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?",
+        "answer": "3",
+    },
+    {
+        "id": "gsm8k-003",
+        "question": "Josh decides to try flipping a house. He buys a house for $80,000 and then puts in $50,000 in repairs. This increased the value of the house by 150%. How much profit did he make?",
+        "answer": "70000",
+    },
+    {
+        "id": "gsm8k-004",
+        "question": "James decides to run 3 sprints 3 times a week. He runs 60 meters each sprint. How many total meters does he run a week?",
+        "answer": "540",
+    },
+    {
+        "id": "gsm8k-005",
+        "question": "Every day, Wendi feeds each of her chickens three cups of mixed chicken feed, containing seeds, mealworms and vegetables to help keep them healthy. She gives the chickens their feed in three separate meals. In the morning, she gives her flock of chickens 15 cups of feed. In the afternoon, she gives her chickens another 25 cups of feed. If Wendi's flock eats a combined total of 15 cups of feed in the final meal of the day, how many chickens does Wendi have?",
+        "answer": "20",
+    },
+    {
+        "id": "gsm8k-006",
+        "question": "Kylar went to the store to get water and some batteries. He spent $10 on the water and three times that on batteries. How much did he spend total?",
+        "answer": "40",
+    },
+    {
+        "id": "gsm8k-007",
+        "question": "Toulouse has twice as many sheep as Charleston. Charleston has 4 times as many sheep as Seattle. How many sheep do Toulouse, Charleston, and Seattle have together if Seattle has 20 sheep?",
+        "answer": "260",
+    },
+    {
+        "id": "gsm8k-008",
+        "question": "Carla is downloading a 200 GB file. Normally she can download 2 GB/minute, but 40% of the way through the download, Windows forces a restart to install updates, which takes 20 minutes. Then Carla has to restart the download from the beginning. How long does it take to download the file?",
+        "answer": "160",
+    },
+    {
+        "id": "gsm8k-009",
+        "question": "John drives for 3 hours at a speed of 60 mph and then turns around because he realizes he forgot something very important at home. He tries to get home in 4 hours but spends the first 2 hours in standstill traffic. He spends the rest of the time driving at a speed of 30 mph. How far is he from home?",
+        "answer": "120",
+    },
+    {
+        "id": "gsm8k-010",
+        "question": "Eliza's rate per hour for the first 40 hours she works each week is $10. She also receives an overtime pay of 1.2 times her regular hourly rate. If Eliza worked for 45 hours this week, how much are her earnings for this week?",
+        "answer": "460",
+    },
+    {
+        "id": "gsm8k-011",
+        "question": "A new program had 60 downloads in the first month. The number of downloads in the second month was three times as many as the downloads in the first month, but then reduced by 30% in the third month. How many downloads did the program have total over the three months?",
+        "answer": "366",
+    },
+    {
+        "id": "gsm8k-012",
+        "question": "Thomas has 6 more apples than friends. He also has twice as many friends as bags. If he has 20 bags, how many apples does he have?",
+        "answer": "46",
+    },
+    {
+        "id": "gsm8k-013",
+        "question": "A store sells pencils in packs of 8. If a teacher needs 120 pencils for her class, how many packs does she need to buy?",
+        "answer": "15",
+    },
+    {
+        "id": "gsm8k-014",
+        "question": "Sam bought a dozen boxes, each with 30 highlighter pens inside, for $10 each box. He rearranged five of these boxes into packages of six highlighters each and sold them for $3 per package. He sold the rest of the highlighters separately at the rate of three pens for $2. How much profit did Sam make in total, in dollars?",
+        "answer": "115",
+    },
+    {
+        "id": "gsm8k-015",
+        "question": "There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?",
+        "answer": "6",
+    },
+]
+
+
+def _extract_number(text: str) -> str | None:
+    """Extract the last number from a text string."""
+    text = text.replace(",", "").replace("$", "").strip()
+    # Find all numbers (including decimals and negatives)
+    numbers = re.findall(r"-?\d+\.?\d*", text)
+    return numbers[-1] if numbers else None
+
+
+class GSM8KBenchmark(BenchmarkSuite):
+    """GSM8K: grade school math word problems."""
+
+    name = "GSM8K"
+    description = "Grade school math word problems requiring multi-step reasoning"
+    category = "math"
+
+    def load_dataset(self) -> list[dict[str, Any]]:
+        jsonl_path = Path(self.data_dir) / "gsm8k.jsonl"
+        if jsonl_path.exists():
+            problems: list[dict[str, Any]] = []
+            with open(jsonl_path) as fh:
+                for line in fh:
+                    line = line.strip()
+                    if line:
+                        problems.append(json.loads(line))
+            if self.verbose:
+                print(f"  Loaded {len(problems)} problems from {jsonl_path}")
+            return problems
+
+        if self.verbose:
+            print(f"  {jsonl_path} not found — using built-in 15-problem subset")
+        return list(_BUILTIN_PROBLEMS)
+
+    def build_prompt(self, problem: dict[str, Any]) -> str:
+        question = problem["question"]
+        return (
+            f"Solve the following grade school math problem step by step. "
+            f"Write ONLY the final numerical answer (just the number) to a file called answer.txt.\n\n"
+            f"Problem: {question}\n\n"
+            f"Save just the number to answer.txt — no units, no explanation."
+        )
+
+    def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
+        pid = problem.get("id", "unknown")
+        expected = str(problem["answer"])
+        answer_file = os.path.join(workspace, "answer.txt")
+
+        if not os.path.exists(answer_file):
+            return BenchmarkResult(
+                problem_id=pid, passed=False, expected=expected,
+                error="answer.txt not found",
+            )
+
+        with open(answer_file) as fh:
+            actual_raw = fh.read().strip()
+
+        actual_num = _extract_number(actual_raw)
+        expected_num = _extract_number(expected)
+
+        if actual_num is None:
+            return BenchmarkResult(
+                problem_id=pid, passed=False,
+                expected=expected, actual=actual_raw,
+                error=f"Could not extract number from: {actual_raw}",
+            )
+
+        try:
+            passed = abs(float(actual_num) - float(expected_num)) < 1e-6
+        except (ValueError, TypeError):
+            passed = actual_num == expected_num
+
+        return BenchmarkResult(
+            problem_id=pid, passed=passed,
+            expected=expected, actual=actual_raw,
+            error="" if passed else f"expected={expected}, got={actual_raw}",
+        )

benchmarks/suites/humaneval.py

@@ -0,0 +1,246 @@
+"""
+HumanEval benchmark suite.
+
+OpenAI's HumanEval evaluates code generation from docstrings.
+164 hand-written Python programming problems with function signatures and
+docstrings. The agent must produce a correct implementation.
+
+Dataset: https://github.com/openai/human-eval
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import subprocess
+import sys
+import textwrap
+from pathlib import Path
+from typing import Any
+
+from .base import BenchmarkResult, BenchmarkSuite
+
+# ---------------------------------------------------------------------------
+# Built-in mini dataset (subset of 20 representative problems)
+# ---------------------------------------------------------------------------
+
+_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
+    {
+        "task_id": "HumanEval/0",
+        "prompt": "from typing import List\n\n\ndef has_close_elements(numbers: List[float], threshold: float) -> bool:\n    \"\"\" Check if in given list of numbers, are any two numbers closer to each other than\n    given threshold.\n    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)\n    False\n    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)\n    True\n    \"\"\"\n",
+        "canonical_solution": "    for idx, elem in enumerate(numbers):\n        for idx2, elem2 in enumerate(numbers):\n            if idx != idx2:\n                distance = abs(elem - elem2)\n                if distance < threshold:\n                    return True\n\n    return False\n",
+        "test": "def check(candidate):\n    assert candidate([1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.3) == True\n    assert candidate([1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.05) == False\n    assert candidate([1.0, 2.0, 5.9, 4.0, 5.0], 0.95) == True\n    assert candidate([1.0, 2.0, 5.9, 4.0, 5.0], 0.8) == False\n    assert candidate([1.0, 2.0, 3.0, 4.0, 5.0, 2.0], 0.1) == True\n    assert candidate([1.1, 2.2, 3.1, 4.1, 5.1], 1.0) == True\n    assert candidate([1.1, 2.2, 3.1, 4.1, 5.1], 0.5) == False\n",
+        "entry_point": "has_close_elements",
+    },
+    {
+        "task_id": "HumanEval/1",
+        "prompt": "from typing import List\n\n\ndef separate_paren_groups(paren_string: str) -> List[str]:\n    \"\"\" Input to this function is a string containing multiple groups of nested parentheses. Your goal is to\n    separate those group into separate strings and return the list of those.\n    Separate groups are balanced (each open brace is properly closed) and not nested within each other\n    Ignore any spaces in the input string.\n    >>> separate_paren_groups('( ) (( )) (( )( ))')\n    ['()', '(())', '(()())']\n    \"\"\"\n",
+        "canonical_solution": "    result = []\n    current_string = []\n    current_depth = 0\n\n    for c in paren_string:\n        if c == '(':\n            current_depth += 1\n            current_string.append(c)\n        elif c == ')':\n            current_depth -= 1\n            current_string.append(c)\n\n            if current_depth == 0:\n                result.append(''.join(current_string))\n                current_string.clear()\n\n    return result\n",
+        "test": "def check(candidate):\n    assert candidate('(()()) ((())) () ((())()())') == ['(()())', '((()))', '()', '((())()())']\n    assert candidate('() (()) ((())) (((())))') == ['()', '(())', '((()))', '(((())))']\n    assert candidate('(()(()))') == ['(()(()))']\n    assert candidate('( ) (( )) (( )( ))') == ['()', '(())', '(()())']\n",
+        "entry_point": "separate_paren_groups",
+    },
+    {
+        "task_id": "HumanEval/2",
+        "prompt": "\n\ndef truncate_number(number: float) -> float:\n    \"\"\" Given a positive floating point number, it can be decomposed into\n    and integer part (largest integer smaller than given number) and decimals\n    (leftover part always smaller than 1).\n\n    Return the decimal part of the number.\n    >>> truncate_number(3.5)\n    0.5\n    \"\"\"\n",
+        "canonical_solution": "    return number % 1.0\n",
+        "test": "def check(candidate):\n    assert candidate(3.5) == 0.5\n    assert abs(candidate(1.33) - 0.33) < 1e-6\n    assert abs(candidate(123.456) - 0.456) < 1e-6\n",
+        "entry_point": "truncate_number",
+    },
+    {
+        "task_id": "HumanEval/3",
+        "prompt": "from typing import List\n\n\ndef below_zero(operations: List[int]) -> bool:\n    \"\"\" You're given a list of deposit and withdrawal operations on a bank account that starts with\n    zero balance. Your task is to detect if at any point the balance of account falls below zero, and\n    at that point function should return True. Otherwise it should return False.\n    >>> below_zero([1, 2, 3])\n    False\n    >>> below_zero([1, 2, -4, 5])\n    True\n    \"\"\"\n",
+        "canonical_solution": "    balance = 0\n\n    for op in operations:\n        balance += op\n        if balance < 0:\n            return True\n\n    return False\n",
+        "test": "def check(candidate):\n    assert candidate([]) == False\n    assert candidate([1, 2, -3, 1, 2, -3]) == False\n    assert candidate([1, 2, -4, 5, 6]) == True\n    assert candidate([1, -1, 2, -2, 5, -5, 4, -4]) == False\n    assert candidate([1, -1, 2, -2, 5, -5, 4, -5]) == True\n    assert candidate([1, -2, 2, -2, 5, -5, 4, -4]) == True\n",
+        "entry_point": "below_zero",
+    },
+    {
+        "task_id": "HumanEval/4",
+        "prompt": "from typing import List\n\n\ndef mean_absolute_deviation(numbers: List[float]) -> float:\n    \"\"\" For a given list of input numbers, calculate Mean Absolute Deviation\n    around the mean of this dataset.\n    Mean Absolute Deviation is the average absolute difference between each\n    element and a centerpoint (mean in this case):\n    MAD = average | x - x_mean |\n    >>> mean_absolute_deviation([1.0, 2.0, 3.0, 4.0])\n    1.0\n    \"\"\"\n",
+        "canonical_solution": "    mean = sum(numbers) / len(numbers)\n    return sum(abs(x - mean) for x in numbers) / len(numbers)\n",
+        "test": "def check(candidate):\n    assert abs(candidate([1.0, 2.0, 3.0]) - 2.0/3.0) < 1e-6\n    assert abs(candidate([1.0, 2.0, 3.0, 4.0]) - 1.0) < 1e-6\n    assert abs(candidate([1.0, 2.0, 3.0, 4.0, 5.0]) - 1.2) < 1e-6\n",
+        "entry_point": "mean_absolute_deviation",
+    },
+    {
+        "task_id": "HumanEval/5",
+        "prompt": "from typing import List\n\n\ndef intersperse(numbers: List[int], delimeter: int) -> List[int]:\n    \"\"\" Insert a number 'delimeter' between every two consecutive elements of input list `numbers'\n    >>> intersperse([], 4)\n    []\n    >>> intersperse([1, 2, 3], 4)\n    [1, 4, 2, 4, 3]\n    \"\"\"\n",
+        "canonical_solution": "    if not numbers:\n        return []\n\n    result = []\n\n    for n in numbers[:-1]:\n        result.append(n)\n        result.append(delimeter)\n\n    result.append(numbers[-1])\n\n    return result\n",
+        "test": "def check(candidate):\n    assert candidate([], 7) == []\n    assert candidate([5, 6, 3, 2], 8) == [5, 8, 6, 8, 3, 8, 2]\n    assert candidate([2, 2, 2], 2) == [2, 2, 2, 2, 2]\n",
+        "entry_point": "intersperse",
+    },
+    {
+        "task_id": "HumanEval/6",
+        "prompt": "from typing import List\n\n\ndef parse_nested_parens(paren_string: str) -> List[int]:\n    \"\"\" Input to this function is a string represented multiple groups of nested parentheses separated by spaces.\n    For each of the group, output the deepest level of nesting of parentheses.\n    E.g. (()()) has maximum two levels of nesting while ((())) has three.\n\n    >>> parse_nested_parens('(()()) ((())) () ((())()())')\n    [2, 3, 1, 3]\n    \"\"\"\n",
+        "canonical_solution": "    def parse_paren_group(s):\n        depth = 0\n        max_depth = 0\n        for c in s:\n            if c == '(':\n                depth += 1\n                max_depth = max(depth, max_depth)\n            else:\n                depth -= 1\n\n        return max_depth\n\n    return [parse_paren_group(x) for x in paren_string.split(' ') if x]\n",
+        "test": "def check(candidate):\n    assert candidate('(()()) ((())) () ((())()())') == [2, 3, 1, 3]\n    assert candidate('() (()) ((())) (((())))') == [1, 2, 3, 4]\n    assert candidate('(()(())((())))') == [4]\n",
+        "entry_point": "parse_nested_parens",
+    },
+    {
+        "task_id": "HumanEval/7",
+        "prompt": "from typing import List\n\n\ndef filter_by_substring(strings: List[str], substring: str) -> List[str]:\n    \"\"\" Filter an input list of strings only for ones that contain given substring\n    >>> filter_by_substring([], 'a')\n    []\n    >>> filter_by_substring(['abc', 'bacd', 'cde', 'array'], 'a')\n    ['abc', 'bacd', 'array']\n    \"\"\"\n",
+        "canonical_solution": "    return [x for x in strings if substring in x]\n",
+        "test": "def check(candidate):\n    assert candidate([], 'john') == []\n    assert candidate(['xxx', 'asd', 'xxy', 'john doe', 'xxxuj', 'xxx'], 'xxx') == ['xxx', 'xxxuj', 'xxx']\n    assert candidate(['xxx', 'asd', 'aaber', 'john doe', 'xxxuj', 'xxx'], 'john') == ['john doe']\n    assert candidate(['grunt', 'hierarchies', 'aaaxyz'], 'hi') == ['hierarchies']\n",
+        "entry_point": "filter_by_substring",
+    },
+    {
+        "task_id": "HumanEval/8",
+        "prompt": "from typing import List, Tuple\n\n\ndef sum_product(numbers: List[int]) -> Tuple[int, int]:\n    \"\"\" For a given list of integers, return a tuple consisting of a sum and a product of all the integers in a list.\n    Empty sum should be equal to 0 and empty product should be equal to 1.\n    >>> sum_product([])\n    (0, 1)\n    >>> sum_product([1, 2, 3, 4])\n    (10, 24)\n    \"\"\"\n",
+        "canonical_solution": "    sum_value = 0\n    prod_value = 1\n\n    for n in numbers:\n        sum_value += n\n        prod_value *= n\n\n    return sum_value, prod_value\n",
+        "test": "def check(candidate):\n    assert candidate([]) == (0, 1)\n    assert candidate([1, 1, 1]) == (3, 1)\n    assert candidate([100, 0]) == (100, 0)\n    assert candidate([3, 5, 7]) == (15, 105)\n    assert candidate([10]) == (10, 10)\n",
+        "entry_point": "sum_product",
+    },
+    {
+        "task_id": "HumanEval/9",
+        "prompt": "from typing import List, Tuple\n\n\ndef rolling_max(numbers: List[int]) -> List[int]:\n    \"\"\" From a given list of integers, generate a list of rolling maximum element found until given moment\n    in the sequence.\n    >>> rolling_max([1, 2, 3, 2, 3, 4, 2])\n    [1, 2, 3, 3, 3, 4, 4]\n    \"\"\"\n",
+        "canonical_solution": "    running_max = None\n    result = []\n\n    for n in numbers:\n        if running_max is None:\n            running_max = n\n        else:\n            running_max = max(running_max, n)\n\n        result.append(running_max)\n\n    return result\n",
+        "test": "def check(candidate):\n    assert candidate([]) == []\n    assert candidate([1, 2, 3, 4]) == [1, 2, 3, 4]\n    assert candidate([4, 3, 2, 1]) == [4, 4, 4, 4]\n    assert candidate([3, 3, 3, 3]) == [3, 3, 3, 3]\n",
+        "entry_point": "rolling_max",
+    },
+    {
+        "task_id": "HumanEval/10",
+        "prompt": "\n\ndef is_palindrome(string: str) -> bool:\n    \"\"\" Test if given string is a palindrome \"\"\"\n    return string == string[::-1]\n\n\ndef make_palindrome(string: str) -> str:\n    \"\"\" Find the shortest palindrome that begins with a supplied string.\n    Algorithm idea is simple:\n    - Find the longest postfix of supplied string that is a palindrome.\n    - Append to the end of the string reverse of a string prefix that comes before the palindromic suffix.\n    >>> make_palindrome('')\n    ''\n    >>> make_palindrome('cat')\n    'catac'\n    >>> make_palindrome('cata')\n    'catac'\n    \"\"\"\n",
+        "canonical_solution": "    if not string:\n        return ''\n\n    beginning_of_suffix = 0\n\n    while not is_palindrome(string[beginning_of_suffix:]):\n        beginning_of_suffix += 1\n\n    return string + string[:beginning_of_suffix][::-1]\n",
+        "test": "def check(candidate):\n    assert candidate('') == ''\n    assert candidate('x') == 'x'\n    assert candidate('xyz') == 'xyzyx'\n    assert candidate('xyx') == 'xyx'\n    assert candidate('jerry') == 'jerryrrej'\n",
+        "entry_point": "make_palindrome",
+    },
+    {
+        "task_id": "HumanEval/11",
+        "prompt": "from typing import List\n\n\ndef string_xor(a: str, b: str) -> str:\n    \"\"\" Input are two strings a and b consisting only of 1s and 0s.\n    Perform binary XOR on these inputs and return result also as a string.\n    >>> string_xor('010', '110')\n    '100'\n    \"\"\"\n",
+        "canonical_solution": "    def xor(i, j):\n        if i == j:\n            return '0'\n        else:\n            return '1'\n\n    return ''.join(xor(x, y) for x, y in zip(a, b))\n",
+        "test": "def check(candidate):\n    assert candidate('111000', '101010') == '010010'\n    assert candidate('1', '1') == '0'\n    assert candidate('0101', '0000') == '0101'\n",
+        "entry_point": "string_xor",
+    },
+    {
+        "task_id": "HumanEval/12",
+        "prompt": "from typing import List, Optional\n\n\ndef longest(strings: List[str]) -> Optional[str]:\n    \"\"\" Out of list of strings, return the longest one. Return the first one in case of multiple\n    strings of the same length. Return None in case the input list is empty.\n    >>> longest([])\n\n    >>> longest(['a', 'b', 'c'])\n    'a'\n    >>> longest(['a', 'bb', 'ccc'])\n    'ccc'\n    \"\"\"\n",
+        "canonical_solution": "    if not strings:\n        return None\n\n    maxlen = max(len(x) for x in strings)\n    for s in strings:\n        if len(s) == maxlen:\n            return s\n",
+        "test": "def check(candidate):\n    assert candidate([]) == None\n    assert candidate(['x', 'y', 'z']) == 'x'\n    assert candidate(['x', 'yyy', 'zzzz', 'www', 'kkkk', 'abc']) == 'zzzz'\n",
+        "entry_point": "longest",
+    },
+    {
+        "task_id": "HumanEval/13",
+        "prompt": "\n\ndef greatest_common_divisor(a: int, b: int) -> int:\n    \"\"\" Return a greatest common divisor of two integers a and b\n    >>> greatest_common_divisor(3, 5)\n    1\n    >>> greatest_common_divisor(25, 15)\n    5\n    \"\"\"\n",
+        "canonical_solution": "    while b:\n        a, b = b, a % b\n    return a\n",
+        "test": "def check(candidate):\n    assert candidate(3, 7) == 1\n    assert candidate(10, 15) == 5\n    assert candidate(49, 14) == 7\n    assert candidate(144, 60) == 12\n",
+        "entry_point": "greatest_common_divisor",
+    },
+    {
+        "task_id": "HumanEval/14",
+        "prompt": "from typing import List\n\n\ndef all_prefixes(string: str) -> List[str]:\n    \"\"\" Return list of all prefixes from shortest to longest of the input string\n    >>> all_prefixes('abc')\n    ['a', 'ab', 'abc']\n    \"\"\"\n",
+        "canonical_solution": "    result = []\n\n    for i in range(len(string)):\n        result.append(string[:i+1])\n    return result\n",
+        "test": "def check(candidate):\n    assert candidate('') == []\n    assert candidate('asdfgh') == ['a', 'as', 'asd', 'asdf', 'asdfe', 'asdfgh']\n    assert candidate('WWW') == ['W', 'WW', 'WWW']\n",
+        "entry_point": "all_prefixes",
+    },
+    {
+        "task_id": "HumanEval/15",
+        "prompt": "\n\ndef string_sequence(n: int) -> str:\n    \"\"\" Return a string containing space-delimited numbers starting from 0 upto n inclusive.\n    >>> string_sequence(0)\n    '0'\n    >>> string_sequence(5)\n    '0 1 2 3 4 5'\n    \"\"\"\n",
+        "canonical_solution": "    return ' '.join([str(x) for x in range(n + 1)])\n",
+        "test": "def check(candidate):\n    assert candidate(0) == '0'\n    assert candidate(3) == '0 1 2 3'\n    assert candidate(10) == '0 1 2 3 4 5 6 7 8 9 10'\n",
+        "entry_point": "string_sequence",
+    },
+    {
+        "task_id": "HumanEval/16",
+        "prompt": "\n\ndef count_distinct_characters(string: str) -> int:\n    \"\"\" Given a string, find out how many distinct characters (regardless of case) does it consist of\n    >>> count_distinct_characters('xyzXYZ')\n    3\n    >>> count_distinct_characters('Jerry')\n    4\n    \"\"\"\n",
+        "canonical_solution": "    return len(set(string.lower()))\n",
+        "test": "def check(candidate):\n    assert candidate('') == 0\n    assert candidate('abcde') == 5\n    assert candidate('abcde' + 'cade' + 'CADE') == 5\n    assert candidate('aaaaAAAAaaaa') == 1\n    assert candidate('Jerry jbread') == 7\n",
+        "entry_point": "count_distinct_characters",
+    },
+    {
+        "task_id": "HumanEval/17",
+        "prompt": "from typing import List\n\n\ndef parse_music(music_string: str) -> List[int]:\n    \"\"\" Input to this function is a string representing musical notes in a special ASCII format.\n    Your task is to parse this string and return list of integers corresponding to how many beats does each\n    not last.\n\n    Here is a legend:\n    'o' - whole note, lasts four beats\n    'o|' - half note, lasts two beats\n    '.|' - quarter note, lasts one beat\n\n    >>> parse_music('o o| .| o| o| .| .| .| .| o o')\n    [4, 2, 1, 2, 2, 1, 1, 1, 1, 4, 4]\n    \"\"\"\n",
+        "canonical_solution": "    note_map = {'o': 4, 'o|': 2, '.|': 1}\n    return [note_map[x] for x in music_string.split(' ') if x]\n",
+        "test": "def check(candidate):\n    assert candidate('') == []\n    assert candidate('o o o o') == [4, 4, 4, 4]\n    assert candidate('.| .| .| .|') == [1, 1, 1, 1]\n    assert candidate('o| o| .| .| o o o o') == [2, 2, 1, 1, 4, 4, 4, 4]\n    assert candidate('o| .| o| .| o o| o o|') == [2, 1, 2, 1, 4, 2, 4, 2]\n",
+        "entry_point": "parse_music",
+    },
+    {
+        "task_id": "HumanEval/18",
+        "prompt": "\n\ndef how_many_times(string: str, substring: str) -> int:\n    \"\"\" Find how many times a given substring can be found in the original string. Count overlapping cases.\n    >>> how_many_times('', 'a')\n    0\n    >>> how_many_times('aaa', 'a')\n    3\n    >>> how_many_times('aaaa', 'aa')\n    3\n    \"\"\"\n",
+        "canonical_solution": "    times = 0\n\n    for i in range(len(string) - len(substring) + 1):\n        if string[i:i+len(substring)] == substring:\n            times += 1\n\n    return times\n",
+        "test": "def check(candidate):\n    assert candidate('', 'x') == 0\n    assert candidate('xyxyxyx', 'x') == 4\n    assert candidate('cacacacac', 'cac') == 4\n    assert candidate('john doe', 'john') == 1\n",
+        "entry_point": "how_many_times",
+    },
+    {
+        "task_id": "HumanEval/19",
+        "prompt": "from typing import List\n\n\ndef sort_numbers(numbers: str) -> str:\n    \"\"\" Input is a space-delimited string of numberals from 'zero' to 'nine'.\n    Valid choices are 'zero', 'one', 'two', 'three', 'four', 'five', 'six', 'seven', 'eight' and 'nine'.\n    Return the string with numbers sorted from smallest to largest\n    >>> sort_numbers('three one five')\n    'one three five'\n    \"\"\"\n",
+        "canonical_solution": "    value_map = {\n        'zero': 0,\n        'one': 1,\n        'two': 2,\n        'three': 3,\n        'four': 4,\n        'five': 5,\n        'six': 6,\n        'seven': 7,\n        'eight': 8,\n        'nine': 9\n    }\n    return ' '.join(sorted([x for x in numbers.split(' ') if x], key=lambda x: value_map[x]))\n",
+        "test": "def check(candidate):\n    assert candidate('') == ''\n    assert candidate('three') == 'three'\n    assert candidate('three five nine') == 'three five nine'\n    assert candidate('five zero four seven nine eight') == 'zero four five seven eight nine'\n    assert candidate('six five four three two one zero') == 'zero one two three four five six'\n",
+        "entry_point": "sort_numbers",
+    },
+]
+
+
+class HumanEvalBenchmark(BenchmarkSuite):
+    """HumanEval: code generation from docstrings (164 problems)."""
+
+    name = "HumanEval"
+    description = "Code generation from Python function docstrings"
+    category = "coding"
+
+    def load_dataset(self) -> list[dict[str, Any]]:
+        """Load HumanEval problems.
+
+        Checks for a JSONL file at ``data_dir/humaneval.jsonl`` first.
+        Falls back to the built-in 20-problem subset.
+        """
+        jsonl_path = Path(self.data_dir) / "humaneval.jsonl"
+        if jsonl_path.exists():
+            problems: list[dict[str, Any]] = []
+            with open(jsonl_path) as fh:
+                for line in fh:
+                    line = line.strip()
+                    if line:
+                        problems.append(json.loads(line))
+            if self.verbose:
+                print(f"  Loaded {len(problems)} problems from {jsonl_path}")
+            return problems
+
+        if self.verbose:
+            print(
+                f"  {jsonl_path} not found — using built-in 20-problem subset"
+            )
+        return list(_BUILTIN_PROBLEMS)
+
+    def build_prompt(self, problem: dict[str, Any]) -> str:
+        prompt_code = problem["prompt"]
+        entry = problem["entry_point"]
+        return (
+            f"Complete the following Python function. Write ONLY the function body. "
+            f"Save the complete file (including the signature shown below) as solution.py.\n\n"
+            f"```python\n{prompt_code}```\n\n"
+            f"The function name is `{entry}`. Make sure the file is valid Python."
+        )
+
+    def setup_workspace(self, problem: dict[str, Any], workspace: str) -> None:
+        # Write the test harness so we can verify after the agent runs
+        test_code = problem["test"]
+        entry = problem["entry_point"]
+        harness = textwrap.dedent(f"""\
+            import sys
+            sys.path.insert(0, ".")
+            from solution import {entry}
+
+            {test_code}
+
+            check({entry})
+            print("ALL_TESTS_PASSED")
+        """)
+        with open(os.path.join(workspace, "test_harness.py"), "w") as fh:
+            fh.write(harness)
+
+    def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
+        pid = problem.get("task_id", problem.get("id", "unknown"))
+        sol = os.path.join(workspace, "solution.py")
+        if not os.path.exists(sol):
+            return BenchmarkResult(
+                problem_id=pid, passed=False, error="solution.py not found"
+            )
+
+        code, output = self._run_shell(
+            f"{sys.executable} test_harness.py", cwd=workspace, timeout=30.0
+        )
+        passed = code == 0 and "ALL_TESTS_PASSED" in output
+        return BenchmarkResult(
+            problem_id=pid,
+            passed=passed,
+            actual=output[:500],
+            error="" if passed else output[:500],
+        )

benchmarks/suites/ifeval.py

@@ -0,0 +1,232 @@
+"""
+IFEval benchmark suite.
+
+IFEval (Instruction Following Evaluation) tests whether a model can
+follow verifiable natural-language instructions — e.g., "write at least
+200 words", "include exactly 3 bullet points", "respond in all lowercase".
+
+Paper: https://arxiv.org/abs/2311.07911
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import re
+from pathlib import Path
+from typing import Any
+
+from .base import BenchmarkResult, BenchmarkSuite
+
+# ---------------------------------------------------------------------------
+# Built-in mini dataset — verifiable instruction-following tasks
+# ---------------------------------------------------------------------------
+
+_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
+    {
+        "id": "ifeval-001",
+        "instruction": "Write a short paragraph about the benefits of exercise. Your response must be entirely in lowercase. Save it to output.txt.",
+        "checks": [
+            {"type": "file_exists", "file": "output.txt"},
+            {"type": "all_lowercase", "file": "output.txt"},
+            {"type": "min_words", "file": "output.txt", "value": 20},
+        ],
+    },
+    {
+        "id": "ifeval-002",
+        "instruction": "Write a poem about the ocean. It must have exactly 4 lines. Save it to output.txt.",
+        "checks": [
+            {"type": "file_exists", "file": "output.txt"},
+            {"type": "exact_line_count", "file": "output.txt", "value": 4},
+        ],
+    },
+    {
+        "id": "ifeval-003",
+        "instruction": "List 5 programming languages. Format each as a numbered item (1. Language). Save to output.txt.",
+        "checks": [
+            {"type": "file_exists", "file": "output.txt"},
+            {"type": "min_lines", "file": "output.txt", "value": 5},
+            {"type": "contains_pattern", "file": "output.txt", "pattern": r"^\d+\.", "min_count": 5},
+        ],
+    },
+    {
+        "id": "ifeval-004",
+        "instruction": "Write a summary of what Python is in EXACTLY 3 sentences. Each sentence must end with a period. Save to output.txt.",
+        "checks": [
+            {"type": "file_exists", "file": "output.txt"},
+            {"type": "exact_sentence_count", "file": "output.txt", "value": 3},
+        ],
+    },
+    {
+        "id": "ifeval-005",
+        "instruction": "Write a response that contains the word 'innovation' at least 3 times. The response should be about technology. Save to output.txt.",
+        "checks": [
+            {"type": "file_exists", "file": "output.txt"},
+            {"type": "word_frequency", "file": "output.txt", "word": "innovation", "min_count": 3},
+        ],
+    },
+    {
+        "id": "ifeval-006",
+        "instruction": "Write a short paragraph about space exploration. It must contain exactly 2 bullet points, each starting with '- '. Save to output.txt.",
+        "checks": [
+            {"type": "file_exists", "file": "output.txt"},
+            {"type": "contains_pattern", "file": "output.txt", "pattern": r"^- ", "min_count": 2, "max_count": 2},
+        ],
+    },
+    {
+        "id": "ifeval-007",
+        "instruction": "Write the numbers from 1 to 10, each on a separate line. Save to output.txt.",
+        "checks": [
+            {"type": "file_exists", "file": "output.txt"},
+            {"type": "exact_line_count", "file": "output.txt", "value": 10},
+            {"type": "contains", "file": "output.txt", "text": "1"},
+            {"type": "contains", "file": "output.txt", "text": "10"},
+        ],
+    },
+    {
+        "id": "ifeval-008",
+        "instruction": "Write a haiku (3 lines: 5 syllables, 7 syllables, 5 syllables) about nature. Save to output.txt. The response must be exactly 3 lines.",
+        "checks": [
+            {"type": "file_exists", "file": "output.txt"},
+            {"type": "exact_line_count", "file": "output.txt", "value": 3},
+        ],
+    },
+    {
+        "id": "ifeval-009",
+        "instruction": "Write a paragraph about artificial intelligence. Every sentence must start with the word 'AI'. Save to output.txt.",
+        "checks": [
+            {"type": "file_exists", "file": "output.txt"},
+            {"type": "sentences_start_with", "file": "output.txt", "prefix": "AI"},
+        ],
+    },
+    {
+        "id": "ifeval-010",
+        "instruction": "Create a JSON file called output.txt containing an object with exactly 3 keys: 'name', 'age', and 'city'. Values can be anything.",
+        "checks": [
+            {"type": "file_exists", "file": "output.txt"},
+            {"type": "valid_json", "file": "output.txt"},
+            {"type": "json_has_keys", "file": "output.txt", "keys": ["name", "age", "city"]},
+        ],
+    },
+]
+
+
+def _run_check(check: dict[str, Any], workspace: str) -> bool:
+    """Run a single verification check. Returns True if passed."""
+    check_type = check["type"]
+    filepath = os.path.join(workspace, check.get("file", "output.txt"))
+
+    if check_type == "file_exists":
+        return os.path.exists(filepath)
+
+    if not os.path.exists(filepath):
+        return False
+
+    with open(filepath) as fh:
+        content = fh.read()
+
+    lines = [l for l in content.strip().split("\n") if l.strip()]
+
+    if check_type == "all_lowercase":
+        # Check that alphabetic characters are lowercase
+        alpha_chars = [c for c in content if c.isalpha()]
+        return all(c.islower() for c in alpha_chars) if alpha_chars else True
+
+    if check_type == "min_words":
+        words = content.split()
+        return len(words) >= check["value"]
+
+    if check_type == "exact_line_count":
+        return len(lines) == check["value"]
+
+    if check_type == "min_lines":
+        return len(lines) >= check["value"]
+
+    if check_type == "contains_pattern":
+        pattern = check["pattern"]
+        matches = sum(1 for line in lines if re.search(pattern, line))
+        if "min_count" in check and matches < check["min_count"]:
+            return False
+        if "max_count" in check and matches > check["max_count"]:
+            return False
+        return True
+
+    if check_type == "exact_sentence_count":
+        # Count sentences ending with . ! or ?
+        sentences = re.split(r"[.!?]+", content.strip())
+        sentences = [s.strip() for s in sentences if s.strip()]
+        return len(sentences) == check["value"]
+
+    if check_type == "word_frequency":
+        word = check["word"].lower()
+        count = content.lower().count(word)
+        return count >= check["min_count"]
+
+    if check_type == "contains":
+        return check["text"] in content
+
+    if check_type == "sentences_start_with":
+        prefix = check["prefix"]
+        sentences = re.split(r"[.!?]\s+", content.strip())
+        sentences = [s.strip() for s in sentences if s.strip()]
+        return all(s.startswith(prefix) for s in sentences) if sentences else False
+
+    if check_type == "valid_json":
+        try:
+            json.loads(content)
+            return True
+        except json.JSONDecodeError:
+            return False
+
+    if check_type == "json_has_keys":
+        try:
+            data = json.loads(content)
+            return all(k in data for k in check["keys"])
+        except (json.JSONDecodeError, TypeError):
+            return False
+
+    return False
+
+
+class IFEvalBenchmark(BenchmarkSuite):
+    """IFEval: instruction following evaluation."""
+
+    name = "IFEval"
+    description = "Verifiable instruction-following evaluation"
+    category = "instruction-following"
+
+    def load_dataset(self) -> list[dict[str, Any]]:
+        jsonl_path = Path(self.data_dir) / "ifeval.jsonl"
+        if jsonl_path.exists():
+            problems: list[dict[str, Any]] = []
+            with open(jsonl_path) as fh:
+                for line in fh:
+                    line = line.strip()
+                    if line:
+                        problems.append(json.loads(line))
+            if self.verbose:
+                print(f"  Loaded {len(problems)} problems from {jsonl_path}")
+            return problems
+
+        if self.verbose:
+            print(f"  {jsonl_path} not found — using built-in 10-problem subset")
+        return list(_BUILTIN_PROBLEMS)
+
+    def build_prompt(self, problem: dict[str, Any]) -> str:
+        return problem["instruction"]
+
+    def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
+        pid = problem.get("id", "unknown")
+        checks = problem.get("checks", [])
+
+        failed_checks: list[str] = []
+        for check in checks:
+            if not _run_check(check, workspace):
+                failed_checks.append(check["type"])
+
+        passed = len(failed_checks) == 0
+        return BenchmarkResult(
+            problem_id=pid, passed=passed,
+            error=f"Failed checks: {', '.join(failed_checks)}" if failed_checks else "",
+            metadata={"total_checks": len(checks), "failed_checks": failed_checks},
+        )

benchmarks/suites/livecodebench.py

@@ -0,0 +1,186 @@
+"""
+LiveCodeBench benchmark suite.
+
+LiveCodeBench evaluates coding agents on competitive-programming-style
+problems that were published *after* the model's training cutoff, preventing
+data contamination.
+
+Reference: https://livecodebench.github.io/
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import sys
+import textwrap
+from pathlib import Path
+from typing import Any
+
+from .base import BenchmarkResult, BenchmarkSuite
+
+# ---------------------------------------------------------------------------
+# Built-in mini dataset — competitive-programming style problems
+# ---------------------------------------------------------------------------
+
+_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
+    {
+        "id": "lcb-001",
+        "title": "Two Sum Sorted",
+        "description": (
+            "Given a sorted array of integers `nums` and a target integer `target`, "
+            "return the indices (0-based) of the two numbers that add up to target. "
+            "You may assume each input has exactly one solution, and you may not use "
+            "the same element twice. Return them as a list [i, j] where i < j."
+        ),
+        "test_cases": [
+            {"input": {"nums": [2, 7, 11, 15], "target": 9}, "expected": [0, 1]},
+            {"input": {"nums": [1, 3, 5, 7], "target": 8}, "expected": [0, 3]},
+            {"input": {"nums": [-1, 0, 3, 5], "target": 4}, "expected": [0, 3]},
+            {"input": {"nums": [1, 2], "target": 3}, "expected": [0, 1]},
+        ],
+        "function_name": "two_sum_sorted",
+    },
+    {
+        "id": "lcb-002",
+        "title": "Maximum Subarray",
+        "description": (
+            "Given an array of integers `nums`, find the contiguous subarray "
+            "(containing at least one number) which has the largest sum and return its sum."
+        ),
+        "test_cases": [
+            {"input": {"nums": [-2, 1, -3, 4, -1, 2, 1, -5, 4]}, "expected": 6},
+            {"input": {"nums": [1]}, "expected": 1},
+            {"input": {"nums": [-1, -2, -3]}, "expected": -1},
+            {"input": {"nums": [5, 4, -1, 7, 8]}, "expected": 23},
+        ],
+        "function_name": "max_subarray",
+    },
+    {
+        "id": "lcb-003",
+        "title": "Valid Parentheses",
+        "description": (
+            "Given a string `s` containing just the characters '(', ')', '{', '}', "
+            "'[' and ']', determine if the input string is valid. An input string is "
+            "valid if open brackets are closed by the same type of brackets in the "
+            "correct order."
+        ),
+        "test_cases": [
+            {"input": {"s": "()"}, "expected": True},
+            {"input": {"s": "()[]{}"}, "expected": True},
+            {"input": {"s": "(]"}, "expected": False},
+            {"input": {"s": "([)]"}, "expected": False},
+            {"input": {"s": "{[]}"}, "expected": True},
+            {"input": {"s": ""}, "expected": True},
+        ],
+        "function_name": "is_valid",
+    },
+    {
+        "id": "lcb-004",
+        "title": "Longest Common Prefix",
+        "description": (
+            "Write a function that finds the longest common prefix string "
+            "amongst a list of strings. If there is no common prefix, return "
+            "an empty string."
+        ),
+        "test_cases": [
+            {"input": {"strs": ["flower", "flow", "flight"]}, "expected": "fl"},
+            {"input": {"strs": ["dog", "racecar", "car"]}, "expected": ""},
+            {"input": {"strs": ["abc"]}, "expected": "abc"},
+            {"input": {"strs": ["", "b"]}, "expected": ""},
+            {"input": {"strs": ["ab", "ab", "ab"]}, "expected": "ab"},
+        ],
+        "function_name": "longest_common_prefix",
+    },
+    {
+        "id": "lcb-005",
+        "title": "Count Inversions",
+        "description": (
+            "Given an array of integers, count the number of inversions. "
+            "An inversion is a pair (i, j) where i < j but nums[i] > nums[j]. "
+            "Return the total number of inversions."
+        ),
+        "test_cases": [
+            {"input": {"nums": [2, 4, 1, 3, 5]}, "expected": 3},
+            {"input": {"nums": [1, 2, 3, 4]}, "expected": 0},
+            {"input": {"nums": [4, 3, 2, 1]}, "expected": 6},
+            {"input": {"nums": [1]}, "expected": 0},
+        ],
+        "function_name": "count_inversions",
+    },
+]
+
+
+class LiveCodeBenchBenchmark(BenchmarkSuite):
+    """LiveCodeBench: competitive programming problems."""
+
+    name = "LiveCodeBench"
+    description = "Competitive programming problems (post-training-cutoff)"
+    category = "coding"
+
+    def load_dataset(self) -> list[dict[str, Any]]:
+        jsonl_path = Path(self.data_dir) / "livecodebench.jsonl"
+        if jsonl_path.exists():
+            problems: list[dict[str, Any]] = []
+            with open(jsonl_path) as fh:
+                for line in fh:
+                    line = line.strip()
+                    if line:
+                        problems.append(json.loads(line))
+            if self.verbose:
+                print(f"  Loaded {len(problems)} problems from {jsonl_path}")
+            return problems
+
+        if self.verbose:
+            print(f"  {jsonl_path} not found — using built-in 5-problem subset")
+        return list(_BUILTIN_PROBLEMS)
+
+    def build_prompt(self, problem: dict[str, Any]) -> str:
+        desc = problem["description"]
+        fname = problem["function_name"]
+        # Build example from first test case
+        tc = problem["test_cases"][0]
+        args = ", ".join(f"{k}={repr(v)}" for k, v in tc["input"].items())
+        expected = repr(tc["expected"])
+        return (
+            f"Solve the following competitive programming problem.\n\n"
+            f"## {problem.get('title', 'Problem')}\n\n"
+            f"{desc}\n\n"
+            f"Example: {fname}({args}) => {expected}\n\n"
+            f"Write a Python function called `{fname}` and save it to solution.py."
+        )
+
+    def setup_workspace(self, problem: dict[str, Any], workspace: str) -> None:
+        fname = problem["function_name"]
+        test_lines = [
+            "import sys",
+            'sys.path.insert(0, ".")',
+            f"from solution import {fname}",
+            "",
+        ]
+        for i, tc in enumerate(problem["test_cases"]):
+            args = ", ".join(f"{k}={repr(v)}" for k, v in tc["input"].items())
+            test_lines.append(
+                f"assert {fname}({args}) == {repr(tc['expected'])}, "
+                f"'Test case {i+1} failed'"
+            )
+        test_lines.append("")
+        test_lines.append('print("ALL_TESTS_PASSED")')
+        with open(os.path.join(workspace, "test_harness.py"), "w") as fh:
+            fh.write("\n".join(test_lines))
+
+    def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
+        pid = problem.get("id", "unknown")
+        sol = os.path.join(workspace, "solution.py")
+        if not os.path.exists(sol):
+            return BenchmarkResult(
+                problem_id=pid, passed=False, error="solution.py not found"
+            )
+        code, output = self._run_shell(
+            f"{sys.executable} test_harness.py", cwd=workspace, timeout=30.0
+        )
+        passed = code == 0 and "ALL_TESTS_PASSED" in output
+        return BenchmarkResult(
+            problem_id=pid, passed=passed, actual=output[:500],
+            error="" if passed else output[:500],
+        )

benchmarks/suites/math_bench.py

@@ -0,0 +1,212 @@
+"""
+MATH benchmark suite.
+
+The MATH dataset consists of 12,500 problems from mathematics competitions
+(AMC, AIME, etc.) covering topics like algebra, counting, geometry, number
+theory, and more.  Difficulty levels 1–5.
+
+Paper: https://arxiv.org/abs/2103.03874
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import re
+import sys
+from pathlib import Path
+from typing import Any
+
+from .base import BenchmarkResult, BenchmarkSuite
+
+# ---------------------------------------------------------------------------
+# Built-in mini dataset (15 representative problems)
+# ---------------------------------------------------------------------------
+
+_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
+    {
+        "id": "math-001",
+        "problem": "What is the value of $2^{10}$?",
+        "answer": "1024",
+        "subject": "algebra",
+        "level": 1,
+    },
+    {
+        "id": "math-002",
+        "problem": "What is $15 \\% $ of $200$?",
+        "answer": "30",
+        "subject": "algebra",
+        "level": 1,
+    },
+    {
+        "id": "math-003",
+        "problem": "Solve for $x$: $3x + 7 = 22$.",
+        "answer": "5",
+        "subject": "algebra",
+        "level": 1,
+    },
+    {
+        "id": "math-004",
+        "problem": "What is the sum of the first 10 positive integers?",
+        "answer": "55",
+        "subject": "algebra",
+        "level": 1,
+    },
+    {
+        "id": "math-005",
+        "problem": "What is $\\gcd(12, 18)$?",
+        "answer": "6",
+        "subject": "number_theory",
+        "level": 1,
+    },
+    {
+        "id": "math-006",
+        "problem": "How many ways can you choose 3 items from a set of 5 items?",
+        "answer": "10",
+        "subject": "counting",
+        "level": 2,
+    },
+    {
+        "id": "math-007",
+        "problem": "What is the area of a triangle with base 10 and height 6?",
+        "answer": "30",
+        "subject": "geometry",
+        "level": 1,
+    },
+    {
+        "id": "math-008",
+        "problem": "Evaluate $\\sum_{k=1}^{5} k^2$.",
+        "answer": "55",
+        "subject": "algebra",
+        "level": 2,
+    },
+    {
+        "id": "math-009",
+        "problem": "Find the remainder when $2^{20}$ is divided by $7$.",
+        "answer": "4",
+        "subject": "number_theory",
+        "level": 3,
+    },
+    {
+        "id": "math-010",
+        "problem": "What is the least common multiple of 12 and 18?",
+        "answer": "36",
+        "subject": "number_theory",
+        "level": 2,
+    },
+    {
+        "id": "math-011",
+        "problem": "A bag contains 3 red and 5 blue marbles. What is the probability of drawing a red marble? Express as a simplified fraction.",
+        "answer": "3/8",
+        "subject": "counting",
+        "level": 2,
+    },
+    {
+        "id": "math-012",
+        "problem": "What is the value of $\\sqrt{144}$?",
+        "answer": "12",
+        "subject": "algebra",
+        "level": 1,
+    },
+    {
+        "id": "math-013",
+        "problem": "If $f(x) = 2x^2 - 3x + 1$, what is $f(3)$?",
+        "answer": "10",
+        "subject": "algebra",
+        "level": 2,
+    },
+    {
+        "id": "math-014",
+        "problem": "How many prime numbers are there between 1 and 20?",
+        "answer": "8",
+        "subject": "number_theory",
+        "level": 2,
+    },
+    {
+        "id": "math-015",
+        "problem": "What is the value of $3! + 4!$?",
+        "answer": "30",
+        "subject": "counting",
+        "level": 1,
+    },
+]
+
+
+def _normalize_answer(text: str) -> str:
+    """Normalize a math answer for comparison.
+
+    Strips whitespace, dollar signs, \\boxed{}, etc.
+    """
+    text = text.strip()
+    # Remove \boxed{...}
+    m = re.search(r"\\boxed\{(.+?)\}", text)
+    if m:
+        text = m.group(1)
+    # Remove dollar signs and whitespace
+    text = text.replace("$", "").replace(",", "").strip()
+    # Try to normalize fractions
+    text = text.replace("\\frac{", "").replace("}{", "/").replace("}", "")
+    return text.strip()
+
+
+class MATHBenchmark(BenchmarkSuite):
+    """MATH: competition mathematics problems."""
+
+    name = "MATH"
+    description = "Competition mathematics problems (algebra, geometry, number theory, etc.)"
+    category = "math"
+
+    def load_dataset(self) -> list[dict[str, Any]]:
+        jsonl_path = Path(self.data_dir) / "math.jsonl"
+        if jsonl_path.exists():
+            problems: list[dict[str, Any]] = []
+            with open(jsonl_path) as fh:
+                for line in fh:
+                    line = line.strip()
+                    if line:
+                        problems.append(json.loads(line))
+            if self.verbose:
+                print(f"  Loaded {len(problems)} problems from {jsonl_path}")
+            return problems
+
+        if self.verbose:
+            print(f"  {jsonl_path} not found — using built-in 15-problem subset")
+        return list(_BUILTIN_PROBLEMS)
+
+    def build_prompt(self, problem: dict[str, Any]) -> str:
+        prob = problem["problem"]
+        return (
+            f"Solve the following math problem. Write ONLY the final numerical "
+            f"answer (a single number or simple expression) to a file called answer.txt.\n\n"
+            f"Problem: {prob}\n\n"
+            f"Save just the answer to answer.txt — no explanation, no work, just the answer."
+        )
+
+    def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
+        pid = problem.get("id", str(problem.get("task_id", "unknown")))
+        expected_raw = str(problem["answer"])
+        answer_file = os.path.join(workspace, "answer.txt")
+
+        if not os.path.exists(answer_file):
+            return BenchmarkResult(
+                problem_id=pid, passed=False, expected=expected_raw,
+                error="answer.txt not found",
+            )
+
+        with open(answer_file) as fh:
+            actual_raw = fh.read().strip()
+
+        expected_norm = _normalize_answer(expected_raw)
+        actual_norm = _normalize_answer(actual_raw)
+
+        # Try numeric comparison
+        try:
+            passed = abs(float(actual_norm) - float(expected_norm)) < 1e-6
+        except (ValueError, ZeroDivisionError):
+            passed = actual_norm == expected_norm
+
+        return BenchmarkResult(
+            problem_id=pid, passed=passed,
+            expected=expected_raw, actual=actual_raw,
+            error="" if passed else f"expected={expected_raw}, got={actual_raw}",
+        )

benchmarks/suites/mbpp.py

@@ -0,0 +1,241 @@
+"""
+MBPP (Mostly Basic Python Problems) benchmark suite.
+
+974 crowd-sourced Python programming problems designed to be solvable by
+entry-level programmers.  Each problem has a task description, code solution
+and 3 automated test cases.
+
+Paper: https://arxiv.org/abs/2108.07732
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import sys
+import textwrap
+from pathlib import Path
+from typing import Any
+
+from .base import BenchmarkResult, BenchmarkSuite
+
+# ---------------------------------------------------------------------------
+# Built-in mini dataset (15 representative MBPP problems)
+# ---------------------------------------------------------------------------
+
+_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
+    {
+        "task_id": 1,
+        "text": "Write a function to find the minimum cost path to reach (m, n) from (0, 0) for the given cost matrix.",
+        "code": "R = 3\r\nC = 3\r\ndef min_cost(cost, m, n):\r\n\ttc = [[0 for x in range(C)] for x in range(R)]\r\n\ttc[0][0] = cost[0][0]\r\n\tfor i in range(1, m+1):\r\n\t\ttc[i][0] = tc[i-1][0] + cost[i][0]\r\n\tfor j in range(1, n+1):\r\n\t\ttc[0][j] = tc[0][j-1] + cost[0][j]\r\n\tfor i in range(1, m+1):\r\n\t\tfor j in range(1, n+1):\r\n\t\t\ttc[i][j] = min(tc[i-1][j-1], tc[i-1][j], tc[i][j-1]) + cost[i][j]\r\n\treturn tc[m][n]",
+        "test_list": [
+            "assert min_cost([[1, 2, 3], [4, 8, 2], [1, 5, 3]], 2, 2) == 8",
+            "assert min_cost([[2, 3, 4], [5, 9, 3], [2, 6, 4]], 2, 2) == 12",
+            "assert min_cost([[3, 4, 5], [6, 10, 4], [3, 7, 5]], 2, 2) == 16",
+        ],
+    },
+    {
+        "task_id": 2,
+        "text": "Write a function to find the similar elements from the given two tuple lists.",
+        "code": "def similar_elements(test_tup1, test_tup2):\r\n  res = tuple(set(test_tup1) & set(test_tup2))\r\n  return (res) ",
+        "test_list": [
+            "assert similar_elements((3, 4, 5, 6),(5, 7, 4, 10)) == (4, 5)",
+            "assert similar_elements((1, 2, 3, 4),(5, 4, 3, 7)) == (3, 4)",
+            "assert similar_elements((11, 12, 14, 13),(17, 15, 14, 13)) == (13, 14)",
+        ],
+    },
+    {
+        "task_id": 3,
+        "text": "Write a python function to identify non-prime numbers.",
+        "code": "import math\ndef is_not_prime(n):\n    result = False\n    for i in range(2,int(math.sqrt(n)) + 1):\n        if n % i == 0:\n            result = True\n    return result",
+        "test_list": [
+            "assert is_not_prime(2) == False",
+            "assert is_not_prime(10) == True",
+            "assert is_not_prime(35) == True",
+        ],
+    },
+    {
+        "task_id": 4,
+        "text": "Write a function to find the largest integers from a given list of numbers using heap queue algorithm.",
+        "code": "import heapq as hq\ndef heap_queue_largest(nums,n):\n  largest_nums = hq.nlargest(n, nums)\n  return largest_nums",
+        "test_list": [
+            "assert heap_queue_largest( [25, 35, 22, 85, 14, 65, 75, 22, 58],3)==[85, 75, 65]",
+            "assert heap_queue_largest( [25, 35, 22, 85, 14, 65, 75, 22, 58],2)==[85, 75]",
+            "assert heap_queue_largest( [25, 35, 22, 85, 14, 65, 75, 22, 58],5)==[85, 75, 65, 58, 35]",
+        ],
+    },
+    {
+        "task_id": 5,
+        "text": "Write a function to find the number of ways to fill it with 2 x 1 dominoes for the given 3 x n board.",
+        "code": "def count_ways(n):\n\tA = [0] * (n + 1)\n\tB = [0] * (n + 1)\n\tA[0] = 1\n\tA[1] = 0\n\tB[0] = 0\n\tB[1] = 1\n\tfor i in range(2, n+1):\n\t\tA[i] = A[i - 2] + 2 * B[i - 1]\n\t\tB[i] = A[i - 1] + B[i - 2]\n\treturn A[n]",
+        "test_list": [
+            "assert count_ways(2) == 3",
+            "assert count_ways(8) == 153",
+            "assert count_ways(12) == 2131",
+        ],
+    },
+    {
+        "task_id": 6,
+        "text": "Write a python function to check whether the two numbers differ at one bit position only or not.",
+        "code": "def differ_At_One_Bit_Pos(a,b):\n    return ((a ^ b) & (a ^ b - 1)) == 0 and a ^ b != 0",
+        "test_list": [
+            "assert differ_At_One_Bit_Pos(13,9) == True",
+            "assert differ_At_One_Bit_Pos(15,8) == False",
+            "assert differ_At_One_Bit_Pos(2,4) == False",
+        ],
+    },
+    {
+        "task_id": 7,
+        "text": "Write a function to find all words which are at least 4 characters long in a string.",
+        "code": 'import re\ndef find_char_long(text):\n  return (re.findall(r"\\b\\w{4,}\\b", text))',
+        "test_list": [
+            'assert find_char_long(\'Please move back to stream\') == [\'Please\', \'move\', \'back\', \'stream\']',
+            'assert find_char_long(\'Joveacvber gfnbb vcdf\') == [\'Jove\', \'acvber\', \'gfnbb\', \'vcdf\']' + "",
+            'assert find_char_long(\'Davvede aridge\') == [\'Davvede\', \'aridge\']',
+        ],
+    },
+    {
+        "task_id": 8,
+        "text": "Write a function to find squares of individual elements in a list.",
+        "code": "def square_nums(nums):\n return list(map(lambda x: x ** 2, nums))",
+        "test_list": [
+            "assert square_nums([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])==[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]",
+            "assert square_nums([10,20,30])==[100,400,900]",
+            "assert square_nums([12,15])==[144,225]",
+        ],
+    },
+    {
+        "task_id": 9,
+        "text": "Write a python function to find the minimum number of rotations required to get the same string.",
+        "code": "def find_Rotations(s):\n    n = len(s)\n    s2 = s + s\n    for i in range(1, n + 1):\n        if s2[i:i+n] == s:\n            return i\n    return n",
+        "test_list": [
+            'assert find_Rotations("aaaa") == 1',
+            'assert find_Rotations("ab") == 2',
+            'assert find_Rotations("abc") == 3',
+        ],
+    },
+    {
+        "task_id": 10,
+        "text": "Write a function to get the n smallest items from a dataset.",
+        "code": "import heapq\ndef small_nnum(list1,n):\n  smallest=heapq.nsmallest(n,list1)\n  return smallest",
+        "test_list": [
+            "assert small_nnum([10, 20, 50, 70, 90, 20, 50, 40, 60, 80, 100],2)==[10,20]",
+            "assert small_nnum([10, 20, 50, 70, 90, 20, 50, 40, 60, 80, 100],5)==[10,20,20,40,50]",
+            "assert small_nnum([10, 20, 50, 70, 90, 20, 50, 40, 60, 80, 100],3)==[10,20,20]",
+        ],
+    },
+    {
+        "task_id": 11,
+        "text": "Write a python function to remove first and last occurrence of a given character from the string.",
+        "code": "def remove_Occ(s,ch):\n    # Remove first occurrence\n    for i in range(len(s)):\n        if (s[i] == ch):\n            s = s[0 : i] + s[i + 1:]\n            break\n    # Remove last occurrence\n    for i in range(len(s) - 1,-1,-1):\n        if (s[i] == ch):\n            s = s[0 : i] + s[i + 1:]\n            break\n    return s",
+        "test_list": [
+            'assert remove_Occ("hello","l") == "heo"',
+            'assert remove_Occ("abcda","a") == "bcd"',
+            'assert remove_Occ("PHP","P") == "H"',
+        ],
+    },
+    {
+        "task_id": 12,
+        "text": "Write a function to sort a given matrix in ascending order according to the sum of its rows.",
+        "code": "def sort_matrix(M):\n    result = sorted(M, key=sum)\n    return result",
+        "test_list": [
+            "assert sort_matrix([[1, 2, 3], [2, 4, 5], [1, 1, 1]])==[[1, 1, 1], [1, 2, 3], [2, 4, 5]]",
+            "assert sort_matrix([[1, 2, 3], [-2, 4, -5], [1, -1, 1]])==[[-2, 4, -5], [1, -1, 1], [1, 2, 3]]",
+            "assert sort_matrix([[5,8,9],[6,4,3],[2,1,4]])==[[2, 1, 4], [6, 4, 3], [5, 8, 9]]",
+        ],
+    },
+    {
+        "task_id": 13,
+        "text": "Write a function to count the most common words in a dictionary.",
+        "code": "from collections import Counter\ndef count_common(words):\n  word_counts = Counter(words)\n  top_four = word_counts.most_common(4)\n  return top_four",
+        "test_list": [
+            "assert count_common(['red','green','black','pink','black','white','black','eyes','white','black','orange','pink','pink','red','red','white','orange','white','black','pink','green','green','pink','green','pink','white','orange','orange','red']) == [('pink', 6), ('black', 5), ('white', 5), ('red', 4)]",
+            "assert count_common(['one', 'two', 'three', 'four', 'five', 'one', 'two', 'one', 'three', 'one']) == [('one', 4), ('two', 2), ('three', 2), ('four', 1)]",
+            "assert count_common(['Facebook', 'Apple', 'Amazon', 'Netflix', 'Google', 'Apple', 'Netflix', 'Amazon']) == [('Apple', 2), ('Amazon', 2), ('Netflix', 2), ('Facebook', 1)]",
+        ],
+    },
+    {
+        "task_id": 14,
+        "text": "Write a python function to find the volume of a triangular prism.",
+        "code": "def find_Volume(l,b,h):\n    return (l * b * h) / 2",
+        "test_list": [
+            "assert find_Volume(10,8,6) == 240",
+            "assert find_Volume(3,2,2) == 6",
+            "assert find_Volume(1,2,1) == 1",
+        ],
+    },
+    {
+        "task_id": 15,
+        "text": "Write a function to split a string at uppercase letters.",
+        "code": "import re\ndef split_upperstring(text):\n return re.findall('[A-Z][^A-Z]*', text)",
+        "test_list": [
+            'assert split_upperstring("PythonProgramLanguage")==[\'Python\',\'Program\',\'Language\']',
+            'assert split_upperstring("PythonProgram")==[\'Python\',\'Program\']',
+            'assert split_upperstring("ProgramLanguage")==[\'Program\',\'Language\']',
+        ],
+    },
+]
+
+
+class MBPPBenchmark(BenchmarkSuite):
+    """MBPP: Mostly Basic Python Problems (974 problems)."""
+
+    name = "MBPP"
+    description = "Crowd-sourced basic Python programming problems"
+    category = "coding"
+
+    def load_dataset(self) -> list[dict[str, Any]]:
+        jsonl_path = Path(self.data_dir) / "mbpp.jsonl"
+        if jsonl_path.exists():
+            problems: list[dict[str, Any]] = []
+            with open(jsonl_path) as fh:
+                for line in fh:
+                    line = line.strip()
+                    if line:
+                        problems.append(json.loads(line))
+            if self.verbose:
+                print(f"  Loaded {len(problems)} problems from {jsonl_path}")
+            return problems
+
+        if self.verbose:
+            print(f"  {jsonl_path} not found — using built-in 15-problem subset")
+        return list(_BUILTIN_PROBLEMS)
+
+    def build_prompt(self, problem: dict[str, Any]) -> str:
+        desc = problem["text"]
+        tests = "\n".join(problem.get("test_list", []))
+        return (
+            f"{desc}\n\n"
+            f"Write the Python code and save it to solution.py.\n"
+            f"The following test assertions must pass:\n```python\n{tests}\n```"
+        )
+
+    def setup_workspace(self, problem: dict[str, Any], workspace: str) -> None:
+        tests = problem.get("test_list", [])
+        test_code = "\n".join(tests)
+        harness = textwrap.dedent(f"""\
+            import sys
+            sys.path.insert(0, ".")
+            from solution import *
+
+            {test_code}
+
+            print("ALL_TESTS_PASSED")
+        """)
+        with open(os.path.join(workspace, "test_harness.py"), "w") as fh:
+            fh.write(harness)
+
+    def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
+        pid = str(problem.get("task_id", "unknown"))
+        sol = os.path.join(workspace, "solution.py")
+        if not os.path.exists(sol):
+            return BenchmarkResult(problem_id=pid, passed=False, error="solution.py not found")
+
+        code, output = self._run_shell(
+            f"{sys.executable} test_harness.py", cwd=workspace, timeout=30.0
+        )
+        passed = code == 0 and "ALL_TESTS_PASSED" in output
+        return BenchmarkResult(
+            problem_id=pid, passed=passed, actual=output[:500],
+            error="" if passed else output[:500],
+        )

benchmarks/suites/swe_bench.py

@@ -0,0 +1,211 @@
+"""
+SWE-Bench benchmark suite.
+
+SWE-Bench tests whether an agent can resolve real GitHub issues.
+Each problem provides a repository, an issue description, and a patch
+that should make the failing tests pass.
+
+Paper: https://arxiv.org/abs/2310.06770
+Dataset: https://huggingface.co/datasets/princeton-nlp/SWE-bench_Lite
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import sys
+import textwrap
+from pathlib import Path
+from typing import Any
+
+from .base import BenchmarkResult, BenchmarkSuite
+
+# ---------------------------------------------------------------------------
+# Built-in mini dataset — simplified SWE-Bench-style problems
+# These are self-contained (no actual repo cloning needed).
+# ---------------------------------------------------------------------------
+
+_BUILTIN_PROBLEMS: list[dict[str, Any]] = [
+    {
+        "instance_id": "swe-mini-001",
+        "problem_statement": (
+            "The `StringFormatter.format_title` method should capitalize the first letter "
+            "of each word in the input string. Currently it lowercases everything instead.\n\n"
+            "Expected: format_title('hello world') => 'Hello World'\n"
+            "Actual:   format_title('hello world') => 'hello world'"
+        ),
+        "setup_code": textwrap.dedent("""\
+            cat > formatter.py << 'PYEOF'
+            class StringFormatter:
+                def format_title(self, text):
+                    return text.lower()
+            PYEOF
+            cat > test_formatter.py << 'PYEOF'
+            from formatter import StringFormatter
+            f = StringFormatter()
+            assert f.format_title('hello world') == 'Hello World'
+            assert f.format_title('PYTHON IS FUN') == 'Python Is Fun'
+            assert f.format_title('already Good') == 'Already Good'
+            print("ALL_TESTS_PASSED")
+            PYEOF
+        """),
+        "test_cmd": "python3 test_formatter.py",
+    },
+    {
+        "instance_id": "swe-mini-002",
+        "problem_statement": (
+            "The `DataValidator.validate_email` function returns True for emails "
+            "that don't have a domain extension (e.g., 'user@domain' without .com). "
+            "It should require at least one dot after the @ sign.\n\n"
+            "Expected: validate_email('user@domain') => False\n"
+            "Actual:   validate_email('user@domain') => True"
+        ),
+        "setup_code": textwrap.dedent("""\
+            cat > validator.py << 'PYEOF'
+            class DataValidator:
+                def validate_email(self, email):
+                    return '@' in email
+            PYEOF
+            cat > test_validator.py << 'PYEOF'
+            from validator import DataValidator
+            v = DataValidator()
+            assert v.validate_email('user@example.com') == True
+            assert v.validate_email('bad') == False
+            assert v.validate_email('user@domain') == False
+            assert v.validate_email('a@b.c') == True
+            print("ALL_TESTS_PASSED")
+            PYEOF
+        """),
+        "test_cmd": "python3 test_validator.py",
+    },
+    {
+        "instance_id": "swe-mini-003",
+        "problem_statement": (
+            "The `FileProcessor.count_lines` method crashes with an unhandled "
+            "exception when the file doesn't exist. It should return -1 for missing files.\n\n"
+            "Expected: count_lines('nonexistent.txt') => -1\n"
+            "Actual:   FileNotFoundError is raised"
+        ),
+        "setup_code": textwrap.dedent("""\
+            cat > processor.py << 'PYEOF'
+            class FileProcessor:
+                def count_lines(self, filepath):
+                    with open(filepath) as f:
+                        return len(f.readlines())
+            PYEOF
+            echo -e "line1\\nline2\\nline3" > sample.txt
+            cat > test_processor.py << 'PYEOF'
+            from processor import FileProcessor
+            p = FileProcessor()
+            assert p.count_lines('sample.txt') == 3
+            assert p.count_lines('nonexistent.txt') == -1
+            print("ALL_TESTS_PASSED")
+            PYEOF
+        """),
+        "test_cmd": "python3 test_processor.py",
+    },
+    {
+        "instance_id": "swe-mini-004",
+        "problem_statement": (
+            "The `MathHelper.factorial` function gives wrong results for n=0. "
+            "factorial(0) should return 1, but it currently returns 0.\n\n"
+            "Expected: factorial(0) => 1\n"
+            "Actual:   factorial(0) => 0"
+        ),
+        "setup_code": textwrap.dedent("""\
+            cat > mathhelper.py << 'PYEOF'
+            class MathHelper:
+                def factorial(self, n):
+                    result = 0
+                    for i in range(1, n + 1):
+                        result *= i
+                    return result
+            PYEOF
+            cat > test_math.py << 'PYEOF'
+            from mathhelper import MathHelper
+            m = MathHelper()
+            assert m.factorial(0) == 1
+            assert m.factorial(1) == 1
+            assert m.factorial(5) == 120
+            assert m.factorial(10) == 3628800
+            print("ALL_TESTS_PASSED")
+            PYEOF
+        """),
+        "test_cmd": "python3 test_math.py",
+    },
+    {
+        "instance_id": "swe-mini-005",
+        "problem_statement": (
+            "The `ListUtils.unique` method should return unique elements in the "
+            "order they first appear, but it currently sorts them alphabetically.\n\n"
+            "Expected: unique(['b', 'a', 'b', 'c', 'a']) => ['b', 'a', 'c']\n"
+            "Actual:   unique(['b', 'a', 'b', 'c', 'a']) => ['a', 'b', 'c']"
+        ),
+        "setup_code": textwrap.dedent("""\
+            cat > listutils.py << 'PYEOF'
+            class ListUtils:
+                def unique(self, items):
+                    return sorted(set(items))
+            PYEOF
+            cat > test_list.py << 'PYEOF'
+            from listutils import ListUtils
+            u = ListUtils()
+            assert u.unique(['b', 'a', 'b', 'c', 'a']) == ['b', 'a', 'c']
+            assert u.unique([3, 1, 2, 1, 3]) == [3, 1, 2]
+            assert u.unique([]) == []
+            assert u.unique([1]) == [1]
+            print("ALL_TESTS_PASSED")
+            PYEOF
+        """),
+        "test_cmd": "python3 test_list.py",
+    },
+]
+
+
+class SWEBenchBenchmark(BenchmarkSuite):
+    """SWE-Bench: resolve real-world GitHub issues."""
+
+    name = "SWE-Bench"
+    description = "Resolve GitHub issues by editing source code"
+    category = "coding"
+
+    def load_dataset(self) -> list[dict[str, Any]]:
+        jsonl_path = Path(self.data_dir) / "swe_bench.jsonl"
+        if jsonl_path.exists():
+            problems: list[dict[str, Any]] = []
+            with open(jsonl_path) as fh:
+                for line in fh:
+                    line = line.strip()
+                    if line:
+                        problems.append(json.loads(line))
+            if self.verbose:
+                print(f"  Loaded {len(problems)} problems from {jsonl_path}")
+            return problems
+
+        if self.verbose:
+            print(f"  {jsonl_path} not found — using built-in 5-problem subset")
+        return list(_BUILTIN_PROBLEMS)
+
+    def build_prompt(self, problem: dict[str, Any]) -> str:
+        stmt = problem["problem_statement"]
+        return (
+            f"You are a software engineer fixing a bug. Here is the issue:\n\n"
+            f"{stmt}\n\n"
+            f"Read the code in the workspace, find the bug, and fix it. "
+            f"Do not modify the test files."
+        )
+
+    def setup_workspace(self, problem: dict[str, Any], workspace: str) -> None:
+        setup = problem.get("setup_code", "")
+        if setup:
+            self._run_shell(setup, cwd=workspace, timeout=30.0)
+
+    def evaluate(self, problem: dict[str, Any], workspace: str) -> BenchmarkResult:
+        pid = problem.get("instance_id", "unknown")
+        test_cmd = problem.get("test_cmd", "echo FAIL && exit 1")
+        code, output = self._run_shell(test_cmd, cwd=workspace, timeout=30.0)
+        passed = code == 0 and "ALL_TESTS_PASSED" in output
+        return BenchmarkResult(
+            problem_id=pid, passed=passed, actual=output[:500],
+            error="" if passed else output[:500],
+        )