Refresh papers and define Agent evaluation
This commit is contained in:
@@ -0,0 +1,24 @@
|
||||
# Evaluation Tools
|
||||
|
||||
## `score_agent_runs.py`
|
||||
|
||||
汇总一个或多个 Agent 变体的 JSONL 运行结果:
|
||||
|
||||
```bash
|
||||
python3 tools/evaluation/score_agent_runs.py results.jsonl \
|
||||
--baseline agent-loop-v3 \
|
||||
--variant agent-loop-v4 \
|
||||
--format markdown
|
||||
```
|
||||
|
||||
它会分开报告:
|
||||
|
||||
- strict success 和假完成;
|
||||
- provider/infrastructure error;
|
||||
- gains 和 regressions;
|
||||
- 成功率/假完成率的 Wilson 95% 区间,以及配对差异的精确检验;
|
||||
- 工具失败与恢复;
|
||||
- 安全违规;
|
||||
- p50/p95、Token 和工具调用。
|
||||
|
||||
脚本给出的 promotion/reject 只是一条机械建议,最终决策仍需检查任务覆盖、统计区间和回归轨迹。
|
||||
Reference in New Issue
Block a user