wuyang
62 lines
1.4 KiB
Markdown
62 lines
1.4 KiB
Markdown
# Paper: Agent Safety Benchmark Taxonomy
|
|
|
|
---
|
|
type: paper
|
|
title: "Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents"
|
|
authors:
|
|
year: 2026
|
|
venue:
|
|
url: https://arxiv.org/html/2605.16282v1
|
|
code_url:
|
|
source: arxiv
|
|
collected_at: 2026-07-08
|
|
status: queued
|
|
relevance: medium
|
|
topics:
|
|
- agent-safety
|
|
- agent-evaluation
|
|
methods:
|
|
- benchmark-taxonomy
|
|
- coverage-matrix
|
|
benchmarks:
|
|
- agent-safety-benchmarks
|
|
models:
|
|
-
|
|
datasets:
|
|
-
|
|
related_concepts:
|
|
- guardrail
|
|
- red-teaming
|
|
related_jobs:
|
|
- 2026-07-08-tencent-cloud-ai-agent-test-engineer
|
|
related_experiments:
|
|
-
|
|
related_projects:
|
|
-
|
|
---
|
|
|
|
## One-line Takeaway
|
|
|
|
Agent safety benchmark 还没有收敛，做评估时不能只依赖单一 benchmark。
|
|
|
|
## Problem
|
|
|
|
不同安全 benchmark 的风险覆盖、任务环境、评估方法和结论可能不一致。
|
|
|
|
## Core Idea
|
|
|
|
该论文整理 2023-2026 年的 behavioral agent-safety benchmarks，并用六轴 taxonomy 和 coverage matrix 分析方法选择如何影响结论。
|
|
|
|
## Evidence
|
|
|
|
搜索摘要显示论文 catalog 了 40 个 behavioral agent-safety benchmarks 和 5 个相邻 evaluator/defense/dataset artifact。
|
|
|
|
## Useful For Us
|
|
|
|
- 适合作为选择 Agent safety benchmark 的二级阅读材料。
|
|
- 与 OpenAgentSafety、AgentRx、腾讯 Agent 测试岗位形成交叉验证。
|
|
|
|
## Follow-up Experiments
|
|
|
|
- 根据 taxonomy 给知识库建立自己的 Agent evaluation source map。
|