wuyang
65 lines
1.7 KiB
Markdown
65 lines
1.7 KiB
Markdown
# Paper: OpenAgentSafety
|
|
|
|
---
|
|
type: paper
|
|
title: OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety
|
|
authors: Sanidhya Vijayvargiya, Aditya Bharat Soni, Xuhui Zhou, Zora Zhiruo Wang, Nouha Dziri, Graham Neubig, Maarten Sap
|
|
year: 2025
|
|
venue: ICLR 2026 / IASEAI 2026
|
|
url: https://arxiv.org/abs/2507.06134
|
|
code_url:
|
|
source: arxiv
|
|
collected_at: 2026-07-08
|
|
status: skimmed
|
|
relevance: high
|
|
topics:
|
|
- agent-evaluation
|
|
- agent-safety
|
|
- tool-use
|
|
methods:
|
|
- multi-turn-agent-evaluation
|
|
- rule-based-analysis
|
|
- llm-as-judge
|
|
benchmarks:
|
|
- OpenAgentSafety
|
|
models:
|
|
-
|
|
datasets:
|
|
-
|
|
related_concepts:
|
|
- guardrail
|
|
- human-in-the-loop
|
|
related_jobs:
|
|
- 2026-07-08-baidu-aidu-agent-algorithm-engineer-beijing
|
|
- 2026-07-08-tencent-cloud-ai-agent-test-engineer
|
|
related_experiments:
|
|
-
|
|
related_projects:
|
|
-
|
|
---
|
|
|
|
## One-line Takeaway
|
|
|
|
Agent safety 评估正在从简单拒答测试转向真实工具、多用户、多轮任务和可扩展评估框架。
|
|
|
|
## Problem
|
|
|
|
真实 Agent 会调用浏览器、文件系统、shell、消息平台和代码执行环境，传统安全 benchmark 对工具、任务和对抗场景覆盖不足。
|
|
|
|
## Core Idea
|
|
|
|
OpenAgentSafety 提供模块化框架，用真实工具和多轮任务评估 Agent 在八类风险下的行为，并结合规则分析和 LLM-as-judge。
|
|
|
|
## Evidence
|
|
|
|
论文报告覆盖 350+ 多轮、多用户任务，并评估五个 LLM 在 agentic 场景下的安全表现。
|
|
|
|
## Useful For Us
|
|
|
|
- 可作为 Agent eval harness 的安全维度参考。
|
|
- 与 JD 中的 Agent 评测体系、成功率、稳定性、安全边界直接相关。
|
|
|
|
## Follow-up Experiments
|
|
|
|
- 选 10 个高风险工具任务，建立一个本地 mini OpenAgentSafety 风格测试集。
|