agent/papers/paper-insights.md

Paper Insights

status: seed

这个文件沉淀论文阅读后的研究判断。单篇论文放在 papers/items/，这里记录跨论文的趋势和结论。

Current Snapshot

• date: 2026-07-27
• baseline scope: arXiv Agent-related papers from 2025-07-08 to 2026-07-08
• incremental scope: 2026-07-09 to 2026-07-27
• incremental candidates seen: 249
• incremental high-relevance candidates: 157
• incremental new paper items: 151
• incremental reserve candidates: 92
• total paper items after refresh: 1125
• baseline summary: 2026-07-08 corpus
• incremental summary: 2026-07-27 corpus increment
• caveat: high-recall automated sweep; most items are queued and need human skim.

Strong Signals

• Agent safety/eval 正在转向真实工具、多轮任务、trajectory、失败定位和系统级 guardrail。
• Memory 不能只看事实召回；更重要的是是否改善流程执行和跨任务决策。
• Coding-agent benchmark 正在从单 issue patch 转向长程软件演进和人机对话。
• JD 高频词和论文趋势高度重合：memory、eval、planning/tool use、coding agent。
• 最近一年 Agent 论文的重心明显落在 evaluation、tool use、memory、safety、coding agent、computer-use/GUI agent。
• 2026-05 到 2026-07 的论文密度明显上升，说明 Agent research 正在快速扩张。
• 最新增量继续把评测推向 protocol validity、动态工具环境、trajectory 诊断、确定性 verification、成本和部署条件。
• 组件评测需要同时报告 gains 与 regressions；平均提升可能掩盖原本成功任务被新组件破坏。

Weak or Unproven Claims

• “加 memory 就更强”不成立；EvoMemBench 显示长上下文 baseline 很强，memory 价值依赖任务。
• 单一 safety benchmark 不足以说明 Agent 安全；不同 benchmark 的方法差异可能改变结论。
• SWE-Bench 类单点任务不足以代表真实 coding agent 生产力。
• 自动标签是召回导向，不能直接当作精确分类；后续需要人工把高价值论文升到 skimmed。

Mature Directions

• Agent evaluation harness
• Memory benchmark and memory architecture comparison
• Coding agent benchmark beyond SWE-Bench
• Agent failure diagnosis and trajectory analysis
• Runtime governance and agent security
• Computer-use / GUI agent benchmark

Emerging Directions

• Dialogue-driven coding agents
• Stateful enterprise task benchmarks
• Agent safety benchmark taxonomy
• Skill/prompt optimization as trainable artifacts
• Agent memory poisoning and memory provenance
• Workflow-agent and enterprise automation benchmarks
• Multi-agent shared memory and governance
• Agentic RAG for technical/scientific literature

Experiment Candidates

Idea	Related Papers	Metric	Notes
Mini Agent Safety Suite	OpenAgentSafety, Agent safety benchmark taxonomy	unsafe action rate, refusal precision, tool misuse	选 10-20 个本地工具任务先跑
Memory vs Long Context	Memory survey, EvoMemBench, STATE-Bench	success rate, token cost, latency, contradiction rate	必须包含 long-context baseline
Long-horizon Coding Eval	SWE-EVO, Dialogue-SWEBench	task success, fix rate, clarification quality	从真实 repo release notes 构造小任务
Failure Step Diagnosis	AgentRx, OpenAgentSafety	critical failure step accuracy	需要统一 trace schema
GUI Agent Safety	OSWorld2.0, MacAgentBench, computer-use papers	irreversible action rate, prompt-injection catch rate	需要 sandbox 和人工确认策略
Agent Memory Security	memory poisoning / forged reasoning papers	poisoned recall rate, provenance coverage	和长期记忆系统强相关

Links Back to Knowledge Base

• docs to update: evaluation, memory, coding-agent, safety
• experiments to create: memory comparison, mini safety suite, coding-agent eval, GUI-agent safety eval
• projects affected: future front-end/indexing project, coding-agent experiments
• job skills connected: agent-evaluation, memory, coding-agent, tool-use, planning