wuyang
77 lines
4.0 KiB
Markdown
77 lines
4.0 KiB
Markdown
# Paper Insights
|
|
|
|
status: seed
|
|
|
|
这个文件沉淀论文阅读后的研究判断。单篇论文放在 `papers/items/`，这里记录跨论文的趋势和结论。
|
|
|
|
## Current Snapshot
|
|
|
|
- date: 2026-07-27
|
|
- baseline scope: arXiv Agent-related papers from 2025-07-08 to 2026-07-08
|
|
- incremental scope: 2026-07-09 to 2026-07-27
|
|
- incremental candidates seen: 249
|
|
- incremental high-relevance candidates: 157
|
|
- incremental new paper items: 151
|
|
- incremental reserve candidates: 92
|
|
- total paper items after refresh: 1125
|
|
- baseline summary: [2026-07-08 corpus](corpus-summary-2026-07-08.md)
|
|
- incremental summary: [2026-07-27 corpus increment](corpus-summary-2026-07-27.md)
|
|
- caveat: high-recall automated sweep; most items are `queued` and need human skim.
|
|
|
|
## Strong Signals
|
|
|
|
- Agent safety/eval 正在转向真实工具、多轮任务、trajectory、失败定位和系统级 guardrail。
|
|
- Memory 不能只看事实召回；更重要的是是否改善流程执行和跨任务决策。
|
|
- Coding-agent benchmark 正在从单 issue patch 转向长程软件演进和人机对话。
|
|
- JD 高频词和论文趋势高度重合：memory、eval、planning/tool use、coding agent。
|
|
- 最近一年 Agent 论文的重心明显落在 evaluation、tool use、memory、safety、coding agent、computer-use/GUI agent。
|
|
- 2026-05 到 2026-07 的论文密度明显上升，说明 Agent research 正在快速扩张。
|
|
- 最新增量继续把评测推向 protocol validity、动态工具环境、trajectory 诊断、确定性
|
|
verification、成本和部署条件。
|
|
- 组件评测需要同时报告 gains 与 regressions；平均提升可能掩盖原本成功任务被新组件破坏。
|
|
|
|
## Weak or Unproven Claims
|
|
|
|
- “加 memory 就更强”不成立；EvoMemBench 显示长上下文 baseline 很强，memory 价值依赖任务。
|
|
- 单一 safety benchmark 不足以说明 Agent 安全；不同 benchmark 的方法差异可能改变结论。
|
|
- SWE-Bench 类单点任务不足以代表真实 coding agent 生产力。
|
|
- 自动标签是召回导向，不能直接当作精确分类；后续需要人工把高价值论文升到 `skimmed`。
|
|
|
|
## Mature Directions
|
|
|
|
- Agent evaluation harness
|
|
- Memory benchmark and memory architecture comparison
|
|
- Coding agent benchmark beyond SWE-Bench
|
|
- Agent failure diagnosis and trajectory analysis
|
|
- Runtime governance and agent security
|
|
- Computer-use / GUI agent benchmark
|
|
|
|
## Emerging Directions
|
|
|
|
- Dialogue-driven coding agents
|
|
- Stateful enterprise task benchmarks
|
|
- Agent safety benchmark taxonomy
|
|
- Skill/prompt optimization as trainable artifacts
|
|
- Agent memory poisoning and memory provenance
|
|
- Workflow-agent and enterprise automation benchmarks
|
|
- Multi-agent shared memory and governance
|
|
- Agentic RAG for technical/scientific literature
|
|
|
|
## Experiment Candidates
|
|
|
|
| Idea | Related Papers | Metric | Notes |
|
|
| --- | --- | --- | --- |
|
|
| Mini Agent Safety Suite | OpenAgentSafety, Agent safety benchmark taxonomy | unsafe action rate, refusal precision, tool misuse | 选 10-20 个本地工具任务先跑 |
|
|
| Memory vs Long Context | Memory survey, EvoMemBench, STATE-Bench | success rate, token cost, latency, contradiction rate | 必须包含 long-context baseline |
|
|
| Long-horizon Coding Eval | SWE-EVO, Dialogue-SWEBench | task success, fix rate, clarification quality | 从真实 repo release notes 构造小任务 |
|
|
| Failure Step Diagnosis | AgentRx, OpenAgentSafety | critical failure step accuracy | 需要统一 trace schema |
|
|
| GUI Agent Safety | OSWorld2.0, MacAgentBench, computer-use papers | irreversible action rate, prompt-injection catch rate | 需要 sandbox 和人工确认策略 |
|
|
| Agent Memory Security | memory poisoning / forged reasoning papers | poisoned recall rate, provenance coverage | 和长期记忆系统强相关 |
|
|
|
|
## Links Back to Knowledge Base
|
|
|
|
- docs to update: evaluation, memory, coding-agent, safety
|
|
- experiments to create: memory comparison, mini safety suite, coding-agent eval, GUI-agent safety eval
|
|
- projects affected: future front-end/indexing project, coding-agent experiments
|
|
- job skills connected: agent-evaluation, memory, coding-agent, tool-use, planning
|