wuyang
66 lines
1.6 KiB
Markdown
66 lines
1.6 KiB
Markdown
# Paper: SWE-EVO
|
|
|
|
---
|
|
type: paper
|
|
title: "SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios"
|
|
authors: Minh Vu Thai Pham, Tue Le, Dung Nguyen Manh, Huy Nhat Phan, Nghi D. Q. Bui
|
|
year: 2026
|
|
venue:
|
|
url: https://arxiv.org/html/2512.18470v5
|
|
code_url: https://github.com/SWE-EVO/SWE-EVO
|
|
source: arxiv
|
|
collected_at: 2026-07-08
|
|
status: skimmed
|
|
relevance: high
|
|
topics:
|
|
- coding-agent
|
|
- agent-evaluation
|
|
methods:
|
|
- long-horizon-software-evolution
|
|
- benchmark
|
|
benchmarks:
|
|
- SWE-EVO
|
|
models:
|
|
- OpenAI
|
|
- DeepSeek
|
|
- Zhipu
|
|
- Qwen
|
|
- Moonshot
|
|
datasets:
|
|
- release-notes
|
|
related_concepts:
|
|
- coding-agent
|
|
- long-horizon-task
|
|
related_jobs:
|
|
- 2026-07-08-deepseek-agent-hiring-wave-beijing-hangzhou
|
|
related_experiments:
|
|
-
|
|
related_projects:
|
|
-
|
|
---
|
|
|
|
## One-line Takeaway
|
|
|
|
Coding agent 评估正在从单 issue 修 bug 转向多文件、多 PR、版本演进式长任务。
|
|
|
|
## Problem
|
|
|
|
SWE-Bench 类任务偏单点 issue，而真实工程经常要求根据 release notes 进行系统性版本演进。
|
|
|
|
## Core Idea
|
|
|
|
SWE-EVO 从成熟 Python 项目的 release notes 构造 48 个长程任务，平均涉及 21 个文件，并提出 Fix Rate 捕捉部分进展。
|
|
|
|
## Evidence
|
|
|
|
论文显示强模型在 SWE-EVO 上的解决率远低于 SWE-Bench Verified，失败更多来自语义理解和长期指令跟随，而不是简单工具操作。
|
|
|
|
## Useful For Us
|
|
|
|
- 适合指导 coding-agent 项目的内部评测集设计。
|
|
- 和 DeepSeek JD 中的 AI coding tool 使用、Agent infra、评测闭环相关。
|
|
|
|
## Follow-up Experiments
|
|
|
|
- 选一个开源 Python 项目，用 release notes 构造小规模 SWE-EVO 风格任务。
|