wuyang
64 lines
1.5 KiB
Markdown
64 lines
1.5 KiB
Markdown
# Paper: Dialogue-SWEBench
|
|
|
|
---
|
|
type: paper
|
|
title: "Dialogue-SWEBench: A Benchmark for Dialogue-Driven Coding Agents"
|
|
authors: Brendan King, Jeffrey Flanigan
|
|
year: 2026
|
|
venue:
|
|
url: https://arxiv.org/html/2606.13995v1
|
|
code_url: https://jlab-nlp.github.io/dialogue-swe-bench/
|
|
source: arxiv
|
|
collected_at: 2026-07-08
|
|
status: skimmed
|
|
relevance: high
|
|
topics:
|
|
- coding-agent
|
|
- human-in-the-loop
|
|
- agent-evaluation
|
|
methods:
|
|
- user-simulator
|
|
- dialogue-benchmark
|
|
- schema-guided-agent
|
|
benchmarks:
|
|
- Dialogue-SWEBench
|
|
models:
|
|
-
|
|
datasets:
|
|
- SWE-Bench Verified
|
|
related_concepts:
|
|
- human-in-the-loop
|
|
- coding-agent
|
|
related_jobs:
|
|
- 2026-07-08-baidu-aidu-agent-fullstack-engineer-beijing
|
|
related_experiments:
|
|
-
|
|
related_projects:
|
|
-
|
|
---
|
|
|
|
## One-line Takeaway
|
|
|
|
Coding agent 的真实能力不能只看自动修复，还要看它能不能通过对话澄清需求。
|
|
|
|
## Problem
|
|
|
|
现实中的 coding assistant 经常和用户交互，但主流 benchmark 多把它们评估为全自动系统。
|
|
|
|
## Core Idea
|
|
|
|
Dialogue-SWEBench 使用 persona-grounded user simulator，把 SWE task 改造成多轮对话评估，并提出 schema-guided agent 改善对话能力。
|
|
|
|
## Evidence
|
|
|
|
论文报告 schema-guided agent 相比强 baseline 提升 3-14%，并指出更强 coding model 不一定是更强 dialogue agent。
|
|
|
|
## Useful For Us
|
|
|
|
- 对“Agent 何时应该提问”很有启发。
|
|
- 可以作为 coding-agent / human-in-the-loop 评估维度。
|
|
|
|
## Follow-up Experiments
|
|
|
|
- 给本知识库的 coding-agent 项目设计“澄清问题质量”评估。
|