# Paper: Counsel: A Meta-Evaluation Dataset for Agentic Tasks

---
type: paper
title: "Counsel: A Meta-Evaluation Dataset for Agentic Tasks"
authors: Sashank Pisupati, Henry Broomfield, Eujeong Choi, Antonia Calvi, Charlie Wang, Roman Engeler, Max Bartolo, Patrick Lewis
year: 2026
venue: arXiv
url: https://arxiv.org/abs/2606.21627
code_url:
source: arxiv
collected_at: 2026-07-08
published_at: 2026-06-19
updated_at: 2026-06-19
status: queued
relevance: high
topics:
  - agent-evaluation
  - agent-safety
  - coding-agent
  - reasoning
methods:
  -
benchmarks:
  -
models:
  -
datasets:
  - cs.AI
  - cs.LG
related_concepts:
  -
related_jobs:
  -
related_experiments:
  -
related_projects:
  -
collection_score: 20
collection_queries: agent-evaluation, coding-agent
---

## One-line Takeaway

Auto-collected from arXiv because it matched the Agent collection queries. Needs human skim.

## Why Collected

- matched queries: agent-evaluation, coding-agent
- inferred topics: agent-evaluation, agent-safety, coding-agent, reasoning
- arXiv categories: cs.AI, cs.LG
- collection score: 20

## Review Checklist

- Does this paper directly inform Agent architecture, evaluation, memory, tools, safety, coding agents, GUI/browser agents, or multi-agent workflows?
- Does it include a benchmark, dataset, code, or reproducible experimental setup?
- Should it be promoted from `queued` to `skimmed` or `summarized`?

## Links

- arXiv: https://arxiv.org/abs/2606.21627