wuyang
72 lines
1.9 KiB
Markdown
72 lines
1.9 KiB
Markdown
# Industry: Microsoft - STATE-Bench
|
|
|
|
---
|
|
type: industry
|
|
company: Microsoft
|
|
team: Microsoft Open Source
|
|
title: "Introducing STATE-Bench: a benchmark for AI agent memory"
|
|
url: https://opensource.microsoft.com/blog/2026/05/19/introducing-state-bench-a-benchmark-for-ai-agent-memory/
|
|
source_name: Microsoft Open Source Blog
|
|
source_type: benchmark
|
|
source_quality: official
|
|
published_at: 2026-05-19
|
|
collected_at: 2026-07-08
|
|
status: analyzed
|
|
topics:
|
|
- memory
|
|
- agent-evaluation
|
|
- enterprise-ai
|
|
implementation_signals:
|
|
- stateful-environment
|
|
- user-simulator
|
|
- deterministic-assertions
|
|
- bring-your-own-memory
|
|
product_area:
|
|
- customer-support
|
|
- travel
|
|
- shopping
|
|
models:
|
|
-
|
|
tools:
|
|
- STATE-Bench
|
|
benchmarks:
|
|
- STATE-Bench
|
|
related_papers:
|
|
- 2026-memory-agent-survey
|
|
- 2026-wang-evomembench
|
|
related_jobs:
|
|
- 2026-07-08-baidu-aidu-agent-algorithm-engineer-beijing
|
|
related_experiments:
|
|
-
|
|
related_projects:
|
|
-
|
|
evidence_level: high
|
|
relevance: high
|
|
---
|
|
|
|
## One-line Takeaway
|
|
|
|
Memory 评估应衡量 Agent 是否因经验而更会执行流程，而不是只测试远距离事实召回。
|
|
|
|
## What They Built or Claimed
|
|
|
|
Microsoft 发布 STATE-Bench，一个开源、memory-agnostic benchmark，用现实企业任务评估 Agent 是否能通过经验改进。
|
|
|
|
## Technical Signals
|
|
|
|
- architecture: 多轮对话 loop + stateful environment + deterministic assertions。
|
|
- tool use: 任务涉及查询、校验策略、计算费用、确认并执行。
|
|
- memory: bring-your-own-memory 接口。
|
|
- evaluation: 450 个任务，覆盖 travel、customer support、shopping。
|
|
- safety: 状态改变任务会带来真实成本，强调流程合规。
|
|
- deployment: open-source benchmark。
|
|
- data: 预填充数据库和用户模拟器。
|
|
|
|
## Evidence Quality
|
|
|
|
官方开源博客，包含任务数、领域、评估 loop 和 GitHub 入口，证据强。
|
|
|
|
## Gaps for Us
|
|
|
|
- 可以基于 STATE-Bench 思路做一个小型企业流程 Agent eval。
|