← Back to the feed
AI & softwareResearch findings

Test an agent on delivery, not just a well-written ticket

Original published 2026.10.16 02:57 KSTAdded to Blackdeck 2026.10.18 03:10 KST

What has changed?

The benchmark makes requirement discovery and customer outcomes part of agent evaluation.

Key points

  1. FDE-Gym evaluates agents on 95 delivery tasks with initially ambiguous requirements.
  2. It combines 20 customer scenarios across 18 industries with outcome and interaction-cost measures.
  3. The strongest realistic setup reportedly solved 14 tasks, exposing a substantial delivery gap.

Connect it to your work

Product and engineering teams can test whether agents improve an actual workflow after clarifying constraints.

What remains unproven

Results from this task set are not a universal forecast of customer-project success.

Originals used for this note

Links open the original publisher in a new tab.

  1. arXiv · authors' researchFDE-Gym: The Last Mile of AI Delivery