AI & softwareResearch findings
Test an agent on delivery, not just a well-written ticket
Original published 2026.10.16 02:57 KSTAdded to Blackdeck 2026.10.18 03:10 KST
What has changed?
The benchmark makes requirement discovery and customer outcomes part of agent evaluation.
Key points
- FDE-Gym evaluates agents on 95 delivery tasks with initially ambiguous requirements.
- It combines 20 customer scenarios across 18 industries with outcome and interaction-cost measures.
- The strongest realistic setup reportedly solved 14 tasks, exposing a substantial delivery gap.
Connect it to your work
Product and engineering teams can test whether agents improve an actual workflow after clarifying constraints.
What remains unproven
Results from this task set are not a universal forecast of customer-project success.
Originals used for this note
Links open the original publisher in a new tab.