AI & softwareResearch findings
SciResearch-Eval · Introduction of a Benchmark for Scientific Judgment in Autonomous Experimental Agents
Original published 2026.11.17 03:27 KSTAdded to Blackdeck 2026.11.17 23:23 KST
What has changed?
The first benchmark that combines agent-designed experiments, a deliberately flawed simulator, and calibrated interval scoring against ground truth.
Key points
- Introduces SciResearch-Eval, a benchmark for scientific judgment in autonomous agents
- Agents design experiments and report 95% confidence intervals using a flawed simulator
- Calibrated interval accuracy varies widely across models, especially when physics is missing
Connect it to your work
When the simulator lacks physical accuracy, agents often report incorrect values while claiming 95% confidence, with only 20–62% of intervals containing thetrue
What remains unproven
The abstract reports simulation results; real-world performance is not established by this input.
Originals used for this note
Links open the original publisher in a new tab.