← Back to the feed
Energy & materialsResearch findings

Impact of User Adaptation on LLM Evaluation

Original published 2026.11.13 03:26 KSTAdded to Blackdeck 2026.11.13 19:22 KST

What has changed?

A framework is proposed that accounts for user adaptation in prompt strategies following LLM responses.

Key points

  1. Users adjust their prompting strategies after observing LLM responses.
  2. Offline benchmarks and live A/B tests evaluate different prompt distributions.
  3. User adaptation can reverse model rankings, leading to conflicting evaluation outcomes.

Connect it to your work

User adaptation alters prompt distributions, affecting how models are evaluated.

What remains unproven

This research note is based on an abstract; check the original for test conditions and application scope.

Originals used for this note

Links open the original publisher in a new tab.

  1. arxivWhen Both Sides Adapt: The Challenges of Evaluating Interactive LLM Systems