Under the AI answer sits a pair of thumbs, and the moment you would click one is the moment the answer arrives — not three days later, when you would know whether it actually helped. A Microsoft Research experiment posted on 22 October 2026 shows how much that timing matters. A writing assistant that made people work before it helped them produced better essays. Those same people rated it as the less helpful assistant, and the rating is what most products measure.
The scaffold won on work and lost on feeling
Kewenig, Tankelevitch, Rintel, Sarkar and Sellen ran a randomized study on Prolific, the paid-participant platform. They analyzed 297 people (144 control, 153 scaffold) who wrote 594 policy essays with a GPT-5.3 assistant. The scaffold asked first about the goal — message, target reader, desired outcome, scope — and then about how to break the task down.
Scaffolded writers scored higher on the writing-quality composite: g = 0.29 on the first task, g = 0.21 on a second task where nobody had a scaffold. Their prompts on that second task scored 14.43 against 11.07. About three days later, 64.6% of scaffold users recalled the first task’s topic, against 45.6% of the others.
Then came the questionnaire. Scaffolded participants rated the AI as less helpful, felt less in control and reported more workload. The authors name the tension themselves: scaffolded participants “performed better yet rated the AI as less helpful.” The explanation: offloading “feels helpful precisely because it removes the effort that constitutes engagement.”
Thumbs are taken at the worst moment
The study measured a post-task questionnaire, in one session. Read across to shipped products, that suggests that the instruments teams already use would score such a scaffold last. No source here measured a scaffold’s thumbs-up rate or A/B result in a live product.
The instruments are not exotic. In an ICSI and eBay benchmark of 40 products with generative AI features, 32 had a feedback mechanism, with binary thumbs the most common. The paper, “Designing the Future of User Feedback for Generative AI,” finds most are generic, aimed at overall satisfaction or bug reports. Each fires at the instant of delivery. A thumb is a verdict on how the answer felt, and a design that asks you to work tends to feel worse than one that does the work for you.
Consider the product designer who adds a “what are you trying to achieve?” step before an AI drafts anything. That version will likely lose on thumbs-up rate and perceived helpfulness, even if the documents it yields are better. Without a measure that arrives later, the step gets cut in the next iteration.
A thumb is a verdict on how the answer felt, and a design that asks you to work tends to feel worse than one that does the work for you.
OpenAI already showed where the signal leads
The end state is documented. In April 2025 OpenAI’s GPT-4o update, as Simon Willison quoted the postmortem , “introduced an additional reward signal based on user feedback—thumbs-up and thumbs-down data from ChatGPT.” OpenAI wrote that this “weakened the influence of our primary reward signal,” and that “User feedback in particular can sometimes favor more agreeable responses, likely amplifying the shift we saw.” The update was rolled back.
That case is a model-training loop, not a feature-level product decision, so it does not prove a scaffold would be killed by an A/B test. What it shows is the direction of the bias: when people vote in the moment, agreeable wins — the same pull behind users preferring a reasoning format that verifies worse . The ICSI and eBay paper cites the same episode.
Friction is not a free win
The counterpoint is serious. Friction does not reliably improve outcomes. In a 917-person experiment , the best interventions improved users’ calibration but produced no task-performance gain; its authors call monitoring and performance “separable design targets.” A 24-person study found all six friction designs raised workload and task time, and the same constraint helped or hindered depending on users’ goals.
The Microsoft study has limits too. It was not preregistered. Its LLM judge agreed with human raters only moderately (ICC 0.456), and the quality effects are small. Half of the recall cohort, 50.6%, had no scored recall result. The authors also concede that “a tool that feels less helpful is likely to be declined when it is optional.” Low ratings may simply mean people will not use it.
That last point cuts both ways. If users decline the scaffold, the thumbs were telling the truth about adoption. But adoption and benefit are different questions, and only one of them shows up in a rating widget. A team testing a goal-setting step, as the ETH Zurich and DeepMind generate-button experiment did with timing, needs to look at what happens after the session: the quality of the finished document, the prompts people write next time, what they remember on Thursday.
None of that arrives with the answer. Until teams build it, the thumbs will keep judging the one thing they can see.
