You have a prioritization call to make by Friday, no time to recruit participants, and a tool that will produce a hundred “respondents” before lunch. The panel’s overall answer looks right. Whether its answer for each customer group is right is a different question, and two new studies say the usual check cannot tell you. Synthetic panels can erase real differences between groups and invent ones that don’t exist, and you find out which only by running the human study they were meant to replace.
One question, one label, fewer differences
The first study comes from the Nielsen Norman Group. In “Synthetic Users vs. Human Respondents” , published on 16 October 2026, Rachel Banawa put the same terminology question to 604 human professionals and 120 synthetic respondents from the Synthetic Users platform. Both groups reached the same main conclusion: UX terms dominated.
The headline held. The detail did not. NN/g reports that the synthetic respondents “collapsed onto a dozen labels”, while the humans spread across more than 50. Overall, 84% of synthetic respondents used UX terms against about 70% of humans.
Split by role, the gap widens. Among synthetic product managers, 93% used UX terms. Among real ones, 44% did. The model had pulled everyone toward the dominant label, so the difference between roles shrank. NN/g’s phrase for it is mode collapse: it “reproduces the most common data point but drops the long tail”. Synthetic data, it adds, may show that groups differ “but it may understate those differences”.
The opposite failure: caricature
If that were the whole story, the fix would be simple: assume the panel flattens things and discount accordingly. The second study blocks that route. In “When Synthetic Users Fail” , Zihan Chen, Di Zhu and Lei Nico Zheng tested four models, Claude Haiku 4.5, Sonnet 4.6 and two Llama versions, against General Social Survey and World Values Survey questions. Their intro notes that “product teams query synthetic personas to prioritize features”.
On identity-linked attitudes, the authors write that “we find the opposite” of flattening. Segment gaps were inflated two to four times, and the simulated population is “a caricature”. Take confidence in banks. Among real GSS respondents, the share with “a great deal” of confidence ranges only from 13% to 24% across political categories. The model treated politics as about 67% determinative. Splits like that sent Sonnet 4.6 to the wrong top segment in 50% of GSS and 72% of WVS question and axis pairs.
A team that checks only the overall answer can be misled twice: the panel may erase a real difference or invent a fake one.
The paper’s own verdict is that a team checking only aggregate fidelity, “the common practice”, can be misled “in two ways at once”. Phantom splits showed up in 28% of WVS pairs, and a larger model fixed neither failure.
Why the opposite directions may not be a paradox
There is a fair objection here. The studies are not comparable. NN/g ran one open-ended question about professional roles through a commercial platform that simulates interviews, and it sometimes reworded the question. Chen and colleagues prompted raw models with demographics on public-opinion surveys. Models may well flatten job attributes while exaggerating identity attributes, which would make the two results consistent rather than contradictory.
Both sources also credit synthetic data where it works. A distribution-style prompt in the Chen paper reproduced aggregate distributions closely, and NN/g’s own human sample leaned toward UX-literate people, partly recruited through its channels. If you only need the most common answer, the panel is adequate.
Pipeline’s claim goes further than either study. Neither measured real product teams prioritizing real features. Read across, the pair suggests that in product research the direction of a segment-level error is not something you can know in advance, and that the aggregate match gives no hint of it. That is a leap across two setups, not a finding. But it is the one a team makes when it uses a panel to choose between segments.
What a small human sample buys
For the researcher or PM about to pick a segment from a synthetic panel, “the headline matched real data” can no longer count as validation. The cheaper check is a small human sample split along the same segments, run before anyone trusts a between-group difference. It needn’t be large. Chen and colleagues worked with about 100 respondents per question, and NN/g’s advice is to treat synthetic findings as a starting point.
NN/g also warns against assuming that prompting for variety produces representative variety. Our earlier piece on agreeableness found the panel too pleasant to test anything. This one finds it too confident about who differs from whom. A panel can be right about the crowd and wrong about every group within it.
