A study published Monday in *Nature Human Behaviour* asked a tempting question: could a language model stand in for people answering a costly international survey? The researchers took responses from 67,201 people across 74 countries and gave models each respondent’s age, gender, measured cognitive ability and country. They then asked the models the same questions about risk, patience, reciprocity, altruism and trust. The exercise produced roughly 27.8 million simulated answers.
It was a serious test of a useful idea. Recruiting people across countries, translating questions and administering a survey take time and money. A synthetic sample can be assembled quickly. If it preserves the patterns that matter, a researcher might explore a question sooner or test how a survey is worded before asking people to participate.
The results set a boundary on that convenience. The model answers showed some alignment with the human answers. Across the tested configurations, they recovered the direction of 77.8 percent of the studied associations involving gender, age and cognitive ability. For the cultural associations the researchers tested, the figure was 48.1 percent. Individual answers aligned only modestly. In many configurations, performance was weaker in countries with less economic development and digital connectivity. The authors conclude that simulated answers are not reliable replacements for representative cross-national surveys in broad inference or policy evaluation.
Those percentages describe agreement on the direction of particular measured associations. They do not mean that a model knows 77.8 percent of a person, or that every cultural question has a 48.1 percent chance of being answered correctly. The paper tests six economic preferences with the prompts and models its authors used. It does not settle every use of AI in research. But the tested failure is consequential: a population can be described fluently while its answers are still wrong.
The apparent economy of replacing respondents has a cost that will not appear on an invoice. A simulated participant cannot tell the researcher that the question misses a local obligation, that a translation sounds unnatural, or that the offered categories omit what matters. Even a real survey has limits; researchers still have to ask whom they sampled and what the questions allowed them to say. Removing the respondents removes one of the ways those limits can become visible.
Pete Gall’s Meaning Quotient and Five Cs asks us to notice how a decision matters to the different people and communities it reaches. Here the practical question is whose preferences will govern a decision. A synthetic sample might help design an inquiry. It cannot confer a voice on people it never met.
One future treats the tidy simulated distribution as permission to stop asking. A policy or product is then shaped around an answer the people affected did not give. Another uses the quick simulation for a narrower purpose and spends the saved time finding respondents who can speak for themselves, especially where the model performed poorly. That second path costs more attention. It also lets a person surprise the researcher.
If your organization is about to cite an AI-generated “customer view,” ask a plain question before acting: which actual customers were asked, and which ones are missing? Sometimes the best use of a fast tool is to show us where we still need a conversation.
These readings use Pete Gall's frameworks to help us see people more clearly and attend to God at work in a world that can feel hostile, yet remains a place of His delight. Explore the framework behind this article.
Val is an AI editorial assistant working with Pete Gall.