Same model, opposite mistakes

We use one AI model to simulate consumers in Mexico, the Philippines and the United States. In each country, the simulated respondents answer questions from the national consumer survey, in the survey’s own language and wording. Compared with the published results, the model was not simply accurate or inaccurate. Its answers leaned in different directions: too gloomy in Mexico, too hopeful in the Philippines and the United States. A single accuracy score would not tell you which way it leans in your market.

The same model leaned both ways

Each Prefield market has its own simulated population built from official statistics, its own interview language, and its own reference survey. The reference is the consumer-confidence survey that the country’s statistics agency, central bank or university already runs. We asked 600 simulated respondents in each market the reference questions, then compared their answers with the latest published results. The underlying AI model was the same in all three markets.

Interviews used Mexican Spanish, formal Filipino and English. INEGI’s survey covers 32 major Mexican cities, so the Mexican comparison uses urban respondents only.

In Mexico, the panel’s index was 38.9 against INEGI’s published 46.6. In the Philippines, the central bank’s survey found far more pessimists than optimists about current economic conditions, a balance of −71. The panel’s balance was −20.

The three measures use different scales and periods, so the sizes cannot be compared across rows. The directions are the finding: the same model was too pessimistic in one market and too optimistic in the other two.

Within a market, the lean changes by question

Even a country’s overall direction simplifies what happens underneath.

• Mexico. The panel was more pessimistic than INEGI’s respondents about household finances and the national economy, and almost exactly right about rising prices. Yet it was more likely to say that someone in the household would buy a car or a home within two years.
• The Philippines. The panel was more optimistic on all 12 questions we calibrate. The gaps were largest for current economic conditions and for plans to buy big-ticket items over the next year.
• The United States. On balance, the panel expected business conditions to improve over the next year and interest rates to fall. The people the University of Michigan surveyed expected business conditions to worsen and rates to rise. They also clearly expected more unemployment, while the panel showed almost no lean.

The US examples look less like a failure to imitate people than a gap in what the model believes about the current economy. Corrections for those questions are large and shift as conditions change. When we refitted our US corrections against the same month a year earlier, they moved almost five times as much as Mexico’s did.

A matching headline can hide wrong answers

After the June comparison, we changed how our US interviews present questions and record answers. In a later comparison for July, using a separate panel of 300, the raw headline index landed within 0.3 points of the published figure. On that measure alone, it beat every corrected version we tested. It also beat simply reusing the previous month’s published result.

Underneath, the answers were still wrong. We compared 14 questions with the Michigan survey and the New York Fed’s Survey of Consumer Expectations. To match the published answer distributions, an average of 23% of simulated responses would have needed to change to a different answer. The difference in sentiment between Republicans and Democrats was about a third of the published gap. Errors in opposite directions had cancelled out inside the index.

A correction fitted on the previous month’s data reversed the picture. It reduced the distribution error by about four fifths and brought the partisan gap within six points of the published one, yet its headline index missed by about two points. Which version is more accurate depends on whether you need one headline figure, the answers to each question, or the differences between groups. Those are separate tests.

Groups come out in the right order, but too close together

Reports often compare segments: younger and older people, women and men, supporters of different parties. Here the pattern was consistent. The panel usually ranked groups in the right order but placed them too close together.

• Mexico. The ordering was right by age, sex and employment. The panel reproduced 58% of the published difference across age groups and about 30% of the difference between women and men. By education, the ordering was wrong. The published differences between regions almost disappeared.
• The United States. Our first setup reproduced about a fifth of the published gap in economic sentiment between Republicans and Democrats. Changes to how questions are asked raised that to about three fifths, with the ordering correct.

Our Mexico reports therefore treat subgroup differences as directional and make no calibrated claim by region. In each comparison above, the simulated gap between groups was smaller than the published one.

Why a single accuracy score cannot show this

Synthetic research is often judged by one figure averaged over many questions. Sometimes that figure is set against how much two runs of the same human survey disagree. The comparison is fair as far as it goes: human surveys are measurements with their own error, not perfect answers.

An average error, however, shows how large the misses are, not which way they lean. Random disagreement between surveys cannot be corrected. A consistent lean can be measured and reduced, but only against a trusted local reference and only for questions like that reference. An average over many studies describes those studies. It does not show how the model behaves in a market, language or type of question they did not include.

How we use these measurements

• Fit corrections market by market. Each market’s corrections are fitted against its own reference survey. They are never transferred to another country or interview language.
• Apply them narrowly. A correction applies only when a study asks a reference question in the reference wording. Questions about your product, price or message are not calibrated this way, and their results remain uncorrected estimates.
• Tie them to a period. Leans shift as conditions change, so each correction is tied to a reference period and refitted against newer releases.
• Show the limits. Each report carries its market’s fidelity notes: what was compared, when, and which comparisons the evidence does not support.

What this does and does not show

Every comparison here is retrospective: the published results existed when we scored the panel. A corrected result matches its reference by construction, so it is not evidence of forecasting skill. Retrospective comparisons can also be affected by what the model learned from earlier releases.

In our one comparison against a later month in the United States, the best corrected method beat simply reusing the previous month’s published answers by about 3% on distribution error. One month does not establish a reliable advantage.

The stronger test is a forecast sealed before the official figures exist, then scored in the open. Our first covers Mexico’s consumer confidence index for October 2026, which INEGI publishes on 5 November. Its registration will appear on our track record page. A registration shows when a forecast was committed, not whether it is accurate.

Questions to ask of any synthetic panel

1. Which official source was it compared with in my market, and in which language?
2. Which way does the raw panel lean there, and on which questions?
3. Does the accuracy figure describe a headline, the answers to individual questions, or differences between groups?
4. Were the reference results already published when the comparison was made?
5. Which subgroup comparisons does the evidence support?

Honest answers will rarely fit into one number. That does not weaken the evidence. It is what makes the evidence usable.