People crossing a dark gray plaza seen from above, with circles and a grid marking out groups

Research

Population simulation at the replication floor

Simulation has the potential to be transformative, but only if it’s accurate. Today, we’re sharing results of a comprehensive evaluation across 2,993 questions, with an industry-leading mean TVD of 7.62% and an MAE of 3.53%.

Introduction

Key takeaways

Aaru builds simulations of human behavior. Given a list of questions and a description of an audience, our system simulates how each member of that population (a simulated “agent”) would respond. The result is a set of individual profiles and their responses, so it can be analyzed, segmented, and queried just like human data.

Our customers use these simulations to decide what to build, where to invest, and what to try next. However, a simulation is only useful if it’s accurate. Crosstabs and segments are predicated on aggregate accuracy to be usable: the distribution of age, income, or any other individual trait must align with the real world. So, we tested our simulations against 2,993 questions and 13,727 answer shares, across nine industries and 186 different studies.

We’ll discuss three core capabilities from our evaluation results:

  • Aaru predicts what people say. Our sample-adjusted mean absolute error is 2.78 percentage points. Aaru’s error against survey data is comparable in magnitude to published survey-to-survey disagreement rates. Across 13,727 answer shares, simulations closely track published results, with a Pearson correlation of 0.970.

  • Aaru predicts what people do. We compare simulations with behavioral data—ground-truth transactions from a credit card panel. Across a set of brand-period observations, Aaru’s simulations ranked brands closer to real behavior than published purchase-consideration did: 92.4% of brand pairs in the same order as the purchase panel, compared with 77.8% for the survey. Compared to the survey, this is roughly a two-thirds reduction in error.

  • Aaru predicts its own errors. For each question, Aaru estimates its own error without consulting the published answers. We observe state-of-the-art performance in the ability to predict our own error rates, with a 41% lower pairwise ranking error than the current published best. Tangibly, this means Aaru can trade question coverage for higher accuracy. Keeping roughly half of questions with the highest confidence gave a 40% lower average observed error compared with the full evaluation.

We believe these results speak to the merit of our approach, and we’re excited to share our research more broadly.

Why it matters

Simulation expands our ability to understand the world around us. Much of this lies in hypotheticals: scenarios that don’t exist today, and questions that no survey was written to ask. These questions fall outside the established surface area of survey instruments, the focus of traditional quantitative research for decades. Product, pricing, perception, positioning: so many decisions turn on how people will behave.

Before a model can be trusted on what has never been asked, it must be scored on what has. This is why we begin in the most familiar place possible: with survey instruments themselves. We test our simulations against published survey results and compare their error with how much repeated surveys disagree with each other. We think of that disagreement as a replication floor. Our mean error falls within the range of disagreement reported across repeated surveys, which we see as an important initial bar for simulation.

How a simulation works

A simulation starts with an audience and the questions they will respond to. From that description, Aaru builds a population of agents—computational models of individual people—whose traits, such as age, income, and location, match the audience in aggregate. When the simulation runs, each agent responds to the questions or alternatives on its own.

From audience to modeled responses

AudienceGenerate population+ 9,997 moreSimulated agentsQuestionsRun simulationModeled responsesMicrodataAgent 01Agent 02Agent 03PreferencesObjections Possible behaviors Audience+ 9,997 moreSimulated agentsQuestionsModeled responsesMicrodataGenerate populationAgent 01Agent 02Agent 03Run simulationPreferencesObjections Possible behaviors

The result is a set of modeled responses—preferences, objections, and possible behaviors—that can be read for a single agent or for the population as a whole. In this evaluation, each input is a published survey question and its answer options. The output is microdata: one record per agent, so it can be analyzed, segmented, and queried just like data from real respondents.

Predicting what people say

The first test asks a simple question: when people answer a survey, can Aaru predict how their answers will split? We gave the simulation system the question wording, available answer options, and a description of the audience. From here, we ran the simulations and compared their predictions with published results from studies across nine different industries. Questions in these studies covered experiences (what people have done or know), attitudes (what people think and feel), intentions (what people plan or would choose), and expectations (what people predict will happen). [1]

Neither the evaluation questions nor their published answers were used to develop or tune our simulation system or confidence predictor. We selected questions from studies recorded as fielded in 2026, and external retrieval is restricted to pages stored at least one day prior to commencing fieldwork. Our core, row-level model for predicting an agent’s behavioral trajectory isn’t a language model. Certain confidential alternative data sources have been removed for the purposes of this evaluation, so results reflect a deliberately constrained version of the system.

What we asked

We randomly sampled publicly available research studies across every major industry we operate in. Our evaluation spans financial services, business-to-business, technology, healthcare, consumer packaged goods, retail, and more. Each of the nine industries contains between 191 and 573 questions, covering the research applications that our customers care about—brand health, market sizing, customer experience, products and pricing, and more. We’re building a general-purpose simulation system to answer questions across industries, and the breadth of this evaluation is a step toward that goal.

Select an industry to see three of its questions, with the published answers next to Aaru’s.

Nine industries. 2,993 questions.

Each tile is sized by the questions tested in that industry

Select an industry to see three of its questions
Each question belongs to one primary industry. The evaluation spans 186 studies, some with multiple waves. Example questions come from three different sources per industry, with errors near that industry’s typical result.

We’ll analyze 13,727 answer shares. An answer share is the proportion of respondents who select a particular answer option. For example, a question with three options selected by 50%, 30%, and 20% of respondents contributes three answer shares—one per option. Together, these shares form the question’s answer distribution.

One example comes from an Ipsos survey of U.S. adults. This survey asked about shopping experiences and whether they often felt they could get better discounts online. In the survey, 48% somewhat agreed and 26% strongly agreed. In Aaru’s simulation, exactly 48% somewhat agreed and 22% strongly agreed. [2]

Ipsos · 1,085 U.S. adults

“I often feel like I could get better discounts online”

Share of respondents
PublishedAaru
Strongly agree
Published: 26.0%Aaru: 22.0%
Somewhat agree
Published: 48.0%Aaru: 48.0%
Somewhat disagree
Published: 15.0%Aaru: 16.0%
Strongly disagree
Published: 4.0%Aaru: 5.0%
Don’t know
Published: 7.0%Aaru: 9.0%
Published survey responses and Aaru’s simulated responses, side by side. All answer options are shown. [2]

Against the replication floor

Across 13,727 answer shares, simulations closely track published results. We maintain a Pearson correlation of 0.970 and a Spearman correlation of 0.963. Within individual questions, mean Spearman correlation of ranked response options was 0.90 (median: 0.99) and modal accuracy was 87.5%.

Total variation distance (TVD) measures how much two answer distributions differ; lower is better. A TVD of 6 points means that about 6 in every 100 answers would have to change for the two results to match. Mean observed TVD was 7.62 points. Before the sampling adjustment, at least 50% of questions had a TVD of 6 points or less, and at least 90% had a TVD of 16 points or less. Published survey results include sampling noise, so we also report results after the sampling adjustment, which subtracts the expected sampling error for each question’s sample size. After the sampling adjustment, mean TVD was 5.98 points (median: 4.30) and mean absolute error (MAE) was 2.78 percentage points. Every question has equal weight in these means.

How accurate were the simulations?

2,993 questions · 13,727 answer shares · 186 studies

Predictions track published results

Every answer share, published against simulated

0.97Pearson correlationAcross all 13,727 answer shares; 94% land inside the band, within 10 points of the published share

Average distribution error

Each question’s TVD, before adjustment

5.98TVD points after sampling adjustment7.62 before adjustment

Average answer-share error

Each question’s MAE, before adjustment

2.78MAE in percentage points after sampling adjustment3.53 before adjustment

TVD measures the gap between complete answer distributions; MAE measures the average gap per answer option. Both give each question equal weight, and lower is better. Correlation measures how closely answer shares move together and is not a percentage accuracy score. [1]

How does this compare with human surveys? Imagine you conduct the exact same survey with the same population, only two weeks apart. You might expect the responses to be stable; the data says otherwise. When Callegaro and colleagues fielded the same survey twice, two weeks apart, using four U.S. sample sources with approximately 1,500 respondents per wave, we calculate mean wave-to-wave TVD of 2.3 to 12.0 points. When Redmiles and colleagues repeated a security and privacy survey on Amazon Mechanical Turk 14 months apart, we calculate a mean TVD of approximately 2.6 points across 26 binary items. [3] [4]

What does this mean? If repeated surveys disagree with each other, we shouldn’t expect any one simulation to exactly match any one human survey. Instead, we use total variation distance (TVD) to compare simulation error with disagreement between repeated human surveys. The replication floor is the bar simulations must clear before they are allowed to say anything new.

Across 2,993 comparisons between Aaru and human surveys, Aaru’s mean observed TVD was 7.62 points, within the range of a survey repeated with itself. Aaru’s observed error and the repeated surveys both include sampling noise. After sampling adjustment, which removes 1.64 points of Aaru’s error, the average TVD was 5.98 points.

Putting the error in context

5 repeated surveys · 2 studies · TVD in points · Lower is better

Aaru’s error falls within the range of a survey repeated with itself.

On average, 7.6 answers in 100 differ between Aaru and the published survey, or 6.0 once sampling noise is removed. Asking the same survey twice moves 2.3 to 12.0.

2.3–12.0 5 surveys
Aaru’s observed error and the repeated surveys both include sampling noise. We derived the intervals from resampling 104 groups of related studies 2,000 times.

Predicting what people do

Simulating what people say is critical, but in many sensitive decision-making scenarios, it’s not enough. What people say they’d consider buying and what they actually purchase frequently diverge. In a separate evaluation, we drew on some of our alternative data: card-panel purchase records. We ranked brands by purchase frequency (measured as purchase records per 1,000 active card-days) within the same category and period. We then compared those rankings with purchase-consideration data from YouGov BrandIndex and with Aaru’s predictions for the same brands, across 100 brand-period observations in 14 category-and-period groups.

Stated versus actual

These examples show where brands change order between the survey and the purchase panel, and whether Aaru’s predictions follow the purchases. Select a category to compare the rankings.

Consideration and purchases

YouGov BrandIndexMar 2024 – Feb 20257 brands

Which supermarkets do people buy from?

ALDI leads Kroger on consideration. Kroger leads on purchases.

Survey consideration

What people would consider

  1. ALDI
  2. Kroger
  3. Publix
  4. ShopRite
  5. H-E-B
  6. WinCo Foods
  7. Grocery Outlet

Observed purchases

What the panel recorded

  1. Kroger
  2. Publix
  3. ALDI
  4. H-E-B
  5. ShopRite
  6. WinCo Foods
  7. Grocery Outlet

Aaru predictions

Asked about purchases

  1. Kroger
  2. Publix
  3. ALDI
  4. H-E-B
  5. ShopRite
  6. WinCo Foods
  7. Grocery Outlet

Brand pairs ranked in the same order as the purchase panel

Each crossing above is one reversed pair · Ties count half

Survey consideration

78.6%

Aaru predictions

100.0%

Aaru puts Kroger ahead of ALDI, matching the purchase panel.

These examples show where consideration and purchases diverged. Each column ranks all brands in that group, and each line follows one brand across the columns. Where two lines cross, that pair is in a different order. The results across the full purchase evaluation follow below.

Closer to real purchases

When asked about purchasing behavior, Aaru gets the order correct in 92.4% of 340 brand pairs, compared with an accuracy of 77.8% implied by the published surveys on the same pairs, an error-rate reduction of approximately 66%. A baseline using public measures of brand size, such as store counts, gets the order right in 79.3% of the 333 pairs where a size measure was available. Aaru’s predictions are averaged across three runs before brands are ranked; the runs individually scored 91.5%, 91.8%, and 92.9%. The gain over the published survey was 14.6 percentage points (95% confidence interval: 7.2–23.0 points).

Aaru made 66% fewer ordering errors than the published survey.

An ordering error is a brand pair ranked in the opposite order from observed purchases. Predicted ties count half.

Brand pairs ranked in the same order as the purchase panel

+14.6 percentage points higher agreement when Aaru is asked about purchases, compared with the published survey (95% confidence interval: 7.2–23.0 points). Aaru’s result averages the predictions of three runs before brands are ranked; individually, the runs scored 91.5%, 91.8%, and 92.9%.

Rankings within 14 category-and-period groups. Survey and Aaru results use 340 pairs across 100 brand-period observations; the size baseline uses 333 pairs across 99. Each cell is one brand pair.

On the brands we tested, Aaru’s simulations order brands more closely to the transaction panel than stated-preference surveys do. This gives us a way to test simulations against what people actually buy.

Predicting our own errors

No simulation is right about everything. Alongside accuracy, it’s important to understand where results are likely to be accurate, and where greater caution is warranted. For each question, Aaru returns a predicted TVD alongside a probability that the error exceeds each threshold (e.g., 10 or 16 points). These estimates are made as part of the simulation, without consulting the published answers.

High and low confidence

Before it sees the published results, Aaru estimates how far its answers will be from the survey, in TVD points. A low estimate means high confidence. Once the published results are in, we can check each estimate against the actual error.

These three questions show both ends of the confidence scale and one question in between.

Three questions, three confidence levels

Aaru estimates its error before it sees the published answers · TVD in points

Natural Resources Canada · 2,774 Canadian adults without a zero-emission vehicle

Have you ever driven or ridden in a zero-emission vehicle?

Share of respondents
PublishedAaru
Yes
Published: 59.0%Aaru: 58.0%
No
Published: 39.0%Aaru: 40.0%
Don't know
Published: 2.0%Aaru: 2.0%
We chose one question from each end of the confidence scale and one in between. Across all 2,993 questions, higher estimates go with higher actual errors (rank correlation 0.67), but a single estimate can miss: WalletHub surveyed 200 people, so its own sampling noise is larger than most.

Confidence tracks error

These estimates meaningfully separate the most and least confident simulations. Keeping roughly half of questions with the highest confidence gave a 40% lower average observed error compared with the full evaluation: 4.54 versus 7.62 TVD points. In this retained subset, the share of questions with TVD above 16 points is 1.0%, compared with 9.5% in the full set.

Average error rises steadily as confidence falls, from 3.29 points in the most confident group to 13.08 in the least confident. The less confident half also holds 267 of the 283 questions with a TVD above 16 points. To compare with published methods, we use AUROC, which measures how often a randomly selected question at or above an error threshold receives a higher predicted TVD than one below it. For TVD below 16 points, Aaru achieved an AUROC of 0.845, compared with 0.736 for the strongest method reported by Wesel, Chen, and Liang. This is a 41% lower ranking error when sorting questions from most to least reliable. [5]

More confidence. Less error.

By Aaru’s confidence, set before it sees the published answers

Keeping the most confident half cuts average error by 40%.

4.54 against 7.62 TVD points for all questions. Lower is better.

Most confident half · 1,525 questions

Less confident half · 1,468 questions

Groups follow Aaru’s own error estimate for each question, made before it sees the published answers. A large miss is a TVD above 16 points. Averages give each question equal weight and are before sampling adjustment.

For customers, this provides a way to assess results beyond the prediction alone. Changing the confidence threshold allows us to trade coverage—the percentage of questions retained—for accuracy. The predictions themselves are unchanged.

What to do next

It’s faster and cheaper than ever to move from idea to reality. Despite this, the ability to execute doesn’t tell us what to do. As the cost of doing falls, the value of knowing what to do will only grow.

We founded Aaru to simulate humanity because so many decisions turn on how people will behave, and what they value. We’re building toward a future where ideas, products, policies, and messages are tested in simulation before they’re released into the world. Understanding what people will do next expands our ability to make better decisions about how to serve them. We’re excited to share more of our findings in the future.

Read the full research and methodology →

ReferencesSources cited throughout this article.View sourcesHide sources
  1. Aaru (2026). Population simulation at the replication floor. Full evaluation, methodology, sampling adjustment, and confidence results.

  2. Ipsos (2026). Ipsos Poll on Consumer Behavior. Question 5b, page 7. Survey conducted July 14–15 among 1,085 U.S. adults.

  3. Callegaro, Tsirlin, Yang, and Ma (2022). Test-retest reliability of four U.S. non-probability sample sources. AAPOR presentation, slide 14. Reported mean sums of absolute differences imply TVDs of 2.3, 2.6, 6.6, and 12.0 points.

  4. Redmiles, Kross, and Mazurek (2019). How Well Do My Results Generalize?. IEEE Symposium on Security and Privacy, Appendix VIII-B, Table XII. Mean TVD of approximately 2.6 points across 26 binary items.

  5. Wesel, Chen, and Liang (2026). Building confidence in Simile. Blog post. For predicting which questions have a TVD below 16 points, their strongest method achieved an AUROC of 0.736, against 0.845 for Aaru. The pairwise ranking error, 1 − AUROC, falls from 26.4% to 15.5%.