People crossing a blue plaza seen from above, with circles and a grid marking out groups

Research

Population simulation at the replication floor

Simulation has the potential to be transformative, but only if it’s accurate. Today, we’re sharing results of a comprehensive evaluation across 2,993 questions, with an industry-leading mean TVD of 7.62% and an MAE of 3.53%.

Introduction

Key takeaways

Aaru builds simulations of human behavior. These simulations produce a population of demographically accurate agents—computational models of humans—and their answers across user-defined questions.

By modeling human behavior as a collection of agents within a population, Aaru’s simulations enable detailed analysis of their outputs, just as one may study a human sample. However, for population-level data to be useful, the population aggregates—that is, the distribution of age, income, or any other individual trait—must align with the real world. Higher-order analytic structures like crosstabs and segments are predicated on aggregate accuracy to be usable.

We evaluate on a 2,993-question dataset and discuss the evaluation conditions that were used. We believe this is among the most extensive datasets used for a public evaluation of simulation accuracy to date.

In this article, we discuss three key results.

  • First, predicting what people say. We introduce our central result—Aaru’s errors against survey data are comparable in magnitude to published survey-to-survey disagreement rates. Our sample-adjusted mean absolute error is 2.78 percentage points.

  • Second, predicting what people do. We compare simulations with observed purchasing behavior in a separate evaluation. Across a set of brand-period observations backed by credit card data, Aaru’s predictions ranked brands closer to the real results than published purchase-consideration scores did.

  • Last, predicting our own accuracy. We observe an AUROC of 0.845 for differentiating TVD values below 16 points, compared with 0.736 for the strongest method reported by Wesel, Chen, and Liang on a separate evaluation.

In this post, we’ll focus on univariate distribution accuracy, measuring predicted answer shares rather than microdata rows or relationships between responses. Future posts will cover higher-order accuracy and logical, row-level coherence.

We believe these results speak to the merit of our approach, and we’re excited to share this early stepping stone.

Why it matters

Simulation expands our ability to understand the world around us. Much of this lies in hypotheticals: scenarios that don’t exist today, and questions that no survey was written to ask. These questions fall outside the established surface area of survey instruments, the focus of traditional quantitative research for decades. Product, pricing, perception, positioning: so many decisions turn on how people will behave.

Before a model can be trusted on what has never been asked, it must be scored on what has. This is why we begin in the most familiar place possible: with survey instruments themselves. We test our simulations against published survey results and compare their error with how much repeated surveys disagree with each other. We think of that disagreement as a replication floor. Our mean error falls within the range of disagreement reported across repeated surveys, which we see as an important initial bar for simulation.

How a simulation works

A simulation starts with an audience and the questions they will respond to. From that description, Aaru builds a population of agents—computational models of individual people—whose traits, such as age, income, and location, match the audience in aggregate. When the simulation runs, each agent responds to the questions or alternatives on its own.

From audience to modeled responses

AudienceGenerate population+ 9,997 moreSimulated agentsQuestionsRun simulationModeled responsesMicrodataAgent 01Agent 02Agent 03PreferencesObjections Possible behaviors Audience+ 9,997 moreSimulated agentsQuestionsModeled responsesMicrodataGenerate populationAgent 01Agent 02Agent 03Run simulationPreferencesObjections Possible behaviors

The result is a set of modeled responses—preferences, objections, and possible behaviors—that can be read for a single agent or for the population as a whole. In this evaluation, each input is a published survey question and its answer options. The output is microdata: one record per agent, so it can be analyzed, segmented, and queried just like data from real respondents.

Measuring population simulation

Questions in the evaluation

Our evaluation covers 2,993 questions across nine industries and 186 different studies. Questions may address experiences, attitudes, intentions, and expectations. During testing, Aaru’s simulation system receives the question phrasing, answer options, and an audience description. Neither the evaluation questions nor their published answers were used to develop or tune our simulation system or confidence predictor.

We’ll analyze 13,727 answer shares—an “answer share” represents the proportion of respondents who select a particular answer option. For example, a question with three options, each selected by 50%, 30%, and 20% of respondents, would contribute three answer shares—one per option. Together, these shares form the question’s answer distribution.

Each industry brings a different mix of questions

Example questions

One example comes from an Ipsos survey of U.S. adults. This survey asked about shopping experiences and whether they often felt they could get better discounts online. In the survey, 48% somewhat agreed and 26% strongly agreed. In Aaru’s simulation, 48% somewhat agreed and 22% strongly agreed. [1]

IpsosJuly 14–15, 20261,085 U.S. adults

“I often feel like I could get better discounts online”

Agreement with a statement about recent in-store shopping experiences.

Answer distribution
PublishedSimulated
Strongly agree
Published: 26.0%Simulated: 22.0%
Somewhat agree
Published: 48.0%Simulated: 48.0%
Somewhat disagree
Published: 15.0%Simulated: 16.0%
Strongly disagree
Published: 4.0%Simulated: 5.0%
Don’t know
Published: 7.0%Simulated: 9.0%
Observed TVD
4.00points
Predicted TVD
4.30points

65% of the 2,993 questions had a larger observed TVD

Dashed outlines show where a simulated share falls short of the published share; light ends show where it goes over. Observed TVD is the total of either side. For each question, our confidence predictor estimates error without consulting the published answers. We compare the simulation with the study sample. Ipsos conducted the survey online among U.S. adults aged 18 and older. We used a knowledge cutoff of July 13, 2026. Question and published results[1].

Evaluation conditions

In production, Aaru draws on a wide range of alternative data—including credit card transactions, foot traffic, social media and SKU-level sales—and we invest heavily in data collection and processing. Certain confidential alternative data sources, and collection and processing practices, have been removed for the purposes of this evaluation. Exa is the only external retrieval tool used in this study. Exa, a web index that offers timestamped snapshots, allows us to restrict retrieved information by date. Results reflect a deliberately constrained version of the system.

Our core, row-level model for predicting an agent’s behavioral trajectory isn’t a language model, nor is the model that generates the agent population. This is critical to minimizing contamination, as language models are pre-trained on a large data corpus and may have memorized the contents of sources that should be excluded for backtesting. We drew on Saxena’s open-source work on LLM knowledge cutoff evaluations [2] and selected questions from studies recorded as fielded in 2026. External retrieval is restricted to pages stored at least one day prior to commencing fieldwork, using the earliest cutoff among the questions in each configuration. Source-publisher domains and prediction-market sources are excluded from external search.

Evaluation metrics

We evaluated the results using the following metrics.

  1. Mean absolute error (MAE) measures the average absolute difference between predicted and published answer shares.
  2. Total variation distance (TVD) captures the distance between full distributions: the smallest amount of probability that must move between answers to make the simulated shares match the published shares.
  3. Pearson correlation measures the linear relationship between predicted and published answer shares; r2r^2 is its square. Spearman correlation captures the relationship between ranks.

The sampling adjustment uses a normal approximation to expected sampling error. For each published share pp and effective sample size nn, the expected absolute sampling error (the sampling reference), in percentage points, is b=1002p(1−p)πnb = 100\sqrt{\frac{2p(1-p)}{\pi n}}. Sample-adjusted MAE subtracts the average bb across answer options; sample-adjusted TVD subtracts half the sum. Questions are equal-weighted. We use 2,648 printed sample sizes and the median of 1,441.5 for 345 missing sizes, with a design effect of one: the effective sample size equals the printed or imputed sample size, with no reduction for weighting or clustering. Negative adjusted values mean the observed error is below the expected sampling error. We retain them before averaging rather than rounding them up to zero.

Mean absolute error

The average gap between simulated and published shares, per answer.

MAE · Typical case2.40points

How do you expect the U.S. economy to perform in 2026, compared to 2025?

Northwestern Mutual · January 2026 · 4,375 respondents · Aaru’s simulated shares from our evaluation

Improve significantly1.0
Improve somewhat4.0
Remain steady6.0
Weaken somewhat1.0
Weaken significantly0.0
Add the gaps12.0
Divide by 5 answers2.40

How it works

Mean absolute error (MAE) measures the average absolute difference between predicted and published answer shares. We calculate MAE within each question, then average across questions with equal weight.

For example, an MAE of 0.50 means the simulated shares differ from the published shares by half a percentage point per answer, on average.

Mathematical formula

MAE in percentage points = 100 divided by K, multiplied by the sum of absolute share differences.
MAE = (1 + 4 + 6 + 1 + 0) divided by 5 = 2.40 percentage points.

p: published share; q: simulated share (both 0–1). K: number of answer options.

A difference between 35%35\% and 40%40\% is a five-percentage-point gap. Each example is a real question from the evaluation, shown with its published answer shares and Aaru’s simulated shares.

We use the sample-adjusted MAE and TVD because we want a perfect forecaster to have an error of 0. Published survey results include sampling noise, so even the underlying population would differ from a published result by the expected sampling error. The sampling adjustment subtracts the expected sampling error directly, rather than in quadrature, which doesn’t require a normal approximation to our system’s errors. The adjustment provides a reference for interpreting observed error under the stated sampling assumptions; it does not establish a bound for any individual question.

Predicting what people say

Across 2,993 questions, Aaru’s errors against survey data are comparable in magnitude to published survey-to-survey disagreement rates. After the sampling adjustment, mean MAE was 2.78 percentage points (median: 2.00) and mean TVD was 5.98 points (median: 4.30). Before the sampling adjustment, mean MAE was 3.53 percentage points (median: 2.73) and mean TVD was 7.62 points (median: 6.00), and at least 90% of questions had a TVD of 16 points or less. Every question has equal weight in these means.

Accuracy metrics

2,993 questions · 13,727 answer shares

MetricMeanMedianScope
Sample-adjusted MAEPercentage points2.782.002,993 questions
Sample-adjusted TVDTVD points5.984.302,993 questions
Observed MAEPercentage points3.532.732,993 questions
Observed TVDTVD points7.626.002,993 questions
Pearson rr0.970 pooled13,727 answer shares
r2r^20.940 pooled13,727 answer shares
Spearman ρ\rho0.963 pooled13,727 answer shares
Wasserstein distanceNormalized, 0–10.03750.0259733 ordered-scale questions
MAE, TVD, and Wasserstein distance give each question equal weight; lower is better. Correlations are pooled across answer shares; higher is better. Wasserstein distance uses the 733 questions with complete, ordered answer scales of at least three options, before sampling adjustment.

Across 13,727 answer shares, simulations closely track published results. We maintain a Pearson correlation of 0.970 and a Spearman correlation of 0.963. Within individual questions, mean Spearman correlation of ranked response options was 0.90 (median: 0.99) and modal accuracy was 87.5%.

Univariate accuracy by industry

Our evaluation spans financial services, business-to-business, technology, healthcare, consumer packaged goods, retail, and more. Each of the nine industries contains between 191 and 573 questions. We observe strong results across industries.

Industry results

Full evaluation · 2,993 questions

Mean sample-adjusted MAE by industry · Lower is better · Best first

Industry scores range from 2.21 to 3.39 points, against 2.78 for the full evaluation.

Mean across questionsMiddle 50% of studiesMiddle 80% of studiesFull evaluation
Full evaluation2,993 questions · 186 studies2.78points
Politics, policy & public affairs573 questions · 75 studies2.21points−0.57
B2B & professional services366 questions · 37 studies2.41points−0.37
Retail, fashion & home191 questions · 20 studies2.51points−0.27
Consumer packaged goods220 questions · 27 studies2.71points−0.07
Technology, media & telecom352 questions · 37 studies2.72points−0.06
Sports, entertainment & hospitality308 questions · 22 studies2.96points+0.17
Energy, industrials & automotive274 questions · 31 studies3.18points+0.40
Healthcare & life sciences249 questions · 45 studies3.23points+0.45
Financial services & real estate460 questions · 59 studies3.39points+0.61

Dots show each industry’s score across all of its questions, the same measure as the full-evaluation scores above; small numbers give the difference from the full evaluation. Bands show how scores vary between studies, with each study weighted equally: the darker band covers the middle 50% of studies and the lighter band the middle 80%. Arrows mark bands that continue past the axis. MAE is sample-adjusted and averaged across questions within each study, in percentage points.

The 186 studies are study series, with their survey waves kept together. Within each industry, only that industry’s questions are used. A study can appear in more than one industry.

Univariate accuracy by research application

Our customers use these simulations to decide what to build, where to invest, and what to try next. These questions cover the research applications that our customers care about—brand health, market sizing, customer experience, products and pricing, and more.

For questions classified as brand health tracking, our system achieved a mean sample-adjusted MAE of 2.73 percentage points across 270 questions. For questions classified as trends and futures forecasting, mean sample-adjusted MAE was 2.52 percentage points across 533 questions. For questions classified as market sizing and demand estimation, mean sample-adjusted MAE was 3.16 percentage points across 240 questions. Across the board, we observe strong performance.

Research application results

Full evaluation · 2,993 questions

Mean sample-adjusted MAE by research application · Lower is better · Best first

Research application scores range from 1.26 to 3.75 points, against 2.78 for the full evaluation.

2,756 of 2,993 questions shown · Groups with at least 100 questions

Mean across questionsMiddle 50% of studiesMiddle 80% of studiesFull evaluation
Full evaluation2,993 questions · 186 studies2.78points
Segmentation & personas182 questions · 40 studies1.26points−1.52
Electoral & geopolitical forecasting165 questions · 27 studies2.25points−0.53
Trends & futures forecasting533 questions · 73 studies2.52points−0.27
Policy & public opinion232 questions · 45 studies2.63points−0.16
Brand health tracking270 questions · 8 studies2.73points−0.05
Attitudes & usage599 questions · 83 studies2.76points−0.02
Customer experience & journey230 questions · 36 studies2.78points0.00
Perception & sentiment198 questions · 46 studies2.82points+0.04
Market sizing & demand estimation240 questions · 34 studies3.16points+0.38
Product & pricing research107 questions · 22 studies3.75points+0.97

Dots show each research application’s score across all of its questions, the same measure as the full-evaluation scores above; small numbers give the difference from the full evaluation. Bands show how scores vary between studies, with each study weighted equally: the darker band covers the middle 50% of studies and the lighter band the middle 80%. Arrows mark bands that continue past the axis. MAE is sample-adjusted and averaged across questions within each study, in percentage points.

The 186 studies are study series, with their survey waves kept together. Within each research application, only that research application’s questions are used. A study can appear in more than one research application.

Groups with fewer than 100 questions are omitted. The full-evaluation row includes all 2,993 questions, including 237 in the omitted groups.

Comparing simulations with survey replication

We evaluate our simulation system against published survey results and compare the resulting errors with survey-to-survey disagreement. For instance, when Callegaro and colleagues [3] fielded the same survey twice (two weeks apart) using four U.S. sample sources with approximately 1,500 respondents per wave, we calculate mean wave-to-wave TVD of 2.3 to 12.0 points from the reported sums of absolute differences. Pew Research Center (2021) [4] evaluated a single vendor across waves. Shares for 267 of 285 response categories changed by no more than three percentage points.

In a similar study, Redmiles and colleagues [5] repeated a security and privacy survey on Amazon Mechanical Turk (MTurk) 14 months apart. Across 26 binary items in their publication, we calculate a mean TVD of approximately 2.6 points. Tang and colleagues [6] compared response distributions for 30 questions from Pew’s American Trends Panel and reported mean TVD of 13 points for a representative Prolific sample, 14 points for a gender-balanced Prolific sample, and 29 points for MTurk. MTurk’s mean fell to 17 points after filtering, weighting, and restricting the population. These comparisons also reflect differences in recruitment and timing, given the online samples were collected in 2022 and Pew’s sample in 2019.

Survey replication in context

8 survey comparisons · 3 studies · TVD in points

Aaru’s mean error falls within the range of repeated surveys.

A mean observed TVD of 7.62 points, against 2.3–12.0 when the same source is surveyed twice and 13–29 when different online samples are compared with Pew.

2.3–12.0 5 comparisons
Different online samplesTang et al. (2022)[6]
13–29 3 comparisons
Sampling adjustment removes 1.64 points of Aaru’s error, taking mean TVD from 7.62 to 5.98, with every question weighted equally. What remains sits within the range of survey-to-survey disagreement shown above.

Comparing simulations with literature

In SimBench [7], which evaluates 45 language models across 20 datasets, the best overall TVD result was 19.1 points. Meister and colleagues [8] reported 14.6 points of TVD in their strongest OpinionQA configuration. SYN-DIGITS [9], which calibrates simulations using human survey data, achieves a TVD of 17.1 points on OpinionQA. Aaru’s mean observed TVD of 7.62 points is 48–60% lower than these reported results.

Many studies also use Wasserstein distance, which accounts for how far probability moves between ordered answer options. Across 733 questions with complete, ordered answer scales of at least three options, our system achieved a mean normalized Wasserstein distance of 0.0375 (median: 0.0259). With equal weight per study across 77 study series, the mean was 0.0445.

SubPOP [10] reported normalized Wasserstein distances of 0.094 on OpinionQA and 0.096 on the General Social Survey (GSS). SOCRATES [11] reported 0.151 across 40 unseen studies.

Survey distribution accuracy

Observed TVD · Lower is better

Aaru’s mean error is 48–60% lower than published results.

A mean observed TVD of 7.62 points, against 14.6–19.1 in the strongest results other studies report.

Meister et al.[8]100 OpinionQA questions · Five-shot
14.6 Aaru 48% lower
SYN-DIGITS[9]489 OpinionQA questions in total · 20% held out
17.1 Aaru 55% lower
SimBench[7]13,510 population and group test cases
19.1 Aaru 60% lower
We show selected studies that evaluate metrics for survey answer distributions across at least ten questions. For all studies, we report observed TVD values (pre-sampling adjustment or similar adjustment). Not all studies included used the same questions or populations. The correlation view uses Maier et al.[12].

Different literature uses different approaches. Some of the literature shown above evaluates language models against direct elicitation, a technique which prompts a model to state the distributions of a survey question directly. While an important basis of comparison, we don’t consider it equivalent—it produces no respondent-level data; consequently, segments or crosstabs wouldn’t be accessible.

Comparing simulations with baselines

We evaluated Moonshot’s Kimi K3 to build a reproducible baseline for direct elicitation on a randomly sampled subset of 500 questions. We found ~52% of questions missed the published result by more than 10 points, and more than a quarter of the questions missed by more than 16 points. On those same 500 questions, Kimi made three times as many misses above 16 TVD points as Aaru (27.6% vs. 9.2%) and nearly three times as many misses above 25 points (7.6% vs. 2.6%). OpenAI’s gpt-oss-120b, another popular (though much smaller) baseline, does worse still; we observe a mean error of 21.6 points on the same questions.

Direct elicitation baselines

Open-source models asked directly for each distribution, against Aaru

Asked directly, open-source models miss more often, and by more.

On the same 500 random questions, Kimi K3 misses by more than 16 points three times as often as Aaru. On a complex sample, both models get worse, and Aaru’s mean error is 7.43 points.

Mean error on random and complex questions

Mean TVD in points · Lower is better

AaruComplex sample
7.43
Open source model baseline
Kimi K3Moonshot AI
12.4 → 16.5 +4.1 points
gpt-oss-120bOpenAI
21.6 → 24.7 +3.1 points

Large misses per 100 questions

Questions missed by more than 16 points of TVD

Random sample

The same 500 questions for both

Complex sample
Direct elicitation asks a model for the answer distribution to a question in one call, with no research, so it produces toplines only. On the random sample, the two models’ errors of 12.4 and 21.6 points span the 14.6–19.1 points reported in the published studies above. The complex sample was not drawn at random, so its results do not describe the full dataset. Each strip rounds its shares to whole questions.

We also pulled a harder, non-random sample of 500 questions that leans more into complex topics and changing scenarios. On this sample, Kimi K3’s mean error rises from 12.4 to 16.5 points, and gpt-oss-120b’s error from 21.6 to 24.7 points. We find that Kimi K3 misses by more than 16 points on 47% of questions, and by more than 25 points on 15% of questions. On the same set, Aaru’s mean error is 7.43 points, more than 50% lower than Kimi K3’s.

Predicting what people do

Simulating what people say is critical, but in many sensitive decision-making scenarios, it’s not enough. Often, understanding what people do—the products they purchase, the places they go, and the decisions they make—is more useful to come to a decision. We’ve spent much of this article discussing our performance on survey prediction, but we’d like to share results for a second evaluation (this one behavioral), too.

In this evaluation, we used credit card panel purchase records from an alternative data provider. We ranked brands by purchase frequency (measured as purchase records per 1,000 active card-days) within the same category and period. We then evaluated the performance of our system on predicting purchase behavior and compared it to purchase-consideration data for the same brands from the same time period.

Example scenarios

Purchase consideration and purchase behavior can differ massively. We provide examples below demonstrating scenarios where brand ordering changes between surveys and purchase panels, alongside Aaru’s prediction for these same scenarios.

Consideration and purchases

YouGov BrandIndexMar 2024 – Feb 20257 brands

Which supermarkets do people buy from?

ALDI leads Kroger on consideration. Kroger leads on purchases.

Survey consideration

What people would consider

  1. ALDI
  2. Kroger
  3. Publix
  4. ShopRite
  5. H-E-B
  6. WinCo Foods
  7. Grocery Outlet

Observed purchases

What the panel recorded

  1. Kroger
  2. Publix
  3. ALDI
  4. H-E-B
  5. ShopRite
  6. WinCo Foods
  7. Grocery Outlet

Aaru predictions

Asked about purchases

  1. Kroger
  2. Publix
  3. ALDI
  4. H-E-B
  5. ShopRite
  6. WinCo Foods
  7. Grocery Outlet

Brand pairs ranked in the same order as the purchase panel

Each crossing above is one reversed pair · Ties count half

Survey consideration

78.6%

Aaru predictions

100.0%

Aaru puts Kroger ahead of ALDI, matching the purchase panel.

These examples show where consideration and purchases diverged. Each column ranks all brands in that group, and each line follows one brand across the columns. Where two lines cross, that pair is in a different order. The results across the full purchase evaluation follow below.

Accuracy across purchase outcomes

When using purchase consideration to rank observed purchasing, the consideration scores from the survey panel had a 77.8% agreement with the transaction panel across 340 brand pairs. In comparison, Aaru’s purchase predictions, averaged across three runs before brands are ranked, had a 92.4% rate of agreement on the same pairs. Compared to the survey, this is roughly a two-thirds reduction in error. In a baseline comparison using public measures of brand size (e.g. store counts, subscriber counts), we saw a 79.3% agreement to the purchase data across the 333 pairs where a size measure was available.

Aaru made 66% fewer ordering errors than the published survey.

An ordering error is a brand pair ranked in the opposite order from observed purchases. Predicted ties count half.

Brand pairs ranked in the same order as the purchase panel

+14.6 percentage points higher agreement when Aaru is asked about purchases, compared with the published survey (95% confidence interval: 7.2–23.0 points). Aaru’s result averages the predictions of three runs before brands are ranked; individually, the runs scored 91.5%, 91.8%, and 92.9%.

Rankings within 14 category-and-period groups. Survey and Aaru results use 340 pairs across 100 brand-period observations; the size baseline uses 333 pairs across 99. Each cell is one brand pair.

We found the behavioral simulation’s gain over the published survey was 14.6 percentage points. On the brands we tested, Aaru’s simulations order brands more closely to the transaction panel than stated-preference surveys do. This gives us a way to test simulations against what people actually buy.

Predicting our own errors

Alongside accuracy, it’s important to understand where results are likely to be accurate, and where greater caution is warranted. For each question, our system also returns a predicted TVD alongside a probability that the error exceeds each threshold (e.g., 10 or 16 points). These estimates are made by the system as part of the simulation without consulting the scored reference outcomes. We use predicted TVD to rank questions by expected error, reversing the ranking when evaluating whether error falls below a threshold.

Across 2,993 questions, there is meaningful separation between more confident and less confident forecasts.

Keeping roughly half of the questions with the highest confidence gives a mean observed TVD of 4.54 points, compared with 7.62 for the full set. In this retained subset, the share of questions with TVD above 16 points is 1.0%, compared with 9.5% in the full set. Changing the confidence threshold allows us to trade coverage—the percentage of questions retained—for accuracy. The predictions themselves are unchanged.

Confidence and coverage

Error falls as we retain the most confident questions.

Coverage target

Mean TVD

4.54points

40% lower than the full set’s 7.62 points

Questions with TVD above 16 points

1.0%

89% lower than the full set’s 9.5%

Errors above 16 points set aside

267of 283

In the 1,468 least confident questions

Mean TVD of the retained questions

Retaining 1,525 questions (51.0%): mean TVD 4.54 points; 1.0% of questions have errors above 16 points; 267 of the 283 errors above 16 points are in the questions set aside.

Each cell is one question, ordered by Aaru’s predicted TVD from lowest to highest; its color shows the observed TVD. We keep tied predictions together, which can leave us with more questions than the coverage target. The predictions themselves are unchanged. The dashed line marks the full evaluation.

Across the full evaluation, the correlation between predicted and realized error achieved Pearson r=0.595r = 0.595 and Spearman ρ=0.666\rho = 0.666. We also use the area under the receiver operating characteristic curve (AUROC), which measures how often a randomly selected question above an error threshold receives a higher predicted TVD than one at or below it, with half credit for tied predictions. Our confidence predictor achieves an AUROC of 0.848 at a threshold of 16 TVD points and an AUROC of 0.825 at 10 points.

Wesel, Chen, and Liang [13] asked a similar question—how well can a model predict the error of its own simulations? For TVD below 16 points, our system achieved an AUROC of 0.845 vs. 0.736 from their strongest method. This corresponds to a 41% lower pairwise ranking error, calculated as 1−AUROC1 - \mathrm{AUROC} (from 26.4% to 15.5%) when comparing questions on opposite sides of the threshold. Note that errors exactly at 16 points fall outside that event; 19 Aaru questions fall exactly at the boundary.

Aaru’s Pearson correlation was 0.595 compared with 0.565 in Wesel, Chen and Liang, and root mean squared error (RMSE) was 5.32 TVD points, compared with 7.76, a reduction of 31%. We’re thankful for their contributions.

Predicting simulation error

Aaru’s confidence predictor leads all five published methods on each measure.

AUROC · Higher is better

0.845

Strongest published method: 0.736

Pearson rr · Higher is better

0.595

Strongest published method: 0.565

RMSE · Lower is better

5.32points

31% lower than the strongest published method, 7.76

Aaru’s ROC curve

Retaining the most confident questions first

At the 50% coverage target, we keep 56% of the questions below 16 points (1,508 of 2,691) and 6% of the rest (17 of 302).

AUROC by method

Aaru against five other published methods, from chance to perfect

0.845Aaru41% lower ranking error than the next best method
0.736Simulation model fine-tune
0.730Hidden-state probe
0.720General-purpose model fine-tune
0.686Embedding regression
0.566Basic-feature regression

Published methods from Wesel, Chen, and Liang[13], five-fold held-out results on about 8,600 questions.

Aaru’s curve uses all 2,993 evaluated questions. The published results come from a separate evaluation, so we compare their reported AUROC values. Aaru’s calculation and sources.

The estimates also track the magnitude of error. Across deciles of predicted error, the mean realized TVD increases monotonically from 2.2 to 14.6 TVD points. The two highest deciles of predicted error have mean predicted errors of 9.6 and 13.0 points, against mean realized errors of 12.0 and 14.6 points, respectively.

Toward general-purpose population simulation

We began in the most familiar place we could: survey instruments themselves. Across 2,993 questions, we observed simulation error rates comparable in magnitude to the survey-to-survey disagreement rates published in the literature. Furthermore, when compared against ground-truth transaction data, we observed simulated brand ordering tracked the real world with a 66% lower error compared to surveyed consideration. Finally, for each question, the system returned a predicted error that closely tracked the realized one.

Much of the value of simulation lies in hypotheticals: products that haven’t launched, prices that haven’t been set, and messages that haven’t been published. Simulation allows for hypotheticals to be evaluated at a previously infeasible scale. Unlike a human, a simulated agent won’t get tired of evaluating the hundredth flavor concept or the thousandth shelf ordering. Accuracy is foundational to achieving this, and we’re excited to share these results.

We founded Aaru to simulate humanity because so many decisions turn on how people will behave and what they value. Understanding how people behave shapes what we build, where we invest, and what we try next. We’re building toward a future where ideas, products, policies, and messages are simulated before they’re released. This evaluation is an early stepping stone toward that future, and we’re grateful to be able to share it.

ReferencesSources cited throughout this article.View sourcesHide sources
  1. Ipsos (2026). Ipsos Poll on Consumer Behavior. July 17, 2026. Question 5b, page 7; methodology, page 18. The survey was conducted July 14–15 among 1,085 U.S. adults. The figure uses the published answer shares and the archived simulation for this question.

  2. Saxena (2026). Knowledge cutoff evaluation. Open-source benchmark and results, revision ef12ac9, September 2, 2026. This informed the choice of question dates in our evaluation.

  3. Callegaro, Tsirlin, Yang, and Ma (2022). Test-retest reliability of four U.S. non-probability sample sources. AAPOR conference presentation. The mean sums of absolute differences across five attitude questions were 4.6, 5.2, 13.2, and 24.0 points. Dividing by two gives mean TVDs of 2.3, 2.6, 6.6, and 12.0 points. Source: slide 14.

  4. Lau (2021). Assessing the consistency of online opt-in polls. Pew Research Center.

  5. Redmiles, Kross, and Mazurek (2019). How Well Do My Results Generalize? Comparing Security and Privacy Survey Results from MTurk, Web, and Telephone Samples. IEEE Symposium on Security and Privacy. Appendix VIII-B, Table XII.

  6. Tang, Birrell, and Lerner (2022). Replication: How Well Do My Results Generalize Now? The External Validity of Online Privacy and Security Surveys. SOUPS. Sources: Tables 2–3 and Sections 3.2 and 4.1.

  7. Hu et al. (2026). SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors. ICLR. Across 20 datasets and 13,510 population and group test cases, Claude 3.7 Sonnet achieved the lowest overall TVD, 19.1 points, compared with Aaru’s 7.62. The model directly estimates answer shares. Source: Table 10. SimBench’s separate normalized score uses a different aggregation and is not used here.

  8. Meister, Guestrin, and Hashimoto (2025). Benchmarking Distributional Alignment of Large Language Models. NAACL, pp. 24–49. Sources: Tables 1a and 10.

  9. Fan et al. (2026). SYN-DIGITS: A Synthetic Control Framework for Calibrated Digital Twin Simulation. Preprint. Sources: Section 7.3 and Tables 9–10.

  10. Suh et al. (2025; revised 2026). Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinions. ACL. Table 1.

  11. Kolluri et al. (2025). Finetuning LLMs for Human Behavior Prediction in Social Science Experiments. EMNLP, pp. 30096–30111. SOCRATES achieved normalized Wasserstein distance of 0.151 across 40 unseen studies, after training on 170 studies. Responses are scaled to 0–1, then distances are averaged within each study and across studies. Our calculation with equal weight per study is 0.0445 across 77 study series, using the same 733-question subset. A separate SOCRATES setting, trained on other conditions within the same studies and supplied with reasoning traces, achieved 0.063. Sources: Section 3.2 and Tables 2–3.

  12. Maier et al. (2025). LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings. Preprint. Source: Table 1.

  13. Wesel, Chen, and Liang (2026). Building confidence in Simile. Blog post. The continued fine-tune of the simulation model achieved AUROC 0.736, Pearson r=0.565r = 0.565, and RMSE 0.0776 on the 0–1 TVD scale (7.76 points). The post reports five-fold held-out predictions on approximately 8,600 questions, keeping identical question text and all questions from the same experimental sample in the same fold. Our figure includes all five reported methods. Aaru’s calculation and sources use 2,993 archived predictions with equal question weight. The comparison matches the authors’ event, TVD below 16 points; 19 Aaru questions fall exactly at the boundary. This gives AUROC 0.845, versus 0.848 for the strictly-above-16 event used elsewhere in this section. Sources: the five method figures and the evaluation description under “Problem statement.”