
Population simulation at the replication floor
Simulation has the potential to be transformative, but only if it’s accurate. Today, we’re sharing results of a comprehensive evaluation across 2,993 questions, with an industry-leading mean TVD of 7.62% and an MAE of 3.53%.
Introduction
Key takeaways
Aaru builds simulations of human behavior. These simulations produce a population of demographically accurate agents—computational models of humans—and their answers across user-defined questions.
By modeling human behavior as a collection of agents within a population, Aaru’s simulations enable detailed analysis of their outputs, just as one may study a human sample. However, for population-level data to be useful, the population aggregates—that is, the distribution of age, income, or any other individual trait—must align with the real world. Higher-order analytic structures like crosstabs and segments are predicated on aggregate accuracy to be usable.
We evaluate on a 2,993-question dataset and discuss the evaluation conditions that were used. We believe this is among the most extensive datasets used for a public evaluation of simulation accuracy to date.
In this article, we discuss three key results.
-
First, predicting what people say. We introduce our central result—Aaru’s errors against survey data are comparable in magnitude to published survey-to-survey disagreement rates. Our sample-adjusted mean absolute error is 2.78 percentage points.
-
Second, predicting what people do. We compare simulations with observed purchasing behavior in a separate evaluation. Across a set of brand-period observations backed by credit card data, Aaru’s predictions ranked brands closer to the real results than published purchase-consideration scores did.
-
Last, predicting our own accuracy. We observe an AUROC of 0.845 for differentiating TVD values below 16 points, compared with 0.736 for the strongest method reported by Wesel, Chen, and Liang on a separate evaluation.
In this post, we’ll focus on univariate distribution accuracy, measuring predicted answer shares rather than microdata rows or relationships between responses. Future posts will cover higher-order accuracy and logical, row-level coherence.
We believe these results speak to the merit of our approach, and we’re excited to share this early stepping stone.
Why it matters
Simulation expands our ability to understand the world around us. Much of this lies in hypotheticals: scenarios that don’t exist today, and questions that no survey was written to ask. These questions fall outside the established surface area of survey instruments, the focus of traditional quantitative research for decades. Product, pricing, perception, positioning: so many decisions turn on how people will behave.
Before a model can be trusted on what has never been asked, it must be scored on what has. This is why we begin in the most familiar place possible: with survey instruments themselves. We test our simulations against published survey results and compare their error with how much repeated surveys disagree with each other. We think of that disagreement as a replication floor. Our mean error falls within the range of disagreement reported across repeated surveys, which we see as an important initial bar for simulation.
How a simulation works
A simulation starts with an audience and the questions they will respond to. From that description, Aaru builds a population of agents—computational models of individual people—whose traits, such as age, income, and location, match the audience in aggregate. When the simulation runs, each agent responds to the questions or alternatives on its own.
The result is a set of modeled responses—preferences, objections, and possible behaviors—that can be read for a single agent or for the population as a whole. In this evaluation, each input is a published survey question and its answer options. The output is microdata: one record per agent, so it can be analyzed, segmented, and queried just like data from real respondents.
Measuring population simulation
Questions in the evaluation
Our evaluation covers 2,993 questions across nine industries and 186 different studies. Questions may address experiences, attitudes, intentions, and expectations. During testing, Aaru’s simulation system receives the question phrasing, answer options, and an audience description. Neither the evaluation questions nor their published answers were used to develop or tune our simulation system or confidence predictor.
We’ll analyze 13,727 answer shares—an “answer share” represents the proportion of respondents who select a particular answer option. For example, a question with three options, each selected by 50%, 30%, and 20% of respondents, would contribute three answer shares—one per option. Together, these shares form the question’s answer distribution.
Each industry brings a different mix of questions
Example questions
One example comes from an Ipsos survey of U.S. adults. This survey asked about shopping experiences and whether they often felt they could get better discounts online. In the survey, 48% somewhat agreed and 26% strongly agreed. In Aaru’s simulation, 48% somewhat agreed and 22% strongly agreed. [1]
IpsosJuly 14–15, 20261,085 U.S. adults
“I often feel like I could get better discounts online”
Agreement with a statement about recent in-store shopping experiences.
- Observed TVD
- 4.00points
- Predicted TVD
- 4.30points
65% of the 2,993 questions had a larger observed TVD
Evaluation conditions
In production, Aaru draws on a wide range of alternative data—including credit card transactions, foot traffic, social media and SKU-level sales—and we invest heavily in data collection and processing. Certain confidential alternative data sources, and collection and processing practices, have been removed for the purposes of this evaluation. Exa is the only external retrieval tool used in this study. Exa, a web index that offers timestamped snapshots, allows us to restrict retrieved information by date. Results reflect a deliberately constrained version of the system.
Our core, row-level model for predicting an agent’s behavioral trajectory isn’t a language model, nor is the model that generates the agent population. This is critical to minimizing contamination, as language models are pre-trained on a large data corpus and may have memorized the contents of sources that should be excluded for backtesting. We drew on Saxena’s open-source work on LLM knowledge cutoff evaluations [2] and selected questions from studies recorded as fielded in 2026. External retrieval is restricted to pages stored at least one day prior to commencing fieldwork, using the earliest cutoff among the questions in each configuration. Source-publisher domains and prediction-market sources are excluded from external search.
Evaluation metrics
We evaluated the results using the following metrics.
- Mean absolute error (MAE) measures the average absolute difference between predicted and published answer shares.
- Total variation distance (TVD) captures the distance between full distributions: the smallest amount of probability that must move between answers to make the simulated shares match the published shares.
- Pearson correlation measures the linear relationship between predicted and published answer shares; is its square. Spearman correlation captures the relationship between ranks.
The sampling adjustment uses a normal approximation to expected sampling error. For each published share and effective sample size , the expected absolute sampling error (the sampling reference), in percentage points, is . Sample-adjusted MAE subtracts the average across answer options; sample-adjusted TVD subtracts half the sum. Questions are equal-weighted. We use 2,648 printed sample sizes and the median of 1,441.5 for 345 missing sizes, with a design effect of one: the effective sample size equals the printed or imputed sample size, with no reduction for weighting or clustering. Negative adjusted values mean the observed error is below the expected sampling error. We retain them before averaging rather than rounding them up to zero.
Mean absolute error
The average gap between simulated and published shares, per answer.
How do you expect the U.S. economy to perform in 2026, compared to 2025?
Northwestern Mutual · January 2026 · 4,375 respondents · Aaru’s simulated shares from our evaluation
How it works
Mean absolute error (MAE) measures the average absolute difference between predicted and published answer shares. We calculate MAE within each question, then average across questions with equal weight.
For example, an MAE of 0.50 means the simulated shares differ from the published shares by half a percentage point per answer, on average.
Mathematical formula
p: published share; q: simulated share (both 0–1). K: number of answer options.
We use the sample-adjusted MAE and TVD because we want a perfect forecaster to have an error of 0. Published survey results include sampling noise, so even the underlying population would differ from a published result by the expected sampling error. The sampling adjustment subtracts the expected sampling error directly, rather than in quadrature, which doesn’t require a normal approximation to our system’s errors. The adjustment provides a reference for interpreting observed error under the stated sampling assumptions; it does not establish a bound for any individual question.
Predicting what people say
Across 2,993 questions, Aaru’s errors against survey data are comparable in magnitude to published survey-to-survey disagreement rates. After the sampling adjustment, mean MAE was 2.78 percentage points (median: 2.00) and mean TVD was 5.98 points (median: 4.30). Before the sampling adjustment, mean MAE was 3.53 percentage points (median: 2.73) and mean TVD was 7.62 points (median: 6.00), and at least 90% of questions had a TVD of 16 points or less. Every question has equal weight in these means.
| Metric | Mean | Median | Scope |
|---|---|---|---|
| Sample-adjusted MAEPercentage points | 2.78 | 2.00 | 2,993 questions |
| Sample-adjusted TVDTVD points | 5.98 | 4.30 | 2,993 questions |
| Observed MAEPercentage points | 3.53 | 2.73 | 2,993 questions |
| Observed TVDTVD points | 7.62 | 6.00 | 2,993 questions |
| Pearson | 0.970 pooled | 13,727 answer shares | |
| 0.940 pooled | 13,727 answer shares | ||
| Spearman | 0.963 pooled | 13,727 answer shares | |
| Wasserstein distanceNormalized, 0–1 | 0.0375 | 0.0259 | 733 ordered-scale questions |
Across 13,727 answer shares, simulations closely track published results. We maintain a Pearson correlation of 0.970 and a Spearman correlation of 0.963. Within individual questions, mean Spearman correlation of ranked response options was 0.90 (median: 0.99) and modal accuracy was 87.5%.
Univariate accuracy by industry
Our evaluation spans financial services, business-to-business, technology, healthcare, consumer packaged goods, retail, and more. Each of the nine industries contains between 191 and 573 questions. We observe strong results across industries.
Mean sample-adjusted MAE by industry · Lower is better · Best first
Industry scores range from 2.21 to 3.39 points, against 2.78 for the full evaluation.
Dots show each industry’s score across all of its questions, the same measure as the full-evaluation scores above; small numbers give the difference from the full evaluation. Bands show how scores vary between studies, with each study weighted equally: the darker band covers the middle 50% of studies and the lighter band the middle 80%. Arrows mark bands that continue past the axis. MAE is sample-adjusted and averaged across questions within each study, in percentage points.
The 186 studies are study series, with their survey waves kept together. Within each industry, only that industry’s questions are used. A study can appear in more than one industry.
Univariate accuracy by research application
Our customers use these simulations to decide what to build, where to invest, and what to try next. These questions cover the research applications that our customers care about—brand health, market sizing, customer experience, products and pricing, and more.
For questions classified as brand health tracking, our system achieved a mean sample-adjusted MAE of 2.73 percentage points across 270 questions. For questions classified as trends and futures forecasting, mean sample-adjusted MAE was 2.52 percentage points across 533 questions. For questions classified as market sizing and demand estimation, mean sample-adjusted MAE was 3.16 percentage points across 240 questions. Across the board, we observe strong performance.
Mean sample-adjusted MAE by research application · Lower is better · Best first
Research application scores range from 1.26 to 3.75 points, against 2.78 for the full evaluation.
2,756 of 2,993 questions shown · Groups with at least 100 questions
Dots show each research application’s score across all of its questions, the same measure as the full-evaluation scores above; small numbers give the difference from the full evaluation. Bands show how scores vary between studies, with each study weighted equally: the darker band covers the middle 50% of studies and the lighter band the middle 80%. Arrows mark bands that continue past the axis. MAE is sample-adjusted and averaged across questions within each study, in percentage points.
The 186 studies are study series, with their survey waves kept together. Within each research application, only that research application’s questions are used. A study can appear in more than one research application.
Groups with fewer than 100 questions are omitted. The full-evaluation row includes all 2,993 questions, including 237 in the omitted groups.
Comparing simulations with survey replication
We evaluate our simulation system against published survey results and compare the resulting errors with survey-to-survey disagreement. For instance, when Callegaro and colleagues [3] fielded the same survey twice (two weeks apart) using four U.S. sample sources with approximately 1,500 respondents per wave, we calculate mean wave-to-wave TVD of 2.3 to 12.0 points from the reported sums of absolute differences. Pew Research Center (2021) [4] evaluated a single vendor across waves. Shares for 267 of 285 response categories changed by no more than three percentage points.
In a similar study, Redmiles and colleagues [5] repeated a security and privacy survey on Amazon Mechanical Turk (MTurk) 14 months apart. Across 26 binary items in their publication, we calculate a mean TVD of approximately 2.6 points. Tang and colleagues [6] compared response distributions for 30 questions from Pew’s American Trends Panel and reported mean TVD of 13 points for a representative Prolific sample, 14 points for a gender-balanced Prolific sample, and 29 points for MTurk. MTurk’s mean fell to 17 points after filtering, weighting, and restricting the population. These comparisons also reflect differences in recruitment and timing, given the online samples were collected in 2022 and Pew’s sample in 2019.
Aaru’s mean error falls within the range of repeated surveys.
A mean observed TVD of 7.62 points, against 2.3–12.0 when the same source is surveyed twice and 13–29 when different online samples are compared with Pew.
Comparing simulations with literature
In SimBench [7], which evaluates 45 language models across 20 datasets, the best overall TVD result was 19.1 points. Meister and colleagues [8] reported 14.6 points of TVD in their strongest OpinionQA configuration. SYN-DIGITS [9], which calibrates simulations using human survey data, achieves a TVD of 17.1 points on OpinionQA. Aaru’s mean observed TVD of 7.62 points is 48–60% lower than these reported results.
Many studies also use Wasserstein distance, which accounts for how far probability moves between ordered answer options. Across 733 questions with complete, ordered answer scales of at least three options, our system achieved a mean normalized Wasserstein distance of 0.0375 (median: 0.0259). With equal weight per study across 77 study series, the mean was 0.0445.
SubPOP [10] reported normalized Wasserstein distances of 0.094 on OpinionQA and 0.096 on the General Social Survey (GSS). SOCRATES [11] reported 0.151 across 40 unseen studies.
Survey distribution accuracy
Observed TVD · Lower is better
Aaru’s mean error is 48–60% lower than published results.
A mean observed TVD of 7.62 points, against 14.6–19.1 in the strongest results other studies report.
Different literature uses different approaches. Some of the literature shown above evaluates language models against direct elicitation, a technique which prompts a model to state the distributions of a survey question directly. While an important basis of comparison, we don’t consider it equivalent—it produces no respondent-level data; consequently, segments or crosstabs wouldn’t be accessible.
Comparing simulations with baselines
We evaluated Moonshot’s Kimi K3 to build a reproducible baseline for direct elicitation on a randomly sampled subset of 500 questions. We found ~52% of questions missed the published result by more than 10 points, and more than a quarter of the questions missed by more than 16 points. On those same 500 questions, Kimi made three times as many misses above 16 TVD points as Aaru (27.6% vs. 9.2%) and nearly three times as many misses above 25 points (7.6% vs. 2.6%). OpenAI’s gpt-oss-120b, another popular (though much smaller) baseline, does worse still; we observe a mean error of 21.6 points on the same questions.
Asked directly, open-source models miss more often, and by more.
On the same 500 random questions, Kimi K3 misses by more than 16 points three times as often as Aaru. On a complex sample, both models get worse, and Aaru’s mean error is 7.43 points.
Mean error on random and complex questions
Mean TVD in points · Lower is better
Open source model baseline
Large misses per 100 questions
Questions missed by more than 16 points of TVD
Random sample
The same 500 questions for both
Complex sample
We also pulled a harder, non-random sample of 500 questions that leans more into complex topics and changing scenarios. On this sample, Kimi K3’s mean error rises from 12.4 to 16.5 points, and gpt-oss-120b’s error from 21.6 to 24.7 points. We find that Kimi K3 misses by more than 16 points on 47% of questions, and by more than 25 points on 15% of questions. On the same set, Aaru’s mean error is 7.43 points, more than 50% lower than Kimi K3’s.
Predicting what people do
Simulating what people say is critical, but in many sensitive decision-making scenarios, it’s not enough. Often, understanding what people do—the products they purchase, the places they go, and the decisions they make—is more useful to come to a decision. We’ve spent much of this article discussing our performance on survey prediction, but we’d like to share results for a second evaluation (this one behavioral), too.
In this evaluation, we used credit card panel purchase records from an alternative data provider. We ranked brands by purchase frequency (measured as purchase records per 1,000 active card-days) within the same category and period. We then evaluated the performance of our system on predicting purchase behavior and compared it to purchase-consideration data for the same brands from the same time period.
Example scenarios
Purchase consideration and purchase behavior can differ massively. We provide examples below demonstrating scenarios where brand ordering changes between surveys and purchase panels, alongside Aaru’s prediction for these same scenarios.
YouGov BrandIndexMar 2024 – Feb 20257 brands
Which supermarkets do people buy from?
ALDI leads Kroger on consideration. Kroger leads on purchases.
Survey consideration
What people would consider
- ALDI
- Kroger
- Publix
- ShopRite
- H-E-B
- WinCo Foods
- Grocery Outlet
Observed purchases
What the panel recorded
- Kroger
- Publix
- ALDI
- H-E-B
- ShopRite
- WinCo Foods
- Grocery Outlet
Aaru predictions
Asked about purchases
- Kroger
- Publix
- ALDI
- H-E-B
- ShopRite
- WinCo Foods
- Grocery Outlet
Brand pairs ranked in the same order as the purchase panel
Each crossing above is one reversed pair · Ties count halfSurvey consideration
78.6%Aaru predictions
100.0%Aaru puts Kroger ahead of ALDI, matching the purchase panel.
Accuracy across purchase outcomes
When using purchase consideration to rank observed purchasing, the consideration scores from the survey panel had a 77.8% agreement with the transaction panel across 340 brand pairs. In comparison, Aaru’s purchase predictions, averaged across three runs before brands are ranked, had a 92.4% rate of agreement on the same pairs. Compared to the survey, this is roughly a two-thirds reduction in error. In a baseline comparison using public measures of brand size (e.g. store counts, subscriber counts), we saw a 79.3% agreement to the purchase data across the 333 pairs where a size measure was available.
Brand pairs ranked in the same order as the purchase panel
+14.6 percentage points higher agreement when Aaru is asked about purchases, compared with the published survey (95% confidence interval: 7.2–23.0 points). Aaru’s result averages the predictions of three runs before brands are ranked; individually, the runs scored 91.5%, 91.8%, and 92.9%.
We found the behavioral simulation’s gain over the published survey was 14.6 percentage points. On the brands we tested, Aaru’s simulations order brands more closely to the transaction panel than stated-preference surveys do. This gives us a way to test simulations against what people actually buy.
Predicting our own errors
Alongside accuracy, it’s important to understand where results are likely to be accurate, and where greater caution is warranted. For each question, our system also returns a predicted TVD alongside a probability that the error exceeds each threshold (e.g., 10 or 16 points). These estimates are made by the system as part of the simulation without consulting the scored reference outcomes. We use predicted TVD to rank questions by expected error, reversing the ranking when evaluating whether error falls below a threshold.
Across 2,993 questions, there is meaningful separation between more confident and less confident forecasts.
Keeping roughly half of the questions with the highest confidence gives a mean observed TVD of 4.54 points, compared with 7.62 for the full set. In this retained subset, the share of questions with TVD above 16 points is 1.0%, compared with 9.5% in the full set. Changing the confidence threshold allows us to trade coverage—the percentage of questions retained—for accuracy. The predictions themselves are unchanged.
Mean TVD
4.54points
40% lower than the full set’s 7.62 points
Questions with TVD above 16 points
1.0%
89% lower than the full set’s 9.5%
Errors above 16 points set aside
267of 283
In the 1,468 least confident questions
Mean TVD of the retained questions
Retaining 1,525 questions (51.0%): mean TVD 4.54 points; 1.0% of questions have errors above 16 points; 267 of the 283 errors above 16 points are in the questions set aside.
Across the full evaluation, the correlation between predicted and realized error achieved Pearson and Spearman . We also use the area under the receiver operating characteristic curve (AUROC), which measures how often a randomly selected question above an error threshold receives a higher predicted TVD than one at or below it, with half credit for tied predictions. Our confidence predictor achieves an AUROC of 0.848 at a threshold of 16 TVD points and an AUROC of 0.825 at 10 points.
Wesel, Chen, and Liang [13] asked a similar question—how well can a model predict the error of its own simulations? For TVD below 16 points, our system achieved an AUROC of 0.845 vs. 0.736 from their strongest method. This corresponds to a 41% lower pairwise ranking error, calculated as (from 26.4% to 15.5%) when comparing questions on opposite sides of the threshold. Note that errors exactly at 16 points fall outside that event; 19 Aaru questions fall exactly at the boundary.
Aaru’s Pearson correlation was 0.595 compared with 0.565 in Wesel, Chen and Liang, and root mean squared error (RMSE) was 5.32 TVD points, compared with 7.76, a reduction of 31%. We’re thankful for their contributions.
AUROC · Higher is better
0.845
Strongest published method: 0.736
Pearson · Higher is better
0.595
Strongest published method: 0.565
RMSE · Lower is better
5.32points
31% lower than the strongest published method, 7.76
Aaru’s ROC curve
Retaining the most confident questions first
At the 50% coverage target, we keep 56% of the questions below 16 points (1,508 of 2,691) and 6% of the rest (17 of 302).
AUROC by method
Aaru against five other published methods, from chance to perfect
Published methods from Wesel, Chen, and Liang[13], five-fold held-out results on about 8,600 questions.
The estimates also track the magnitude of error. Across deciles of predicted error, the mean realized TVD increases monotonically from 2.2 to 14.6 TVD points. The two highest deciles of predicted error have mean predicted errors of 9.6 and 13.0 points, against mean realized errors of 12.0 and 14.6 points, respectively.
Toward general-purpose population simulation
We began in the most familiar place we could: survey instruments themselves. Across 2,993 questions, we observed simulation error rates comparable in magnitude to the survey-to-survey disagreement rates published in the literature. Furthermore, when compared against ground-truth transaction data, we observed simulated brand ordering tracked the real world with a 66% lower error compared to surveyed consideration. Finally, for each question, the system returned a predicted error that closely tracked the realized one.
Much of the value of simulation lies in hypotheticals: products that haven’t launched, prices that haven’t been set, and messages that haven’t been published. Simulation allows for hypotheticals to be evaluated at a previously infeasible scale. Unlike a human, a simulated agent won’t get tired of evaluating the hundredth flavor concept or the thousandth shelf ordering. Accuracy is foundational to achieving this, and we’re excited to share these results.
We founded Aaru to simulate humanity because so many decisions turn on how people will behave and what they value. Understanding how people behave shapes what we build, where we invest, and what we try next. We’re building toward a future where ideas, products, policies, and messages are simulated before they’re released. This evaluation is an early stepping stone toward that future, and we’re grateful to be able to share it.
ReferencesSources cited throughout this article.View sourcesHide sources
-
Ipsos (2026). Ipsos Poll on Consumer Behavior. July 17, 2026. Question 5b, page 7; methodology, page 18. The survey was conducted July 14–15 among 1,085 U.S. adults. The figure uses the published answer shares and the archived simulation for this question.
-
Saxena (2026). Knowledge cutoff evaluation. Open-source benchmark and results, revision ef12ac9, September 2, 2026. This informed the choice of question dates in our evaluation.
-
Callegaro, Tsirlin, Yang, and Ma (2022). Test-retest reliability of four U.S. non-probability sample sources. AAPOR conference presentation. The mean sums of absolute differences across five attitude questions were 4.6, 5.2, 13.2, and 24.0 points. Dividing by two gives mean TVDs of 2.3, 2.6, 6.6, and 12.0 points. Source: slide 14.
-
Lau (2021). Assessing the consistency of online opt-in polls. Pew Research Center.
-
Redmiles, Kross, and Mazurek (2019). How Well Do My Results Generalize? Comparing Security and Privacy Survey Results from MTurk, Web, and Telephone Samples. IEEE Symposium on Security and Privacy. Appendix VIII-B, Table XII.
-
Tang, Birrell, and Lerner (2022). Replication: How Well Do My Results Generalize Now? The External Validity of Online Privacy and Security Surveys. SOUPS. Sources: Tables 2–3 and Sections 3.2 and 4.1.
-
Hu et al. (2026). SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors. ICLR. Across 20 datasets and 13,510 population and group test cases, Claude 3.7 Sonnet achieved the lowest overall TVD, 19.1 points, compared with Aaru’s 7.62. The model directly estimates answer shares. Source: Table 10. SimBench’s separate normalized score uses a different aggregation and is not used here.
-
Meister, Guestrin, and Hashimoto (2025). Benchmarking Distributional Alignment of Large Language Models. NAACL, pp. 24–49. Sources: Tables 1a and 10.
-
Fan et al. (2026). SYN-DIGITS: A Synthetic Control Framework for Calibrated Digital Twin Simulation. Preprint. Sources: Section 7.3 and Tables 9–10.
-
Suh et al. (2025; revised 2026). Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinions. ACL. Table 1.
-
Kolluri et al. (2025). Finetuning LLMs for Human Behavior Prediction in Social Science Experiments. EMNLP, pp. 30096–30111. SOCRATES achieved normalized Wasserstein distance of 0.151 across 40 unseen studies, after training on 170 studies. Responses are scaled to 0–1, then distances are averaged within each study and across studies. Our calculation with equal weight per study is 0.0445 across 77 study series, using the same 733-question subset. A separate SOCRATES setting, trained on other conditions within the same studies and supplied with reasoning traces, achieved 0.063. Sources: Section 3.2 and Tables 2–3.
-
Maier et al. (2025). LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings. Preprint. Source: Table 1.
-
Wesel, Chen, and Liang (2026). Building confidence in Simile. Blog post. The continued fine-tune of the simulation model achieved AUROC 0.736, Pearson , and RMSE 0.0776 on the 0–1 TVD scale (7.76 points). The post reports five-fold held-out predictions on approximately 8,600 questions, keeping identical question text and all questions from the same experimental sample in the same fold. Our figure includes all five reported methods. Aaru’s calculation and sources use 2,993 archived predictions with equal question weight. The comparison matches the authors’ event, TVD below 16 points; 19 Aaru questions fall exactly at the boundary. This gives AUROC 0.845, versus 0.848 for the strictly-above-16 event used elsewhere in this section. Sources: the five method figures and the evaluation description under “Problem statement.”