A figure can be statistically correct and still answer the wrong management question.
Simpson’s Paradox shows that an association in aggregated data may disappear or reverse when relevant groups are examined separately. The lesson is not “always stratify”, but first to establish which descriptive or causal question you actually want to answer.
A figure can be statistically correct and still answer the wrong management question.
When comparing two treatments, departments or systems, one may consistently perform better within separate groups yet appear worse overall. This happens because the groups are mixed in different proportions. Pearson and Yule had already described related effects; Simpson’s 1951 paper gave the problem its modern name.
The statistical reversal is undisputed; its interpretation is not. A reversal alone does not tell us which summary is “true”. For causal questions, adjusting for a confounder may be necessary, while adjusting for a collider or mediator may instead mislead.
Mixture weights and group composition change marginal associations. The error arises when an aggregate or conditional statistic is chosen without making the inference target and the causal role of the grouping variable explicit.
AI transfer to investigate: AI may institutionalise inappropriate aggregation choices in dashboards, model comparisons and executive summaries. This is not a new mechanism, but it increases the scale and speed at which a single summary can circulate as “the answer”.
Proposed test, not yet conducted: Give participants the same deliberately constructed Simpson reversal in four report formats: an aggregate dashboard; a dashboard with aggregate results and relevant groups; an unconstrained AI summary; and an AI summary that first states the estimand, shows aggregate and group results, and explains the assumed causal role of the grouping variable. Measure interpretation errors and decisions. The hypothesis is that making the target explicit improves the choice.
The lasting lesson is disciplined questioning rather than the paradox itself: first formulate what you want to know, then decide which table or AI summary addresses that question.
An estimand is the precise quantity the research question aims to estimate. Specify that question and the causal assumptions before aggregating or stratifying; stratification is not always appropriate.
Simpson (1951), The Interpretation of Interaction in Contingency Tables
Simpson (1951), full article, §§8–11
Pearl, Simpson’s Paradox: An Anatomy — Accessible prepublication chapter excerpt
Simpson (§§8–11) and Pearl’s causal account show why the same figures can require different summaries depending on the question. The proposed test of AI presentations has not been conducted.
Help improve this case
We review your contribution before updating the text or sources.