Data Fallacies to Avoid

![Im2] Cherry Picking
Selecting results that fit your claim and excluding those that don’t.

Data Dredging
Repeatedly testing new hypotheses against the same set of data, failing to acknowledge that most correlations will be the result of chance. ![Im6] ![Im7]

Survivorship Bias
Drawing conclusions from an incomplete set of data, because that data has ‘survived’ some selection criteria. ![Im15]

Cobra Effect
Setting an incentive that accidentally produces the opposite result to the one intended. Also known as a Perverse Incentive. ![Im1]

Sampling Bias
When something happens that’s unusually good or bad, it will revert back towards the average over time. ![Im9]

Overfitting
Creating a model that’s overly tailored to the data you have and not representative of the general trend. ![Im4]

Drawing conclusions from a set of data that isn’t representative of the population you’re trying to understand. ![Im10]

Regression Towards the Mean

False Causality
Falsely assuming when two events appear related that one must have caused the other.

Gerrymandering
Manipulating the geographical boundaries used to group data in order to change the result. ![Im12]

Gambler’s Fallacy
Mistakenly believing that because something has happened more frequently than usual, it’s now less likely to happen in future (and vice versa). ![Im11]

Simpson’s Paradox
When a trend appears in different subsets of data but disappears or reverses when the groups are combined. ![Im14]

Hawthorne Effect
The act of monitoring someone can affect their behaviour, leading to spurious findings. Also known as the Observer Effect. ![Im13] ![Im3]

Publication Bias
Interesting research findings are more likely to be published, distorting our impression of reality. ![Im8]

McNamara Fallacy

Relying solely on metrics in complex situations and losing sight of the bigger picture. ![Im5]

Danger of Summary Metrics
Only looking at summary metrics and missing big differences in the raw data.