DATA FALLACIES TO AVOID

CHERRY PICKING
Selecting results that fit your claim and excluding
those that don’t.

!
DATA DREDGING

Repeatedly testing new hypotheses against the same
set of data, failing to acknowledge that most
correlations will be the result of chance.

COBRA EFFECT

SURVIVORSHIP BIAS

Drawing conclusions from an incomplete set of data,
because that data has ‘survived’ some selection criteria.

Setting an incentive that accidentally produces the
opposite result to the one intended. Also known as a
Perverse Incentive.

SAMPLING BIAS

Drawing conclusions from a set of data that isn’t
representative of the population you’re trying to understand.

!
FALSE CAUSALITY

Falsely assuming when two events appear related
that one must have caused the other.

GERRYMANDERING

When something happens that’s unusually good or
bad, it will revert back towards the average over time.

Manipulating the geographical boundaries used to
group data in order to change the result.

REGRESSION TOWARDS THE MEAN

GAMBLER’S FALLACY

Mistakenly believing that because something has
happened more frequently than usual, it’s now less
likely to happen in future (and vice versa).

SIMPSON’S PARADOX

Creating a model that’s overly tailored to the data you
have and not representative of the general trend.

HAWTHORNE EFFECT

When a trend appears in different subsets of data but
disappears or reverses when the groups are combined.

The act of monitoring someone can affect their
behaviour, leading to spurious findings. Also known as
the Observer Effect.

OVERFITTING

MCNAMARA FALLACY

Relying solely on metrics in complex situations and
losing sight of the bigger picture.

PUBLICATION BIAS

Interesting research findings are more likely to be
published, distorting our impression of reality.

DANGER OF SUMMARY METRICS

Only looking at summary metrics and missing big
**differences in the raw data.