FallaciesStatistical
Cherry-Picking
Also called Suppressed evidence · Selective evidence · Texas sharpshooter
Pointing to the evidence that supports a claim while ignoring the evidence that doesn't.
Definition
Cherry-picking presents the data points, studies, or examples that support a conclusion and leaves out the ones that don’t.
Every piece of evidence shown can be accurate. The distortion is in what’s missing. A cherry-picked case can be built from nothing but true facts and still give a completely false picture, which makes it one of the hardest errors to catch and one of the most common.
Form
- Here is evidence E1, E2, and E3, which support conclusion C.
- (Evidence E4 through E20, which point the other way, goes unmentioned.)
- Therefore C.
The problem is the selection, not the evidence. The fair question is always what the full set of evidence shows.
How to spot it
A trend is measured from an unusual starting point, like a record high or a record low.
One study is cited where many exist, or a few anecdotes stand in for the whole picture.
The time range, region, or group seems oddly specific.
Only the successes are shown, and nobody mentions how many attempts failed.
The source can’t tell you what the evidence it left out says.
Examples
01“Global warming stopped in 1998”
For years, skeptics argued that the world hadn’t warmed since 1998, pointing to temperature records that showed no clear rise after that year.
What went wrong
1998 was an unusually hot year because of a strong El Niño. Starting the trend line at a spike makes the following years look flat. Measured over longer periods, or from a typical starting year, the warming trend was clear, and later years broke the 1998 record.
02“Charter schools are failing our kids”
An opponent of charter schools points to several charter schools that closed or posted poor test scores, and concludes that charters don’t work.
What went wrong
Charter schools vary widely, as public schools do. Showing only the failures says nothing about the average, or about how charters compare with nearby public schools. A fair comparison looks at all of them, which is what large studies try to do.
03The testimonial page
A supplement’s website features dozens of glowing customer stories.
What went wrong
Nobody puts their unhappy customers on the homepage. The stories may all be real, but they don’t tell you what happened to the typical buyer, and they can’t, because the selection was made for persuasion.
Why it persuades
Each piece of evidence checks out, so the argument feels well supported. Checking what was left out takes more effort than checking what was shown, and most listeners don’t know what the full evidence would look like. Cherry-picking also works hand in hand with confirmation bias: people who already agree are happy to see only the supporting cases.
How to counter it
1Ask what the full picture shows
Shift the question from the examples shown to the whole set of evidence.
Those examples are real. But what does all the evidence show, not just these?
2Ask about the starting point
For trends, ask why the range starts and ends where it does, and what it looks like with different endpoints.
Why start there? What does it look like if we start a few years earlier or later?
3Ask about the misses
For success stories, ask how many tried and failed.
How many people tried this and didn’t get those results?
When it's not a fallacy
Every argument selects evidence; nobody can cite everything. Choosing the most relevant and reliable evidence is good reasoning, not cherry-picking. A single example is also fine when the claim is that something is possible, not that it’s typical.
Cherry-picking is selection that would change the conclusion if the left-out evidence were included. A good test: if you showed the excluded evidence to the audience, would they feel misled? If so, it was cherry-picked.
Sources & further reading
- Darrell Huff, How to Lie with Statistics (W. W. Norton, 1954).
- Ben Goldacre, Bad Science (Fourth Estate, 2008).
- Naomi Oreskes and Erik M. Conway, Merchants of Doubt (Bloomsbury Press, 2010).
End of entry · Back to top