Fashionable Pessimism: Confidence Gap Revisited

Most of us hold a level of confidence about our factual knowledge and predictions that doesn’t match our abilities. That’s what I learned, summarized in a recent post, from running a website called The Confidence Gap for a year.

When I published that post, someone linked to it from Hacker News, causing 9000 people to read the post, 3000 of which took the trivia quiz(es) and assigned confidence levels to their true/false responses. Presumably, many of those people learned from the post that the each group of ten questions in the survey had one question about the environment or social issues designed to show an extra level of domain-specific overconfidence. Presumably, those readers are highly educated. I would have thought this would have changed the gap between accuracy and confidence in those who used the site before and after that post. But the results from visits after the blog post and Hacker News coverage, even in those categories, were almost identical to those of the earlier group.

Hacker News users pointed out several questions where the the Confidence Gap answers were obviously wrong. Stop signs actually do have eight sides, for example. Readers also reported questions with typos that could have confused the respondents. For example “Feetwood Mac” did not perform “Black Magic Woman” prior to Santana, but Fleetwood Mac did. The site called the statement with “Feetwood” true; it was a typo, not a trick.

A Hacker News reader challenged me on the naïve averaging of confidence estimates, saying he assumed the whole paper was similarly riddled with math errors. The point is arguable, but not a math error. Philip Tetlock, Tony Cox, Gerd Gigerenzer, Sarah Lichtenstein, Baruch Fischhoff, Paul Slovic and other heavyweights of the field used the same approach I used. Thank your university system for teaching that interpretation of probability is a clear-cut matter of math as opposed to vexing issue in analytic philosophy (see The Trouble with Probability and The Trouble with Doomsday).

Those criticisms acknowledged, I deleted the response data from the questions where Hacker News reported errors and typos. This changed none of the results by more than a percentage point. And, as noted above, prior knowledge of “trick” questions had almost no effect on accuracy or overconfidence.

On the point about the media’s impact on people’s confidence about fact claims that involve environmental issues, consider data from the 2nd batch of responses (post blog post) to this question:

According to the United Nations the rate of world deforestation increased between 1995 and 2015

This statement about a claim made by the UN is false, and the UN provides a great deal of evidence on the topic. The average confidence level given by respondents was 71%. The fraction of people answering correctly was 29%. The average confidence value specified by those who answered incorrectly was 69%. Independent of different interpretations of probability and confidence, this seems a clear sign of overconfidence about deforestation facts.

Compare this to the responses given for the question of whether Oregon borders California. 88% of respondents answered correctly and their average confidence specified was 88%.

Another example:

According to, the average number of years of schooling per resident was higher in S. Korea than in USA in 2010

The statement is false. Average confidence was 68%. Average correctness was 20%

For all environmental and media-heavy social questions answered by the 2nd group of respondents (who presumably had some clue that people tend to be overconfident about such issues) the average correctness was 46% and the average confidence was 67%. This is a startling result; the proverbial dart throwing chimps would score vastly higher on environmental issues (50% by chimps, 20% on the schooling question and 46% for all “trick” questions by humans) than American respondents who were specifically warned that environmental questions were designed to demonstrate that people think the world is more screwed up than it is. Is this a sign of deep pessimism about the environment and society?

For comparison, average correctness and confidence on all World War II questions were both 65%. For movies, 70% correct, 71% confidence. For science, 75% and 77%. Other categories were similar, with some showing a slightly higher overconfidence. Most notably, sports mean accuracy was 59% with mean confidence of 68%.

Richard Feynman famously said he preferred not knowing to holding as certain the answers that might be wrong (No Ordinary Genius). Freeman Dyson famously said, “it is better to be wrong than to be vague” (The Scientist As Rebel). For the hyper-educated class, spoon-fed with facts and too educated to grasp the obvious, it seems the preferred system might now be phrased “better wrong than optimistic.”

. . . _ _ _ . . .

The man who despairs when others hope is admired as a sage. – John Stuart Mill. Speech on Perfectibility, 1828

Optimism is cowardice. Otto Spengler, Man and Technics, 1931

The U.S. life expectancy will drop to 42 years by 1980 due to cancer epidemics. Paul Ehrlich. Ramparts, 1969

It is the long ascent of the past that gives the lie to our despair. HG Wells. The Discovery of the Future, 1902

  1. #1 by jennifergriffith71 on August 20, 2021 - 5:37 am

    Hi Bill!

    Thank you for your very interesting post! I wasn’t able to get my comment to submit – I think something is wrong with the site on my end but my confidence in that answer is very low – haha!

    It would be interesting to be able to develop and administer this test with kids – say, kindergarten, 3rd grade, 6th grade – to determine the point at which males develop over confidence and females develop under confidence. It’s notable to me that NO males displayed under confidence – extraordinary! Obviously there are social forces at work here but this has implications for teaching. (I say that with confidence 😊) I see anecdotally how girls at a very young age exhibit higher levels of confidence and resolution, but at some point there is a shift. Additionally, when I used to ask my 28 kindergarteners “who can sing?”, all their little hands would shoot up with confidence, regardless of skill level. By 5th grade that number would dwindle to about 3, although their skill/resolution had increased. Neither perception of themselves as learners/performers is accurate.

    One wonders at what point students become “calibrated” adults – if ever? Can teachers help them get there? Might prevent another Challenger debacle…

    I am Vice Principal and teacher coach now at Marty’s school. Always looking for ways to improve teaching and learning!

    Thanks, Jennifer

    Sent from my iPhone


    • #2 by Bill Storage on August 20, 2021 - 10:18 pm

      Thanks Jennifer. I noticed that overconfidence generally rises with education, PhDs being more afflicted than BS/BAs. It would be interesting to test kids in the same way. I suspect that one could easily communicate the concept of confidence, maybe on a 1 to 5 stars scale, e.g. Easy enough to do on paper. Shall we give it a try?

  2. #3 by Morgan on March 24, 2022 - 2:22 pm

    Hi Bill. Is the Amazon review of “Challenger Launch Decision” yours? Brilliant stuff. Do you have any reccomended reading in:re anything related to the topic/s covered in that book?

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: