I was reminded of this by an article in History Today by Philip Hatfield on the search for the Northwest Passage, a trading route that would link Europe and China via the north of Canada. This fruitless pursuit went on for centuries; the first modern ones began in the 16th century. By the 19th century, writes Hatfield: "So many expeditions were sent out and failed that it was now clear, should anyone admit it, that the Northwest Passage was not a viable trade route, but attempts continued."
Why did they continue? One reason was that men were attracted by the small chance of massive rewards; not just financial, but personal glory. Also, a pound of fact laid the base for a ton of wishful thinking. Later explorers could justify their hopes by the fact that they had better ships, preparation and technology than their predecessors. But still they failed.
This is not an isolated example. For centuries, alchemists tried to transmute base metals into gold with much the same motives: the slim prospect of a massive payoff (at least to the first man to discover the method); and the belief that they could try a method their predecessors had not.
And Werner Troesken at the University of Pittsburgh shows that the market for patent medicines in the US grew massively in the 19th and 20th centuries even though the remedies were mostly useless. Again, the same motives were at play: people would pay for the small chance of getting better; and the failure of many medicines merely fuelled hopes that different ones would work.
We're making these same mistakes today. We know that investors pay too much for lottery-type stocks, many of which are on the Alternative Investment Market (Aim), just as gamblers bet too much on longshots. Like sailors looking for the Northwest Passage, alchemists for gold and 19th century Americans for a cure, they over-rate small chances of success.
Equally, investors stay with actively managed funds despite abundant evidence of poor performance and high charges. One reason for this is wishful thinking: we hope that our preferred fund manager has learned something that the thousands of failures did not, just as later explorers hoped they'd learned from the failures of their predecessors.
Samuel Becket identified a longstanding human habit when he wrote: "I'll try again, I'll fail again. I'll fail better than I did before."
I do not say all this to suggest that people are consistently stupid. The explorers looking for the Northwest Passage were among the best sailors of their day just as the alchemists were many of the cleverest men of their time; they included Sir Isaac Newton.
Clever people make systematic mistakes. Here's a conjecture as to why: natural selection favours some cognitive biases. If our ancestors hadn't been optimistic and willing to bet upon a small chance of success, they'd never have left Africa. We owe our very existence to wishful thinking and the willingness to bet on low probabilities.
It is these same tendencies that have given us scientific, intellectual and economic progress. Richard Nisbett and Lee Ross, two of the pioneers of research into cognitive biases, have written: "We probably would have few novelists, actors or scientists if all potential aspirants to these careers took action based on a normatively justifiable probability of success. We might also have few new products, new medical procedures, new political movements or new scientific theories."
Your purchase of that lousy Aim stock was driven by much the same impulse that made you pluck up the courage to ask out the woman who became your wife - a willingness to take an unlikely chance.
Those of us who point out the countless cognitive biases to which we are prone usually miss a big point - that a world in which we were all rational calculating machines would be a joyless and backward one, if indeed it could exist at all.
