Join our community of smart investors

The information paradox

From one perspective, this is puzzling. The IT revolution means that information has become more freely available and more plentiful in the past 20 years. Not only does the internet transmit data instantly, but more information is being generated, such as big data, or computational textual analysis that allows researchers to analyse company sentiment better or even uncover possible fraud.

Despite this information explosion, however, stock markets don't seem much more efficient than in the pre-internet era. There are several reasons for this.

One has been pointed out by Alexander Todorov at Princeton University. He got students to predict the result of a basketball game and found that as he gave them more information about the teams their predictive ability declined while their confidence in those worse predictions increased. Information, he concluded, can worsen decisions by giving people an "illusion of knowledge". Retail investors are prone to this. Economists at the University of Mannheim have shown that even the most knowledgeable investors often hold high-charging poorly performing funds because their knowledge gives them a misplaced faith in their ability to spot good fund managers.

It's not just students and retail investors, however, who are prone to this error. So too have been banks. In the early 2000s they had big data and fancy mathematical tools to tell them how much risk they were taking. This merely emboldened them to take too much risk, thus vindicating the words of Mark Twain: "It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so."

One reason for this illusion of knowledge has been uncovered by other researchers at Princeton. Experiments by Anthony Bastardi and Eldor Shafir show that people place too much weight upon information if they have taken trouble to get it - even if that trouble is merely waiting a while. Information, like many other things, is subject to an endowment effect: we overvalue it simply because we've taken time and effort to acquire it. This can lead us to trade upon useless or stale information. And the more of it we have, the more likely we are to do so.

It's not just our own information we can have too much faith in: an information-rich world can cause us to have excessive confidence in others' knowledge. Brock Mendel and Andrei Shleifer at Harvard University have shown that this can lead to investors "chasing noise" - buying assets that have risen because they wrongly believe that earlier buyers know something. In this way, bubbles can emerge. They claim that this was the case in the mortgage derivatives market in the mid-2000s. It might also explain the rises in share prices and sterling early last week in the mistaken belief that the UK would vote Remain: individual traders assumed that others knew something, which merely led to the blind leading the blind.

All of this supports a point made by Nikos Askitas at Bonn's Institute for the Study of Labour - that data, like most other things, is prone to diminishing returns. "An abundance of data may be counterproductive," he says.

There's another problem. Even if many investors are both rational and well-informed (two different things) markets might still be inefficient. Stocks that are prone to bubbles are often volatile and illiquid and so hard to short sell: this short sales constraint contributed to the tech bubble in the late 1990s, and it explains why Aim stocks have generally been over-valued most of the time. This means that even if information is available it does not necessarily get embedded into prices.

Above all, though, we must remember that even the best and fullest information can give us only the faintest glimpse into the future simply because this is largely indeterminate, at least to the degree of precision that investors need. This inherent uncertainty gives us a big space in which cognitive biases can distort our judgment.

There's an old story about a judge who, after hearing interminable evidence from a barrister, declared: "I'm none the wiser." The barrister replied: "No wiser, your honour, but much better informed." Financial markets are like that judge.