Coping with the replicability crisis

Chris Dillow

Chris Dillow
Coping with the replicability crisis

We can't trust academic research. This is the troubling implication of the replicabilitycrisis - the fact that researchers often cannot reproduce the results claimed by other researchers. This crisis is a long-standing one: as long ago as 2005 John Ioannidis said that most published research findings in medicine "are false". And it seems to afflict most natural and social sciences.

This matters for investors. Researchers claim to have uncovered over 200 ways of beating the market, based upon companies' valuations, accounts or recent returns. However, Duke University's Campbell Harvey says most of these findings are "likely false".

Which poses the question: how can we protect ourselves from being misled by spurious research? I'd do so by asking two questions.

One is: does this finding fit with other evidence we have?

The notion that there are dozens of ways to beat the market fails this test. If this were the case, we could expect to see most fund managers beat the market at the expense of less-informed investors. But they don't. This leads me to share Professor Harvey's scepticism.

Other findings, however, do pass this test. For example, I was persuaded by Erik Eyster and Georg Weizsacker's claim that people neglect correlations not just by the evidence they presented but by other facts. I've seen lots of readers' portfolios that hold expensive actively managed funds alongside similar investment trusts or tracker funds, which suggests that people over-diversify expensively because they ignore high correlations between assets.

Some deviations from market efficiency also pass this test. The tendency for defensive stocks to do better than they should was first pointed out for US equities in 1972. It has since been found in international stocks and in bonds and commodities. And my own low-risk portfolio tells me it persists in the UK now.

Similarly, the tendency for past good performing assets to continue to do well has been found not just in US equities but also in commodities and currencies, and in my no-thought momentum portfolio.

And in both cases, these results are robust to different measures of momentum and defensiveness.

My second test is: is there a plausible mechanism to explain why this finding exists, and why it might persist?

The easy part of this is finding a cognitive bias to explain it. For example, momentum investing might work because investors underreact to good news, and defensive investing might succeed because investors neglect defensive stocks in favour of glamorous growth ones.

The trickier job is to explain why the smart money doesn't exploit the mispricing and so eliminate it. Sure, it's easy to see that some stocks can be persistently overpriced because it's difficult or impossible for clever investors to short sell them. What's harder to explain is why some shares are persistently underpriced.

In the case of defensives, we've got a good idea. A fund manager who holds a lot of them might well underperform if the general market rises a lot. Because fund managers are judged on their relative performance, this would cost him his bonus and perhaps his job. For fund managers, therefore, defensives are ill-named: they are in fact risky. This keeps them underpriced.

Some economists think the same is true of momentum stocks. It's also possible that these carry growth risk. The idea here is that shares often rise because a company's growth options have become more valuable. But it's difficult to take advantage of this because so many things can go wrong when a company expands; expansion projects can go over time and budget; managers can lose sight of cost control; rivals can muscle into the market; and so on. Such risks cause momentum stocks to underprice and so give good returns to the braver investor willing to take such risks.

I confess, though, that I'm not as satisfied by these theories as I am by the explanation for defensives' good performance: the growth risk explanation seems inconsistent with the tendency for growth stocks to have been overpriced on average. And yet the facts are clear: momentum does work. For me, this is an unresolved puzzle.

My two questions, then, are I suspect strong filters: not much will get through them.

In saying this, I'm taking a dull centrist position. Our attitude to academic research should be neither a philistine rejection of experts nor the sort of uncritical goggle-eyed admiration we sometimes see in science journalism. Instead, it should be one of sceptical engagement.

Related topics

Subscribe today

Full access for just £3.37 a week:

• Tips and recommendations - to beat the market 
• Portfolio clinic & Mr Bearbull - build a well-planned portfolio 
• Expert tools - track and manage investments effortlessly
• Plus free delivery to your home or office

Subscribe Now