Why is this? A new paper by Ben Klemens of the US's Census Bureau suggests an explanation.
To see it, imagine two equally good restaurants in a town. Imagine that customers choose between them based only on private information or whims. On average, both restaurants will then get roughly the same amount of business and it's very unlikely that one will be full and the other empty. Their business from day-to-day will have a binomial distribution.
This cannot explain asset prices, because it doesn't generate the crashes we actually see; restaurants won't go from being busy one day to empty the next. So, imagine diners behave differently. They decide which restaurant to go to purely on the basis of what others do. The first diner will choose a restaurant randomly. The second, seeing one restaurant empty and one with a diner will think 'the one with the diner in is better, so I'll go there'. All subsequent customers will think similarly. In this world, one restaurant will be empty and the other full.
This doesn't describe asset markets either, because we'll only ever see booms and crashes, and never middlingly busy restaurants. But in practice, asset returns are very often average.
However, a combination of the two behaviours does give us the distribution of asset returns we see in the real world. We'll get average returns when people trade on private information or whims, because a seller will quickly find a buyer without have to cut prices much. But we'll get occasional crashes, when traders emulate each other so that selling begets more selling. It's the mix of trading on private opinion and on emulating others that drives markets.
This fits several facts.
■ Crashes are more likely in assets which investors buy in the hope that others will do so, such as commodities or zero-dividend stocks. This is because demand for these assets depends less upon private, idiosyncratic, tastes and more upon attempts to second-guess others - buying in the hope that others will buy later.
■ Crashes in most assets (gold is an exception) are more likely if uncertainty - as distinct from risk - increases. This is because in uncertain times, people attach less weight to their private opinion because - by definition - they trust it less. Instead, they look more to what other people are doing, in the hope that they know something.
■ The risk of a crash is based in the very structure of financial markets. Many professional investors are incentivised to copy others, because their pay and job security depends upon relative rather than absolute performance. As Charles Prince, then CEO of Citigroup said in 2007: "As long as the music is playing, you've got to get up and dance". It's when the music stops that we get crashes.
There is, however, a problem here. None of this enables us to predict crashes. The point is that the mixture of trading on private motives (which gives us stable returns) and imitative trading, which gives us booms and crashes, is itself an unstable mix. We can't forecast when it will change, and perhaps we never will. There is a big difference between explanation and prediction.
