Join our community of smart investors

See into the future

Successful investing is all about making better forecasts. So here is a tool kit that helps investors do that
January 13, 2017

Try this : tap ‘NIE 2002-16HC’ into your search engine. You’ll get a prompt even before you’ve finished ‘NIE 2002’. Google’s algorithm knows this sequence only too well. Soon, you’ll be downloading the 96-page PDF of a document that was written in October 2002 and from the consequences of which the world is still reeling.

You want to know why Britons voted for Brexit? Consult NIE 2002-16HC. Why did America elect Donald Trump as president? Refer to NIE 2002-16HC. As an explanation for the world’s ills, almost nothing does it better. Its opening words explain why: “We judge that Iraq has continued its weapons of mass destruction (WMD) program in defiance of UN resolutions and restrictions.” There is not a trace of doubt in that sentence; no equivocating; no ‘if’, ‘but’ or ‘maybe’. Iraq has continued its programme, it says. You know the rest.

As forecasts go, few have been worse, or extracted a higher price. And such was the blow to the morale and the credibility of the US intelligence services that jointly produced NIE 2002-16HC that they decided to do something about it. If the collective wisdom of 20,000 US intelligence analysts backed by an annual budget of $50bn could be so useless, then something had to be done to improve forecasting skills.

 

 

Thus was born the Intelligence Advanced Research Projects Activity and from that came a massive tournament to see if some teams were better at forecasting than others; more important, to see if the best forecasting teams could be consistent winners and to categorise the methods that the winners used.

The connection with investing is obvious. Making money on the financial markets – whatever the market and the timescale – is always about making good forecasts, even if that forecast is as passive as deciding it’s better to save today rather than to spend. Sure, we all know that some punters look good for a while; their predictions make out and they look smart. Then their luck runs out. What’s needed is to find the forecasters who consistently defy the restraints of the efficient market hypothesis and to understand how they do it. The forecasting tournament put together by the spooked spooks comes up with some ideas.

The tournament was won – and won by an impressive margin – by a team that called itself the Good Judgement Project, which was run by a psychology professor from the University of Pennsylvania, Philip Tetlock. This was probably not by chance. Professor Tetlock had form. He had been studying the nature of forecasting since the early 1980s and had cemented a reputation in the field with Expert Political Judgement, which was published in 2005. While the book was an academic treatise, it reached the headline conclusion that – in terms of their forecasts – “the average expert was roughly as accurate as a dart-throwing chimpanzee”.

Having explained that expert forecasts tend to be useless, Professor Tetlock’s newer, more approachable and highly praised work is a handbook for how to do it better. Superforecasting: The Art and Science of Prediction does roughly what the title says – using the intelligence gathered from the success of the Good Judgement Project, it aims to show what’s needed to make better forecasts more consistently. Chiefly, it comes down to the abilities of the forecaster; some of which can be learned, all of which must be practised. These can be grouped under four headings:

Outlook: You might imagine a superforecaster must be confident and assertive, just like the best talking heads in the media. Yet the talking heads tend to make lousy forecasts. As Daniel Kahneman, a Nobel Prize-winning economist and former colleague of Professor Tetlock best known for his own economics best seller, Thinking, Fast and Slow, says: “Declarations of high confidence mainly tell you that an individual has constructed a coherent story, not necessarily that the story is true.” What’s really needed is for a forecaster to be comfortable with a sense of doubt; to be aware that reality is horribly complex; to know that humans – partly through fear of that complexity – jump to conclusions sooner than they should and to guard against that in their own deliberations.

Thinking style: The superforecasters in the Good Judgement Project were not in the genius range of IQ, but they were smart, tending to be in the top fifth of the population by that measure. More than that, they were strong on what’s labelled ‘rationality quotient’. This means they tackled tasks sensibly and efficiently; setting priorities logically, sifting evidence assiduously and reflecting on its implications. That approach allowed them to be open-minded, eager to sample a range of interpretations on a question and willing to change their mind. Indeed, Professor Tetlock says that if he were to distil superforecasting into its essence – into one sound bite – it would be: “Beliefs are hypotheses to be tested, not treasures to be guarded.”

Forecasting methods: Consistent with people who are relaxed dealing with doubt, star forecasters are good at considering multiple points of view. Professor Tetlock uses the metaphor of the dragonfly’s eye, which has up to 30,000 lenses each of which points in a slightly different direction. That gives a dragonfly extraordinary powers of vision as it gathers visual information from all around itself.

The best forecasters gather information in a similar way, aggregating across a broad spectrum of opinion then synthesising a conclusion. It’s a thought process that doesn’t come naturally – people are generally much happier to jump to a conclusion – and it’s allied to the ability to think in terms of probabilities. That’s a tough one, too, because, as Professor Tetlock says: “People really have a hard time distinguishing shades of maybe.” Put another way, the best forecasters – much like the best poker players – can spot the difference between a 55:45 probability and one of 45:55.

Fine-tuning such as that pays off in the long run if it’s spread over many forecasts; in the case of the poker player, over many hands; and, in the case of an investor, over many investments. The expectation is not to win them all – that would be hopelessly unrealistic – but to win more than you lose. That way, you make money.

The chances of success are also raised if forecasters both learn from where to gather new information and how to respond to it.

As to the gathering, it’s a fact of life that useful information is usually widely dispersed. That means forecasters often work best in teams, which, almost by definition, pool lots of information. In addition, a forecaster must decide which bits of information are best gathered using easy rules of thumb and which bits justify concentrated effort.

For investors, this is especially relevant because they have a great device at hand that’s wonderfully adept at pooling information and showing the result. It’s called ‘the market’. So investors have to decide when it’s best for them simply to take the market’s assessment – say, when they want to guess the sterling/dollar exchange rate as part of a bigger question – and when it’s right to get stuck into the leg work that allows them to make their own forecast (more of that in ‘How to make a good forecast’ on page 23).

Meanwhile, maximising the chances of success is improved if forecasters learn how to respond to new information in a reasoned way. The aim is neither to overreact nor to underreact. Most people oscillate between the two – overreacting when they focus on a project, underreacting when they’re being lazy. But at the best of times it’s hard to get the balance right. The flow of new information is constant and the difficulty is knowing which is ‘noise’ (ignore it) and which is ‘signal’ (important).

The best way to manage this is to be aware of the mental devices – the heuristics – that people use to jump to conclusions. Heuristics – mental rules of thumb – are intuitive responses that save time. As such, they’re often useful, though not so much when rational responses need to be made. Superforecasters are especially good at managing these biases and the first step is to be aware of them.

Getting stuck in: Basically, this means what it says. First, it means believing that success comes as much from practice as from innate ability. Psychologists call this a ‘growth mindset’ as opposed to a ‘fixed mindset’, which assumes that the hand you are dealt can’t be improved upon. Almost needless to say, superforecasters have a growth mindset – they reckon they can learn and improve. Second – and linked to that – superforecasters are likely to persevere. If they get it wrong first of all, they work out why and do it better next time. It’s called ‘grit’, or Professor Tetlock labels it ‘perpetual beta’ – it’s the notion that there is always room for improvement.

 

 

The nitty gritty: Superforecasting and investing

Do the skills required to make a superforecaster translate into making a superinvestor? Go to the Good Judgement Project’s website and you might not think so. There, you’ll find plenty of specific questions on all sorts of important matters, but there is a distinct shortage of questions about financial subjects. So there is lots of stuff about, say, the chance of a general election in Thailand by the end of 2017 or whether Zimbabwe will get a new president by then, but the questions that might absorb readers of the IC are thin on the ground. Granted, there is one on the sterling/dollar exchange rate and an intriguing one asking whether shares in Saudi Aramco will be listed before 2018; besides those, however, not a lot.

Part of the explanation is that the Good Judgement Project is running a geopolitical challenge, so finance and business will play a bit part. But there is also the possibility that the clearly defined, time-specific questions that the project poses are not the sort of questions that investors could answer on their subject, or would even want to. When an investor asks himself whether he should buy shares in any company, he is really posing lots of questions that can’t be answered yet. Compared with the probabilistic, binomial reasoning favoured by superforecasters, assessing possible investments requires a much more nuanced, touchy-feely approach as investors juggle umpteen factors, many of which defy quantification.

That response does not stand up to examination and ‘touchy-feely’ may be just an excuse for being wishy-washy, if not downright lazy. True, asking yourself whether to buy shares in, say, Marks and Spencer (MKS) is a big question that ostensibly defies a probabilistic response because ultimately the answer must be ‘yes’ or ‘no’. Either you buy them or you don’t and lots of factors will drive you to your conclusion.

Yet frame the question in another way and the Superforecasting approach starts to make sense for investors. ‘Should I buy shares in Marks and Spencer?’ can be reconstructed as: ‘Will the price of M&S shares close above 400p on 30 June 2017?’ If it did, it would be 20 per cent higher than its level at the start of this week, which would justify a purchase now.

True, there is the generic investor’s response that ‘I’m buying for the long term so I can’t possibly imagine what the price will be in six months’ time and it doesn’t really matter anyway’. The problem with that line is that – whatever instinct might say – the further into the future that a forecast peers, the less accurate it’s likely to be. So forecasts that have a closing date within a fairly short period (maximum 12 months) are most useful; then – when it’s clear that a forecast can be rolled forward – do so.

A further objection is that the question ‘Will the share price close above etc’ is simply too big to answer. That’s a standard problem with so many questions posed by the Good Judgement Project. The solution is to break it down into component parts, each of which is fairly accessible. To help readers do this, we have an online tool – shown in the picture, right – which allows them to assign weightings and probability factors to however many subsidiary questions they feel will help answer the big one. Of course, it’s axiomatic that it’s impossible to pose all the relevant questions – that’s the problem with the future, some of it is not just unknown but unknowable. Even so, framing lots of relevant questions and doing that in a specific, quantifiable way helps get thoughts moving in a sensible direction.

True, the weightings and the probability factors that are ascribed to each question are mostly subjective, though some less than others. As we say in the section, ‘How to make a good forecast’ (see opposite) – first it helps to stand outside the big question and repackage it so that it’s part of a wider, generic issue. Thus, when we ask if we should buy M&S shares, we’re also asking what’s the chance of the London stock market bouncing 20 per cent in any six-month period; and we’re asking whether it’s likely to rise that amount in the first half of 2017. That’s because the biggest factor to move the shares in any short period is likely to be the market. So that’s an important question, which would get a big weighting, possibly the biggest. Similarly, we can ask if the high dividend yield is likely to give the share price a lift.

Having taken the outside view, we can move inside to examine those M&S-specific factors that are both quantifiable and relevant. Granted, some of those factors will be influenced by the third-quarter results for 2016-17 that Marks was due to report on 12 January. Thus, some subsidiary questions that we have posed in the big question about Marks will be revised in the light of those results or even replaced.

But let’s run through what that spreadsheet is trying to achieve. It begins by saying there are ‘x’ identifiable factors that will drive the answer to the question; in our example, ‘x’ equals 11, though it could easily be more. In addition – and very important – there are all those other factors that we couldn’t spot but which will be relevant. In the spreadsheet, we lump those under the heading ‘All unspecified factors’. Added together, the specified factors plus the unspecified ones are everything that can influence M&S’s share price, so their aggregate weighting must sum to 1.0.

The simultaneous task is to give weightings to the identified factors; the more important the factor, the bigger its weighting. Quantifying the actual number, however, is glorified guesswork. Intuitively, we know that London’s stock market will exert the biggest influence, but what fraction should that be? In practice, it depends on how many other identifiable factors are buzzing around and how influential they seem to be. Also, remember that the more predictable an outcome, the less impact it is likely to have on whatever is being forecast since the effect should already be factored into the price. As a consequence, otherwise important factors might only warrant a lowish weighting.

Then there is the little matter of all those unspecified factors; the ones we can’t spot. Because of the limitations of human judgement, they will be more than we might want to believe. As a result, their weighting should be substantial and most likely higher than the weighting of any single identified factor. The online spreadsheet automatically calculates the residual value for the unspecified factors as weightings for the named ones are assigned. If the residual weighting gets too low, it indicates overconfidence about the influence of the identified factors and that calls for some downward adjustment.

In this exercise we are looking for a weighted probability that’s as high as possible. But let’s be clear what that means. When we named the identified factors, in effect we said that these plus the residual unspecified ones are everything that can influence – in this case the price of Marks & Spencer shares by the end of June. So they are a totality – their weighting must add up to 1.0. Meanwhile, the weighted probability of a factor – its overall importance in the plan – is its weighting times its probability factor. However, each weighted probability is independent of all the others; its value has no effect on them, nor is it influenced by the others. Thus the aggregate value of the weighted probabilities is not bounded by 1.0, though in practice it’s unlikely to approach that figure.

Basically, the higher the aggregate of the weighted probabilities, the better; in this instance, the more likely that M&S shares will be worth buying. That said, there is no trigger point at which the sum of the weighted probabilities says that the possible course of action becomes imperative. In that sense, this is a relative exercise – do this forecasting exercise often enough and you will get a feel for what’s an enticing aggregate weighted probability and what isn’t. In our extremely limited experience, we suggest that an aggregate score of more than 0.5 – assuming a forecaster is being restrained – entices well enough.

Still, there is no getting away from it, following the Superforecasting route is not an easy way out. It can be daunting because questions proliferate. Within every one, there lie lots of others, all of them coming with their own degree of doubt. At least setting them down in a schedule and giving them weightings and probability factors means we are thinking in a disciplined way – and that’s really the main aim. Dwight D Eisenhower once observed that “plans are useless, but planning is indispensable”. We can say much the same about forecasts and forecasting. And we are forecasting here. If only the spooks in US intelligence had thought like that when they prepared NIE 2002-16HC.

 

The Big Question
Will the price of Marks & Spencer shares close above 400p on 30.6.17?
Company: Marks & SpencerShare price:332pDate: 9 Jan 2017
Subsidiary questionsWeightingProbability factorWeighted probabilityNotes
1What are the chances of the UK stock market rising 20% in any six-month period?0.200.250.05
2Will M&S shares be a geared play on the All-Share? (ie, what's its beta?)0.080.40.03
3The shares yield 5.5%. How likely is it that the price will bounce 20% when the yield is at that level?0.070.350.02
4Will free cash flow for 2016-17 cover the dividend at least 1.2 times?0.070.70.05
5The yr-on-yr drop in like-for-like clothing & home sales will be less than 4%0.090.80.071st qtr: -8.9%; 2nd qtr: -2.9%; 1st half: -5.9%
6Will UK food sales for 2016-17 show like-for-like growth?0.090.450.041st half: -0.9%
7For 2016-17, in clothing & home, gross profit margins will widen by more than 0.5% point0.090.650.06Management guidance is for 0 to 50 basis pts
8The estimated costs of closing 53 overseas stores will not rise above £200m0.050.750.04
9The full-year dividend for 2016-17 will be maintained at 18.4p0.050.90.05
10Net debt on 31.3.17 (the yr end) will be less than £1.8bn0.040.850.0331.3.16 - £1.81bn; 30.9.16 - £2.1bn
11Non-underlying charges for 2016-17 will be no more than the 1st half charge of £206m0.050.450.02
120.00
130.00
140.00
150.00
160.00
170.00
18All unspecified factors0.120.5na
1.000.47

 

How good is your forecasting?
The challengeThe targetThe outcomeForecast probabilityYes/NoBalanceYes/NoBrier score
1FTSE 100 closes above 7,500 on 30.6.177,501  01.0010.00
2The sterling exchange rate closes above $1.27 on 30.6.171.27  01.0010.00
3Brent crude closes above $55 on 30.6.1755.1  01.0010.00
4The Nikkei 225 index closes below 17,900 on 30.6.1717,899  11.0002.00
5The price of Marks & Spencer shares closes above 400p on 30.6.17401  01.0010.00
6Three or fewer FTSE 100 companies are taken over during 20174  11.0002.00
7FTSE Aim 100 closes above 5,000 sometime during 20175,001  01.0010.00
8Dow Jones Industrial Average closes above 24,000 on 30.9.1724,001  01.0010.00
9The price of gold closes above $1,400 sometime during 20171,401  01.0010.00
10The price of shares in Apple closes below $85 during 201784.9  11.0002.00
Yes=1; No=0Average score: 0.60