Weathermen make better decisions than investors

October 9, 2012

Ryan Floyd

I have been thinking a lot about how I can improve as an investor. I read Daniel Kahneman's Thinking Fast and Slow with interest, particularly where he mentions the importance of frequent feedback. He states that people can improve on a skill when they receive frequent feedback but can say in stasis or actually get worse at a skill when the feedback loop is long as in our industry. The comparison makes me remember my middle school days of video game addiction with fondness, recognizing that Sega Genesis' NHL '95 presented the ultimate opportunity for feedback. Each game lasted only 17 minutes. Much to my parent’s disappointment, I fine-tuned my skills with each game, learning to leave a defender back on defense, wrapping around the goal with Jeremy Roenick's strong hand, and keeping the first line in until they almost died. Each time I improved, I wanted to play more.

Long-term investors must wait much longer than 17 minutes to see whether the equivalent of their new NHL '95 one-timers go in the net or slide off target. Investing provides a perfect example of a long cycle of feedback.

How do we judge our performance during a year and a half holding period? Results today are the fruit of seeds sown two years ago, not last quarter. I will know in 2014 whether I made solid analysis in the first half of 2012. I think this explains why it's so easy to stand still in this industry, getting stuck doing the same kind of analysis on the same sorts of companies without making any improvement. We can make mistakes without ever knowing it, and without being able to adjust for another year or two while we peer at the horizon. Aside from pure stock prices, we might ask ourselves, can we find out ways to evaluate our research processes and get feedback faster than a third of a decade later?

About two years ago, I started keeping track of the decisions that we made after reading about the "Brier Score," a helpful term from an unusual source. In January, 1950, Glenn Brier described how to improve making weather forecasts in Monthly Weather Review using probabilities. He noted two key elements to making a proper forecast: the actual binary bet, and one's confidence. He created a numerical description of weathermen's forecasts: the "bet" is binary, with 0 meaning that it will rain and 1 assuming it will. And for the score, we square the difference between our best estimate of the probability of the event beforehand and whether the event happened or not. The Brier Score benefits a forecaster if he expects a low probability of an event occurring and it doesn't happen, or a high probability of an event and it does happen. He really gets penalized when he expects a high probability of precipitation and it doesn't happen. The target value is zero.

I realize this is a bit confusing. To give an example, let's go over a few questions from my most recent bout of team trivia at a beer and pizza place on Lombard Street in San Francisco. Yes, I asked my friends if we could keep track of our "confidence" in each round. Here are the results from Round Five.

You can see that we were really dinged on Question Two: What is the third-longest river in the world. We were very confident in our answer but were ultimately wrong. (Do you know what it is? Don't look at Wikipedia.) We can evaluate our trivia team's "bets" in two ways. First, we expected to be correct 69% of the time when in fact it was 60%. We were overconfident. Also, our average Brier Score was .093. To give you an idea, a Brier Score under .2 is considered very good at least for big data sets. I'm not sure what it would be for a 10 question trivia quiz.

I am sure you have thought weathermen are always wrong, but they actually tend to be very well-calibrated. They can adjust their analysis quickly because they receive quick feedback.

How can this be applied to investing? There are two elements to the Brier Score. First, an investor should strive to be "right" as much as he can, with as many 1's as he can muster. But more importantly, he should have a pretty good idea that the world of investing is subject to chance and he will invariably be wrong, adjusting down his confidence level for the respective bets. When in doubt, he should be humble instead of overconfident, and endeavor to have the lowest possible Brier Score that he can by appropriately predicting the probabilities of events occurring.

Sadly, this doesn't often seem to be the case. In an interesting paper called "Expert Judgments: Financial Analysts vs. Weather Forecasters," Tadeusz Tyszka and Piotr Zielonka from the Leon Kozminski Academy of Entrepreneurship and Management in Warsaw, compare weathermen and financial analysts, and find that investors exhibited a higher level of overconfidence than weathermen in their forecasts. Moreover, weathermen tend to see the world in terms of probabilities whereas investors think in terms of one best estimate. A lot of the differences between these two professions arise from the length of the feedback loop. An investor should try to follow shorter steps in his investment process than the multi-year IRR. This will probably force him to be more humble. Also, the very process of keeping track of one's confidence on one's bets and the final results enables an investor to fine-tune his forecasts.

Inspired by weathermen—I'm sure that I'm the first person to utter the phrase—I began keeping track of the underlying bets in our investment portfolio from 2010. I have the results from 2011 and they are provocative.

In all, I kept track of 122 discrete bets. Here are some country-specific examples, the probabilities I attached to them, and whether I was correct:

Sri Lanka will have inflation less than 11% (78% chance, correct)

Sri Lanka won't nationalize something (70% chance, incorrect)

Indonesia will maintain a representative government (98% chance, correct)

Bangladesh won't have a strike that lasts more than 6 days (75% chance, correct)

Ivory Coast will have a real election (70% chance, correct)

Ivory Coast's election won't derail into violence (60% chance, incorrect),

No political violence in Rwanda (90% chance, correct)

The final score: I predicted I would get 81% of the bets correct on average, but ended up only being right 74% of the time. I was over-optimistic just like our trivia team, and as the literature expects. Secondly, our Brier Score was 17.1%, well worse than my trivia team and about in line with very good political forecasters. For 2012, I revised some of my expectations down, adjusted my levels of confidence, and tried to make the bets slightly more quantitative to avoid wiggle room.

How has this helped? First, using Brier Scores to evaluate my analysis has forced me to revise down some growth assumptions for companies because I realize that I am probably too optimistic, like many people in this world. I am forced now to think in terms of probabilities, for our companies to exceed or fall below my expectations. The best investors take account of multiple futures instead of just their best single estimate. Also, I think using Brier Scores shortens the feedback loop, focusing on the most important keys to investing and improving the investment process.

I am still at the very early period in this, and I am sure I will understand this better even one year from now. I recommend this process for anyone in a field with results a year or more after one's decision. If you’re willing to go down the weatherman’s rabbit hole with me, I am happy to elaborate on my Brier Score process.