056: Campbell Harvey on Improving Significance Tests, the Importance of Positive Skew and the Future of Blockchain

Campbell R. Harvey is Professor of Finance at the Fuqua School of Business at Duke University and a Research Associate of the National Bureau of Economic Research in Cambridge, Massachusetts. He served as Editor of The Journal of Finance from 2006-2012 and is President-elect of the American Finance Association.

Professor Harvey obtained his doctorate at the University of Chicago in business finance. He has served on the faculties of the Stockholm School of Economics, the Helsinki School of Economics, and the Booth School of Business at the University of Chicago. He has also been a visiting scholar at the Board of Governors of the Federal Reserve System.

Campbell received the 2014 Reader’s Choice Award for the best paper published in the Financial Analysts Journal and the 2015 prize for the best paper published in the Journal of Portfolio Management. His recent work on evaluating trading strategies has won best paper awards.

Campbell’s research interests include statistical methods, risk management, asset allocation, real assets and cryptocurrencies. He is the Investment Strategy Advisor to the Man Group plc, the world’s largest, publicly listed, global hedge fund.

Economics:

In this interview, Campbell mentions: t-statistics, significance tests, trading strategies, investment premium, beta, correlation, standard deviation, confidence interval, P-value, Bonferroni multiple testing method, Type I error, Type II error, probability, normal distribution, optimal portfolio, volatility, expected returns, portfolio, pay-off, skew, over-fitting, regularisation, Efficient Market Hypothesis, Fractal Markets, stock market anomalies, Straw Man Model, momentum effect, mis-pricing and outliers.

Economists:

In this interview, Campbell mentions: Nassim Taleb, Benoit Mandlebrot, Peter Edgar, Yan Liu and Eugene Fama.

In this episode you will learn:

• why it’s important to use t-statistics and significance tests and how it can be improved.

• about the very simple idea Professor Campbell Harvey applies to his statistical modelling to improve the robustness of his tests.

• why it’s wrong to use 2 standard deviations to have 95% confidence when running many tests.

• about ‘Significant’, the XKCD cartoon that illustrates the vulnerability of statistical significance testing.

• do green jelly beans really cause acne? How significance tests can mislead with a fluke.

• how a trading strategy based upon picking a portfolio of shares based upon the first letter of a ticker symbol showed that those tickers that began with the letter A outperformed other stocks.

• how testing multiple times is effectively data mining and what should be done about it.

• about the meaning of 95% confidence and 5% level of significance.

• what a p-value is and why we ant it to be as small as possible.

• if it’s important for the finance and economics profession to look at how other sciences are applying testing methods?

• whether we need a tougher standard to lower the possibility of false discoveries?

• if there is a chance of a fluke finding and why we should apply the Bonferroni multiple testing method solve this?

• about the decay signature of the Higgs Boson and whether it is just background noise.

• whether the findings of many published academic peer-reviewed papers are wrong.

• about Type I and Type II errors and their trade-off.

• about All Trials’ mission to make all randomised control trials made public.

• the problems when measuring and using volatility in asset returns.

• why the level of skew in a distribution must play more of an important role in risk management and portfolio selection.

• why Taleb’s Black Swan only looks at one side of the distribution – the negative side, and why we must also look at the positive side.

• how applying ‘regularization’ to portfolio selection avoids ‘over-fitting’ the data so that unexpected future outcomes can be considered.

• about the efficient market hypothesis and the 316 anomalies that have been published to refute this hypothesis.

• why the best traders are in Asia and how insider activity makes them so.

• about the rise of crypto currencies and Bitcoin and why schools across US universities are introducing modules on it.

• what is blockchain and why its is safe.

• about the bank’s idea of creating a permission blockchain.

The Problem with Significance Testing and How to Solve It

If you’re trying to see if a variable Y is associated with a variable in a significant way, we usually think of looking at that correlation and determining whether you’re 95% confident that you’ve got it right. Usually what that means is that you’re 2 standard deviations away from zero. So, zero would be there’s no relation.

It turns out that that is perfectly acceptable if we’re looking at one correlation between Y and X. However, if it’s not X, it’s X₁ you try. You try X₂. You try X₃, you try … X₁₀₀. You try 100 different things. Then the criteria of using 2 standard deviations to have 95% confidence is just plain wrong.

The reason why this is wrong, is that when you’re running 100 tests, there is going to be a high probability that something will turn up that’s 2 standard deviations from zero just by chance.

The ‘Jelly Bean’ cartoon by XKCD called ‘Significant’ illustrates how testing a hypothesis can become misleading when conducting a significance test. The hypothesis being tested here is whether jelly beans causes acne.

A randomised control trial is ‘conducted’ by scientists. This is done where, say we have 50 people with jelly beans and 50 people with no jelly beans and we count the acne. And what basically happens is that there is no significance. So the scientists don’t achieve the 95% and conclude that there is no relation between jelly beans and acne.

However, the cartoon further illustrates what happens when the color of each jelly bean is tested to see if a particular color causes acne. 20 additional randomised control trials are conducted. The cartoon shows that the link between the Red Jelly Bean and acne is insignificant. Blue Jelly Bean – insignificant. Until you get to the last jelly bean, the 20th, which is the Green Jelly Bean. They find that there is a significant relation between Green Jelly Beans and acne. The final frame in the cartoon is a headline saying ‘Green Jelly Beans Linked to Acne’.

So, if you do 20 trials, one of those is likely to show up as significant using the standard criteria and it’s a fluke.

“The idea of my research is that we need to raise the bar that 2 standard deviations is no longer – that 2 sigma is no longer – something that should be considered. We need to go much higher.” – Professor Campbell Harvey

http://imgs.xkcd.com/comics/significant.png

The Bonferroni Multiple Testing Method

When we say that there is 95% confidence, we are saying that there is a 5% chance that the finding is a fluke. The 5% is called the p-value. What you would like is for that p-value to be as small as possible. You want as small as possible probability that the finding is a fluke. So the usual p-value for a single test with just X and Y for 5%, would imply 2 standard deviations. When you do multiple tests, you need more than 2 standard deviations from zero. If there is a chance of a fluke finding, then we should apply the Bonferroni multiple testing method solve this.

What the Bonferroni does is a simple correction. What it says is ‘you discover a p-value which is, say, 0.004 and you multiply by the number of things or X’s you’ve tried, which is, say, X₁ to X₁₀₀. All of a sudden, your p-value transforms to 0.4 or 40%. That means there is a 40% chance that in repeated trials that this thing you’ve identified, say X₅₇, is a fluke. So when you use this adjustment, you discard that variable.

Quotes by Professor Campbell Harvey in Episode 56 of the Economic Rockstar Podcast:

In the practice of finance, some investment manager goes to a client and shows a great strategy and looks amazing. But they don’t tell the client or potential client that they tried 499 other possibilities and this is the only one out of 500 that worked – Professor Campbell Harvey. 

“The problem with volatility is that it is a symmetric measure, that if you’re way above the average that contributes to the same volatility as if you’re way below the average” – Professor Campbell Harvey

“I’ve being pushing for the last 15 years to reform the way that we do our portfolio analysis, our standard models, to have the skew play a role.” – Professor Campbell Harvey

“It’s also a fact that it’s really hard to find any asset return that adheres to a normal distribution. If it does, it is very unusual.” – Professor Campbell Harvey

Books:

• The New York Times Dictionary of Money and Investing: The Essential A-to-Z Guide to the Language of the New Market by Campbell Harvey and Gretchen Morgenson
• The Black Swan by Nassim Taleb
• The Ascent of Money by Neil Ferguson

Papers:

• Evaluating Trading Strategies. by Campbell Harvey and Yan Lui
• Where are the World’s Best Analysts? Campbell Harvey, Sam Radnor, Khalil Mohammed and William Ferreira
• Conditional Skewness in Asset Pricing Tests. Campbell Harvey and Akhtar Siddique, Journal of Finance 55, (2000): 1263-1295. (P56)

Other Resources:

• Garden of Econ podcast
• Hypertextual Finance Glossary – Over 8,000 Entries and 18,000 Hyperlinks: The largest financial glossary on the Internet
• The New York Times Dictionary of Money and Investing: The Essential A-to-Z Guide to the Language of the New Market by Campbell Harvey and Gretchen Morgenson

Websites:

• www.alltrials.net

Where to Find Campbell: 

Website: Duke University

Podcast: Play in new window | Download