The paradox of skill

When advisors, consultants or investors select a manager, their approach involves qualitative and quantitative analysis. The latter requires analyzing past performance, focusing typically on managers with a positive alpha (outperformance relative to a benchmark or asset pricing model).

New investors cannot reap past returns. Therefore, the objective of the analysis is to determine whether outperformance in the past is any indication of skill, or simply good luck. In other words, is a manager’s positive alpha likely to persist in the future?

Finding out is challenging. Several studies of manager performance found little persistence in past performance. Investors rarely hire managers with a history of poor relative performance. Manager selection often turns into a merry-go-round of hiring managers who have outperformed in the past; and firing managers who underperform in the future.

Here’s five reasons why this is an exercise in futility.

1. It takes time

To begin with, a quantitative analysis of past performance should incorporate tests of statistical significance to determine the likelihood their true alpha is not zero. In most cases, a long track record is required for the manager’s alpha to be statistically significant.

Given the average alpha and the standard deviation of the alpha, we can determine the track record required (in years) to obtain a t-statistic of two (see Table 1).

TABLE 1: Minimum track record for a statistically significant alpha (t-stat > 2)

		Average Alpha
		1%	2%	3%	4%
Standard Deviation of Alpha	4%	64	16	7	4
	6%	144	36	16	9
	8%	256	64	28	16

A t-statistic tests if an estimated value is reliably different from zero in a statistical sense. It is common to treat a t-stat greater than 2.0 in absolute value as “statistically significant.” If the estimated quantity is from a normal distribution, or if the estimated value is based on a large sample, a t-stat of 2.0 indicates that the probability of the true value being zero is about 5%.

A track record of outperforming a benchmark or asset pricing model by an average of 2% per year (net of fees), over the life of the fund, would get the attention of many investors. Especially when you consider the equity premium might only be around 5%.

A representative standard deviation of alpha in the Morningstar universe of actively managed U.S. equity mutual funds is approximately 6%. A 2% average alpha and a 6% standard deviation of the alpha requires a track record of 36 years before you can be 95%sure the true alpha is not in fact zero—i.e., there was no skill at all (see Table 1). Based on this, by the time you’re reasonably confident, the manager is likely retired and on her yacht.

2. Effects of chance

Identifying a skillful manager involves more than simply narrowing down the universe to funds with positive alphas and a t-stat of 2 or more. This approach ignores the effects of chance. There is still a 2.5% probability the outperformance was due to luck, and the true alpha of the manager is zero. In other words, one out of 40 managers is expected to have a positive alpha with a t-stat of 2 by sheer chance.

With so many funds in the universe, many will have statistically significant alphas even when there’s no skill. For example, in a 5,000-fund universe, 125 managers are expected to have a positive alpha with a t-stat greater than 2.0, even if their true alpha is zero. Unfortunately, the opportunity is not in sorting through the many managers with statistically significant alphas, but in finding these managers—because they are few.

Eugene F. Fama of the University of Chicago Booth School of Business and Kenneth R. French of the Tuck School of Business at Dartmouth College recently studied 3,156 U.S. equity funds and compared their performance to a simulated universe of funds in which the true alpha for every fund was zero.

Their results identified fewer funds with statistically significant alphas than you would expect to find by chance.

3. By then, it’s too late

An investor concluding a statistically significant alpha is evidence of skill could be guilty of data mining—he is making inferences from what might have been a chance outcome in that time period. To counter this, academics conduct out-of-sample tests to confirm a statistically significant result.

For example, out-of-sample data can be obtained by repeating the experiment using an independent time period (e.g., 1926 to 1962 rather than 1963 to 1992) or a different data set from an overlapping time period (e.g., international rather than U.S. market data).

Practitioners should also conduct out-of-sample tests when analyzing manager performances to help rule out that a statistically significant alpha didn’t occur by chance. On the surface, conducting out-of-sample tests might seem like an overly cautious approach, akin to wearing a belt and suspenders. However, even if this were true, when you consider the consequences and what’s at stake, it sure beats getting caught with your pants down.

The only way to test out-of-sample data when doing performance analysis is by using totally independent time periods. Accordingly, the number of years required (in Table 1) gets multiplied by the number of independent periods you’re comfortable with before you have faith there’s some amount of robust and repeatable skill.

Using the prior example of a 2% average alpha and a 6% standard deviation of the alpha means you need a track record of 72 years if you’re satisfied with a statistically significant result from only two independent time periods. But by the time you think there is evidence of skill, the manager may be dead (see Table 2).

TABLE 2: Minimum track record for two independent periods with statistically significant alpha (t-stat > 2)*

		Average Alpha
		1%	2%	3%	4%
Standard Deviation of Alpha	4%	128	32	14	8
	6%	288	72	32	18
	8%	512	128	56	32

* Assumes the average alpha and standard deviation are the same in both time periods.

4. Persistence

Assume you’ve found a manager with statistically significant alpha in multiple independent periods and she is not yet retired or deceased. Will the positive alpha continue? You cannot rule out luck because of the effects of chance noted above, but more importantly, many performance studies conclude winners don’t continue to win, and even when there’s alpha in the extremes, it doesn’t persist. The only slight indication of persistence is among the extreme losers, and high fees and high turnover mostly explain that persistence.

If we eliminate these funds from consideration, manager selection becomes a random draw. But whether investors know they’re picking them at random is another question. Their goal is often to achieve top-quartile performance, and the pursuit of this goal usually boils down to choosing from managers among the top quartile in the past. However, excluding the persistent losers noted above, yesterday’s top-quartile performers have the same 25% probability of being in tomorrow’s top quartile as every other manager.

5. Scarce resources

Ironically, the predicament is not only for investors trying to identify skill, but also for managers trying to prove they have it. A fundamental of economics is that the scarce resource captures the rent. If capital is freely floating and perfectly liquid, then the scarce resource isn’t the investor’s money but the manager’s skill. There’s an enormous economic incentive for managers to indisputably prove they are in possession of this elusive ability.

Let’s assume a manager has a 20-year track record of outperforming by 4% each year. In this extreme example, a t-stat is meaningless since the standard deviation of the alpha is zero. If we rule out the possibility of a Ponzi scheme, the manager has undeniable skill. But when she proves her unique ability, she can capture the rent by increasing her fees to nearly 4%.

An increase in fees of this magnitude may draw the ire of investors, so, alternatively, she could raise more and more assets, thereby distributing her alpha over a larger asset base, which would dilute investor results. This approach may go unnoticed by investors, but either way they lose, as their alpha subsequently becomes zero.

Making progress

Herein lies the paradox of skill. Many investors are searching for the Holy Grail of fund management. Their goal is to identify a skillful manager with certainty and participate in future returns. But confirming skill takes an investment lifetime, and you can never be fully confident that the alpha is not random. Even if you could identify skill ahead of time, you probably would not benefit. Winning managers hike their fees or attract large volumes of new investment long before their skill is statistically confirmed—and both actions can dilute returns.

But this paradox is not a case of “damned if you do, and damned if you don’t” for all investors. You can get off the manager selection merry-go-round and start making progress toward a successful investment experience by following these simple principles:

• Diversify by asset class rather than by fund manager.
• Buy into markets, not managers. Let capitalism be your guru.
• Focus on what you can control—costs, asset allocation, risks and discipline.
• Ignore what you can’t control—media, prognosticators, market returns and your gut

Brad Steiman is Director, Head of Canadian Financial Advisor Services, and Vice President of Dimensional Fund Advisors Canada ULC. brad.steiman@dimensional.com.