Why you should avoid the hottest hedge fund hands

Investors who don't have money with Pershing Square Capital Management are likely salivating at the hedge fund's industry-leading 26 percent return from January through July.

But investing with Bill Ackman and other top-performing managers after a great run is probably a bad idea, according to a new study of long-term hedge fund industry performance.

A white paper by Commonfund, which manages nearly $25 billion for close to 1,500 endowments, pensions and other institutions, shows that putting money with the hottest hedge fund managers can work in the short term, but that sticking with them for three years or more is worse than picking managers at random. Picking up losing hedge fund strategies can even produce slightly positive performance.

"Not only does positive-return persistence tend not to work as a selection strategy, but it is especially ineffective in the medium-to-long-range horizons that institutional investors may prefer, and indistinguishable from a strategy of selecting losers," authors Kristofer Kwait and John Delano wrote.

Kwait and Delano found that picking winning hedge funds produced returns of 13.29 percent after 18 months, versus an average of 10.62 percent for all funds. But the same group held for 36 months gained the same as the average; over 48 and 60 months, they rose just 9.49 percent and 8.48 percent, respectively.

Read More Ackman, Loeb,other activists hardly finished: Pros

In theory, hedge fund allocators could invest with the best managers and then quickly cash out within 18 months. But the vetting and subscription process to get in can take months or even years, especially for cautious institutional investors. Plus, hedge funds often require that their investors commit money for at least a year, and then restrict redemptions by spreading them out over several quarters.

Commonfund said its findings were consistent with a point made by Cliff Asness of AQR Capital Management.

The hedge and mutual fund manager has written that investors often try to catch short-term results in various asset classes but use a multiyear time frame, which often means they instead get hit with losing reversals-or miss winning ones-when the trade inevitably reverts to the mean. Asness declined to comment on the Commonfund study.

Ackman, for example, underperformed stock indexes in 2013 with a 9.3 percent gain, hurt by losses on J.C. Penney and a short position on Herbalife . But Pershing Square has rocketed back this year with wins on Allergan (AGN), Canadian Pacific (Toronto Stock Exchange: CP-CA), a reversal in fortune for Herbalife (HLF) and others, according to a recent letter to investors.

Another famous example of a hedge fund reversal is Paulson & Co. John Paulson saw his client base increase dramatically after he scored huge returns with bets against the housing market before it crashed. But heavy losses in 2011 and 2012 from too-early bets on the U.S. economic recovery caused investors to pull their money (Paulson's hedge funds snapped back in 2013).

Read More He's back: The best hedge fund of 2013 was...

The study also found that teasing out manager skill, or "alpha," from the general market ups and downs, or "beta," is critical to selecting hedge fund managers who will outperform.

"For an allocator, that this relationship between observed alpha and skill is not necessarily certain may leave a door open for inferring a sort of skill even from beta-driven returns, perhaps on the basis of a hard to define but powerful argument that a manager is 'seeing the ball,'" the paper noted.

The Absolute Return Composite Index, which aggregates hedge fund returns across all strategies, gained 3.79 percent this year through July. By comparison, the S&P 500's (^GSPC) total return was 5.66 percent and Barclays Aggregate Bond Index (NYSE Arca: LAG) gained 2.35 percent.

The challenge is parsing out what alpha really is; hedge fund managers are quick to attribute their gains to skill rather than market forces.

Large investors-and their teams of advisors-appear convinced they can draw the distinction. No less than 97 percent of 284 institutional investors surveyed recently by Credit Suisse said they plan to be "highly active" in making hedge fund allocations during the second half of 2014. That's even more than the 85 percent who already made allocations in the first half of the year.

According to industry research firm HFR, investors allocated $56.9 billion of new capital to hedge funds in the first half, pushing total global assets to more than $2.8 trillion, surpassing the previous record of $2.7 trillion from the prior quarter.

Read More The big money still loves hedge funds

-By CNBC's Lawrence Delevingne