Some time ago someone posted a link related to backtesting strategies. I believe the idea was that prior familiarity with the data can cause one to over-assess the signifigance of a strategy, as one can very easily tweak a strategy or come come up with a rule set already knowing how it will turn out. Statistically (and I'm sure I described the issue poorly) I'm sure that this critique makes all the sense in the world. I have seen similar critiques in other places, all suggested that prior familiarity with the data is a bad thing.

The problem i have with the above is that in actual application it seems to suggest that having zero knowledge of how markets function might is an advantage, as then our tests would not be using our prior knowledge of what has already been discovered.

In practice I have found the opposite. In my experience, the more new strategy fits into one of my learned or pre-existing conceptual frameworks, the more likely it is to work in real application. In other words it is more signifigant vs. a random rule that might also test well statistically.

I wonder if the purely statistical critique of such things misses the fact that some regularities or price behavior have tended to persist over time and are related to other rules - meaning it is not just a grab bag of unrelated "ineffeciencies" one is looking for. Rather than being a disadvantage, knowledge of these things is actually a significant advantage, in my opinion. I'm considering if a classification system similar to what is used for the animal kingdom might be a useful way to classify relationships between strategies, and clear up some of the confusion (Perhaps only mine!). Then a test for significance could be done against this smaller subset, vs. (say) the average for all 24 hour periods. Judged this way it might be found that so many different "strategies" (what quantitative hedge fund does not employ at claimed 100,000 or more?) are basically all the same thing.

anonymous writes: 

I concur with what Ed said, and also found that critique confusing. Lately I come to think that it is more meant for data scientists who research on data but don't actually trade. Scientists chew the data hard and can find all kinds of regularities (I have been just through that route). And actually many of these regularities can be due to chance only leading to the situation where one can not tell which are valid and which are not. But I don't think the critique poses as a solution. I think the solution lies in bridging the mentality of scientists and that of traders in a nice and delicate way. One should start from a pure trader's mind and then proceed on to a scientist's way but doesn't get carried away. 

anonymous writes: 

I am agnostic (or given the hyperbole, that should be atheistic) as to the past returns of strategies that seek to position themselves for large, lower probability outcomes over extended periods and those that seek to profit from fleeting latency dependent methodologies.

By the nature of markets that I have studied including the early grain markets of the 19th century up to the new 'Crypto-currency' phenomenon of recent years (Are you reading this Satoshi?), the prospective probabilities of large or small moves keep changing and so must those that manage money.

Here are a few thoughts:

1. It is very instructive to start ones millisecond, second, minute, hour, day, week, month, quarter, year or decade with a view of what strategies worked well and what didn't and think of why that may continue or not. In terms of markets I would refer SpecListers to EdSpec pp (94 - 100) & pp (316 - 319). There is another email from the chair not in this thread re: Trend.

2. In terms of strategy returns (and only looking at the Survivors obviously ) the returns are HUGELY reversionary. It is quite stunning to see the names at the top and bottom of the performance tables over 12 and especially 18 months.

3. It is fully right that some firms in the self declared trend following space have made high double digit returns given the straight line moves experienced in many of the assets in their universe. I leave it to the reader to consider that whether or not these moves (or rather the internal 'structure' of these moves) will continue. Maybe they will!

4. Anyone on this site who thinks testing a set of trend following factors, applying a backtest, going live and then trading things unchanged for the next 20 years has a different view of reality than I.

5. A note from the trainers stable. Over 1/2 the returns from many trend strategies come from the choice of the volatility target and the 'sector weightings'. Whilst there is science behind the volatility target piece, the sector weightings thing is a pot luck gamble–which is just fine–but it should be noted that if a fund continues to apply a 60% weighting to the commodity space (for example) after it has experienced a massive straight line trend then, well, read the disclaimer.

6. If it is about making money and surviving well… Put it this way, three consecutive losing years until the second half of last year for some of the brand names in the space… One wonders how many investors were there left standing for the last 12 months' spectacular returns.

7. Given the above the pro-cyclical nature of flows into alternative investment strategies continues to astonish me. Gotta keep backing those winners…. right?

8. Taking a reasonably long time frame one believes that most of the time the markets behave like a casino and then there are spectacular periods that capture the imagination like recently that skew the thinking. The best combination is the two together but usually what happens is that the guy who had done well recently gets all the assets from his manager despite the negative correlation so the effect is not allowed to work.





Speak your mind


Resources & Links