Jul

22

Would anyone advise on how to determine backtesting periods?

I presume one should choose the most recent period because it may better correlate with the present situation. But is that really true? If it is, then how far back should one include, and how far in the future can it correlate? My experience seems to say that a short backtest period can lead to a very short future prediction or even a very poor prediction. On the other hand, a longer period often leads to poor performances during the present situation.

Shane James replies: 

At the Spec Party I had the privilege to spend a reasonable period of time one to one with the remarkable Sam Eisenstadt.

His work is likely one of the best examples of creative thought in the history of financial markets. He explained to me that there wasn't much backtesting to what he/they did. He came up with some principles that made sense to him and started applying them in real time.

Now, in our so called modern world, things may have moved on (Sam graciously stated as much to the room when he was giving his views on the modern markets). HOWEVER, maybe not so much…..

Try this:

1. If your trading idea has an average holding period of a few days (preferably less) then start from today and run it in real time for the next 90 days or so. By definition, the prices upon which you are testing your ideas did not exist when you had the idea so you have already eliminated most bias if you do this.

2. If you are happy with the structure of the returns (win, lose or draw) then consider if the results were biased by any factor during your live test phase and if related to long only stock index trading then make the requisite adjustments for drift.

3. Perhaps now consider a backtest.

The point being that I think it makes sense to test on data that did not exist BEFORE you perform the backtest.
Some like to 'exclude' certain data and 'pretend' it didn't exist so they can assume that the excluded data is 'out of sample'. For instance they may take 10 years of data and use the odd number years as test data and the even number years as 'out of sample'. This might be a reasonable idea to make yourself feel more comfortable but there is an intangible and very difficult to explain benefit to performing the kind of 'spontaneous' testing set out above on data that did not exist at the genesis of your idea before one starts seeing how well a set of heuristics performed in 1971!

Leo Jia responds: 

Hi Shane!

Thanks very much for the valuable advice.

Wow, Mr Eisenstadt! I would really love to thank him for my early success stories with referencing the Value Line. But I guess it wouldn't matter to him as he might have heard from too many!

Talking about my early experience (back in the 90's), I actually had been using your suggestion all along. There was never backtesting for me — I got an idea and went to buy the stock the next day. It actually worked well overall.

Should I go back doing the "novice" way? That becomes a question worth thinking now that you mentioned it. Perhaps this goes with the valuable lessons where having had enough struggles using complex ways, one discovered the neglected simple way being far superior. In Chinese culture, Tai Chi can be considered as that type of "simple ways".

Now, a couple questions about your suggestion.

1. By putting a new idea directly live, what problem is one trying to solve? Is it the concern that poor backtesting result may make one throw out potentially a good strategy? And is this concern because of the belief that past data are already different from the present situation?

2. In what ways can this idea that seemed to come from nowhere be better than the many ideas one gets by studying historical data? I know inspirations are invaluable, but one doesn't often get those inspirations that are not the results of study. So beyond the mistrust of the correlations between past data and present situation, are there any other reasons?

Thanks again for your thoughts.

Bill Rafter writes:

I am sorry to jump into this discussion late, but think there are a few points that can still be brought.  Looking for beta over a constant period of time (say 6 months) is somewhat meaningless and useless.  It’s a bit like describing a man with one foot in a fire and another in ice as at a tolerable temperature.  You have got fat tails with market volatility and a static window might be good for a journalist, but of limited value for a trader.

At a given time there is a time period over which the study of a market’s behavior will be significant.  And let’s say that at this time it really is 6 months, or 126 trading days.  Assuming no real changes, tomorrow that time window will be 127 trading days, and so on until you get a market change.

When the sea does change, bad things can happen in a hurry and beta value for the preceding 6+ months will be of little value.  Within the last week this happened with biotech:  it had been happily chugging along with good but not extraordinary outperformance of the indices.  Then it got clobbered with huge excessive relative volatility to the downside.  Had you been adapting your monitoring of volatility you would have been prepared, whereas if you stuck with your 6-month window you would have been clobbered along with the group.

My advice to you is to learn how to deal with the market adaptively.  I assure you that if you have a monitoring mechanism which you like, if you make it adaptive you will improve results dramatically. And it doesn’t matter which signal type (momentum, volatility, sentiment) or time frame (intra-day to weekly) you favor.


Comments

Name

Email

Website

Speak your mind

Archives

Resources & Links

Search