Apr

20

When we do a study based on historical data and find a statistically significant result at the 5% level, we really are saying that there is less than a 5% chance that this study is completely attributable to chance. But if we observe some pattern in recent market action and then study it, that can be a problem: the multiple hypotheses problem.

One might think that if only one test is done that only one hypothesis was tested. Sometimes this is true. Other times traders will be intense students of the markets and notice a recurrent pattern. The trader then forms a hypothesis based on this pattern. It is properly tested on the most recent data and shows itself to be statistically significant.

There are two problems with this approach. First, if "the most recent data" include the same patterns that were observed and used to form the hypothesis then we are subject to the multiple hypothesis issue. This is true because that exquisite pattern-matching machine called the human mind continually looks for non-randomness and meaning in everything it sees. The mind tries out incredibly many hypotheses all the time. Most of us cannot even guess how many hypotheses our mind tries out before we identify one as interesting. So including the data, which formed the hypothesis, implicitly includes an element of multiple hypothesis testing.

The other problem is that we already know that the data will validate our study because it was used to help form the hypothesis. So it is not independent data but inherently biased. Thus our significance tests will be biased toward acceptance.

The best way to do these kinds of studies is to form the hypothesis on one data set and to test it on another completely different data set from another period.

Bruno Ombreux adds:

Or consider the same period but another market. For instance, if some phenomenon shows up in US stocks, test it on French and German stocks, too. There must be a reason for the putative phenomenon, either microstructural, behavioral, or economic. If so, it should show up in several markets. This extends the amount of testable data. One must be cautious with microstructure however, because it can differ. 

Philip J. McDonnell responds:

I do not agree with the idea of testing on data from different markets during the same time period, because many markets are highly correlated on a coterminal basis, sometimes as much as 90%. So it is really not an independent test on independent data.

But when one uses different time periods the correlations drop to near zero. So we can conclude that the data are truly out of sample.

Bruno Ombreux replies:

Dr. McDonnell is 100% right, but I still think it is not completely worthless to extend the sample to other markets. If you test a hypothesis on the US market, you'll be interested in the cases when you reject the null. Now, you test the German market and you still reject the null. You're right — not very useful. But if you fail to reject it on the German market, you need to come up with a very good explanation why it would work in the USA and not in Germany.

This is not nearly as good as different time periods, but it can be useful and increase understanding. 

Yishen Kuik adds:

I like to take an idea that has demonstrated its worthiness in actual trading in the US, then port it to other countries to see whether it works or not. If one has a group of countries for which the idea works and another for which it does not, it becomes interesting to try to figure out what members of each group have in common.

Nigel Davies remarks:

Presumably you're also taking account of time zones here. I've noticed that other markets tend to be led by the US during the day session (and even a couple of hours before its open) and have their measure of independence at other times. China is probably leading the overnight action now and Europe dominates during its morning. So perhaps it's not so much cultural as different time snapshots showing a certain similarity.

Martin Lindkvist extends:

Like the human flus that originate in Asia, many market ones seem to come from there too. Now, last night's Chinese flu seems to be of the same strain as that of late February. And as such, the market's immune system should be better prepared now. Perhaps a bit of coughing, and some sneezing for a little while, but not much of a fever this time? 

Henry Carstens adduces:

From a book recently recommended to me: "Routine design involves solving familiar problems, reusing large portions of prior solutions. Innovative design, on the other hand, involves finding novel solutions to unfamiliar problems." To borrow a quote from a friend, "Better necessarily means different." 


Comments

WordPress database error: [Table './dailyspeculations_com_@002d_dailywordpress/wp_comments' is marked as crashed and last (automatic?) repair failed]
SELECT * FROM wp_comments WHERE comment_post_ID = '1381' AND comment_approved = '1' ORDER BY comment_date

Name

Email

Website

Speak your mind

Archives

Resources & Links

Search