I think one of the challenges in speculation is in deciding what data set to use, i.e  what part of the data do you include and exclude, whether it be by filtering for days that share similar characteristics, or by selecting a date range based on a known structural break. How do you decide what portion of history is relevant? And is it valid to use history to decide what portion of history is relevant? Jeff Watson's question is interesting: "what if you were paid to do the exact opposite of what history told you?" Historically, would this have been profitable? In a way, it would be a paradox if it had been. My feeling is that there may be some high-hanging fruit that exists as a premium for trading on smaller sample of data - i.e. lower statistical significance - but with correspondingly greater risk. At the limit, there is what Aaron Brown in his book The Poker Face of Wall Street called "unquantifiable risk". What is the difference between "unquantifiable risk" and mere hunch? Do unquantifiable risk situations exist? Are they truly unquantifiable? And if so, where does the edge come from?

Nick White writes:

I think the real issue is determining the best method of adaptation to new circumstances. How does one free oneself from Pavlovian responses to old market relationships? How can we have courage to believe the data when it changes?

Every speculator's current predicament is to find a path through a new and unfamiliar environment; an environment where it seems that a great deal of what we have learned about markets — from the textbooks as well as from our own experiences — are, at least for the time being, essentially worthless.

Some of the most basic market microstructure foundations — things we take for granted every day in pricing and trading — have come under attack. Even the market-wide reliance on arbitrageurs to keep things reasonably orderly required the a priori assumption that there would always be the odd dollar or two of capital around to eliminate the anomaly… so much for that.

The ever-changing cycles have well and truly thrown us a curve-ball. More than ever, current market conditions - full of unprecedented anomalies and broken relationships as they are - require fresh thinking and an unrelenting dedication to pushing through proximate causes to find the ultimate ones.

So, as it relates to choice of data, the very fact that the market is a different beast post-September to its pre-September form means we have to be more wise in our data choice and analysis. The most stable relationships, instruments and markets have undergone seismic shifts in daily ranges and changes — the correlation shifts alone have been a wonder to observe. Yes, we have a very limited data set of ~70 observations — but, in this new environment, perhaps it is more risky NOT to use that new data set? Imagine you were out sailing on a sunny day — and, all of a sudden, conditions rapidly deteriorated from blue skies to hurricanes and enormous swells — would you continue to sail as though the conditions were still blue skies and a gentle breeze? Our heuristics have to match the conditions, not the other way around.

Having made that concession, so much now seems to be driven by market-exogenous factors. In that case, perhaps the best and most reliable data set of all — studies of human behaviors under stress and uncertainty — can supplement the lack of more numerous traditional observations. In other words, it seems like a good time to apply the appropriately-filtered qualitative data alongside the quantitative.

So much of this website is dedicated to the lessons we can learn from other fields. A recurrent theme is biology and evolution; surely now is the time for the greatest flexibility in strategy and tactics so that we can be amongst the quickest to successfully adapt to the new environment.

I'm sure there would be a great range of evolutionary examples that the more biology-savvy Specs could provide for inspiration…





Speak your mind

5 Comments so far

  1. douglas roberts dimick on December 4, 2008 12:32 pm


    In law school, one studies the paradox of analyzing and distinguishing substantive and procedural laws.

    Does not procedure affect if not effect substantive relative to correlation and assimilation of data?

    I am coding a state machine for execution protocol of my program trading system. Input sets output but program states actually determine output for a synchronous system.

    Thus, how one defines those states (procedural) determines output. Moreover, the relativity of the input to output is governed by parameters (or rules) defining state transition as well as data recognition for input-determinative variables and functions.

    I believe here we arrive at your observation…


  2. michael bonderer on December 4, 2008 12:43 pm

    On the subject of Portfolio diversification, see the recent ECB Bank study. I've been doing some preliminary work on the applications of something called 'Compressive Sensing' and finance.

    Bottom-line: the author tries to conclude that a smaller active portfolio may perform better then a classical Markowitz mean-variance framework.

  3. Matt Johnson on December 4, 2008 3:32 pm

    Portfolio selection is critical, is fewer better? It is for me.

    Here are my thoughts on your statements,

    "How do you decide what portion of history is relevant? And is it valid to use history to decide what portion of history is relevant?"
    If you cheery pick data, your system will fail, keep it robust, use all data.

    "what if you were paid to do the exact opposite of what history told you?"
    This question baffles me. The answers are in your database, which is history to me. To do the opposite of my findings would be unprofitable.

    I think all risk can be quantified - remember liquidity, and % of position(s) compared to the total market are just two components (go/no go) in evaluating portfolio selection.

  4. Musingsofatrader on December 4, 2008 5:40 pm

    “This question baffles me. The answers are in your database, which is history to me.”

    This comment baffles me.
    Whatever happened to the disclaimer that “past performance is not necessarily a guide to the future”? Are we to believe that past data foretells the future with perfect accuracy? Isn’t this belief one of the reasons we’re in this mess?

    More generally, I think betting on history not repeating itself is very risky but can pay off big time when you have an epochal paradigm shift. From a trading point of view, betting on history repeating itself would be analogous to trend-following and betting on history not repeating itself would be similar to reversal trading.

    I think Mark Twain said it best with his famous quip:
    “History doesn’t repeat itself; at best it sometimes rhymes .”

  5. Matt on December 5, 2008 1:04 am

    “…what part of the data do you include and exclude”. That’s easy. You tinker until you find the part of the data that gives you the best theoretical profit. Then you get a lot of people to invest in your “black box”. By pure chance, and with the help of a bull market, you beat the averages for a few years and get recognition for being a superior trader. Finally, when everything crashes around you, the government bails you out.


Resources & Links