I found a nice paper combining items of interest to the Daily Speculations: chess and statistics. Most of it is directly applicable to markets.



 When we do a study based on historical data and find a statistically significant result at the 5% level, we really are saying that there is less than a 5% chance that this study is completely attributable to chance. But if we observe some pattern in recent market action and then study it, that can be a problem: the multiple hypotheses problem.

One might think that if only one test is done that only one hypothesis was tested. Sometimes this is true. Other times traders will be intense students of the markets and notice a recurrent pattern. The trader then forms a hypothesis based on this pattern. It is properly tested on the most recent data and shows itself to be statistically significant.

There are two problems with this approach. First, if "the most recent data" include the same patterns that were observed and used to form the hypothesis then we are subject to the multiple hypothesis issue. This is true because that exquisite pattern-matching machine called the human mind continually looks for non-randomness and meaning in everything it sees. The mind tries out incredibly many hypotheses all the time. Most of us cannot even guess how many hypotheses our mind tries out before we identify one as interesting. So including the data, which formed the hypothesis, implicitly includes an element of multiple hypothesis testing.

The other problem is that we already know that the data will validate our study because it was used to help form the hypothesis. So it is not independent data but inherently biased. Thus our significance tests will be biased toward acceptance.

The best way to do these kinds of studies is to form the hypothesis on one data set and to test it on another completely different data set from another period.

Bruno Ombreux adds:

Or consider the same period but another market. For instance, if some phenomenon shows up in US stocks, test it on French and German stocks, too. There must be a reason for the putative phenomenon, either microstructural, behavioral, or economic. If so, it should show up in several markets. This extends the amount of testable data. One must be cautious with microstructure however, because it can differ. 

Philip J. McDonnell responds:

I do not agree with the idea of testing on data from different markets during the same time period, because many markets are highly correlated on a coterminal basis, sometimes as much as 90%. So it is really not an independent test on independent data.

But when one uses different time periods the correlations drop to near zero. So we can conclude that the data are truly out of sample.

Bruno Ombreux replies:

Dr. McDonnell is 100% right, but I still think it is not completely worthless to extend the sample to other markets. If you test a hypothesis on the US market, you'll be interested in the cases when you reject the null. Now, you test the German market and you still reject the null. You're right — not very useful. But if you fail to reject it on the German market, you need to come up with a very good explanation why it would work in the USA and not in Germany.

This is not nearly as good as different time periods, but it can be useful and increase understanding. 

Yishen Kuik adds:

I like to take an idea that has demonstrated its worthiness in actual trading in the US, then port it to other countries to see whether it works or not. If one has a group of countries for which the idea works and another for which it does not, it becomes interesting to try to figure out what members of each group have in common.

Nigel Davies remarks:

Presumably you're also taking account of time zones here. I've noticed that other markets tend to be led by the US during the day session (and even a couple of hours before its open) and have their measure of independence at other times. China is probably leading the overnight action now and Europe dominates during its morning. So perhaps it's not so much cultural as different time snapshots showing a certain similarity.

Martin Lindkvist extends:

Like the human flus that originate in Asia, many market ones seem to come from there too. Now, last night's Chinese flu seems to be of the same strain as that of late February. And as such, the market's immune system should be better prepared now. Perhaps a bit of coughing, and some sneezing for a little while, but not much of a fever this time? 

Henry Carstens adduces:

From a book recently recommended to me: "Routine design involves solving familiar problems, reusing large portions of prior solutions. Innovative design, on the other hand, involves finding novel solutions to unfamiliar problems." To borrow a quote from a friend, "Better necessarily means different." 



After reading the paper the Chair found, my memory has been jogged. Introduction to Statistical Quality Control, 5th ed., Douglas Montgomery, pages 95-6 discusses the use of the range to estimate the standard deviation.

An unbiased estimator of the standard deviation s of a normal distribution is s(hat) = R/d2
R = range
d2=variable depending on n

So the factor 1.6926 is really d2 for n=3 (the # of GPS measurements Schwarz used).

For n=1 to 10, Appendix Table VI on page 725 of Montgomery gives:
n d2
2 1.128
3 1.693
4 2.059
5 2.326
6 2.534
7 2.704
8 2.847
9 2.970
10 3.078

Montgomery notes that the range method works very well (retains high efficiency) for small samples sizes (n <= 6).

Victor Niederhoffer writes:

An interesting article on
ranges shows that a good estimate of the standard deviation from a normal
distribution is range/1.7. Sequential estimates of the standard deviation from
the range, for example:

              date    range   stand dev

                4 02     10        6
                3 30     22      14
                3 29    14         8
                3 28    12         7
                3 27     8          5
                3 26    15         9 

For S&P futures this might provide a good template for thinking about short-term volatility. 

Bruno Ombreux adds:

Here is one of the early articles on the ratio of range to standard deviation, featuring tables for the ratio. Of course, today one can use resampling methods to get these kinds of ratios, even from non-normal populations.



 Sometime one doesn't need to count. Here is an example. Once I was considering buying a very illiquid small cap with a huge dividend. I called the CFO and said I was an investor interested in their stock. I asked him why such a dividend? He told me it was a one-off; they were getting rid of cash they didn't need. There was no chance of such a high dividend in the future.

Then I told him his stock was too illiquid anyway. He said they could do something about it. The next morning there were 100,000 shares for sale, instead of the usual 1,000 or 2,000. Needless to say, I never bought this stock. There was no need for counting.

There are plenty of examples like that, but to my knowledge they are always in opaque markets, with few players. I could give similar examples in physical oil or even swaps.

However, when it comes to huge markets like stocks indices, big caps, or WTI (even the oil majors or OPEC don't try to call the price of oil), how can anybody believe that he knows more than the market, that he has an information advantage? In those cases, counting is the only solution.

You could reply that information is not enough; you need to process it. And someone with experience and interest in the markets is able to process information better than the rest of the financial community. This may be true. Still, for the rest of us, with less experience and wits, isn't it safer to do what scientists do when confronted with time series, that is, count?

Besides, nobody can deny the incredible efficiency of the scientific method. Just look at its positive impact on everybody's lives from the Age of Enlightenment. To deprive oneself of such a tool doesn't make sense, even if one is a superior analyst. 

Adi Schnytzer replies:

I'd like to present my critique of counting. I assume that we wish to predict where the market is heading, be it in an hour's or a year's time. Counting — as exemplified by Vic and Laurel — generally involves regressing the returns or prices of stocks on one or, at most, two explanatory variables and testing for significance. Thus, using daily data, we may ask was has happened to the S&P500 over the past few years if, on Groundhog Day, the little beast saw its shadow.

captured British sailors
We may check what happened the next day or daily for the next month or whatever. The problem is that rarely are other explanatory variables added to the regression and this is OK if those missing variables are uncorrelated with shadow viewing. But, if this does tell us about a cold winter remaining, it affects energy prices and these should appear in the regression since the S&P 500 is clearly affected by energy stocks which are known to be related to the weather. But this is not the real problem.

The real problem is that since the variance of stock price returns is relatively large in all models that have ever been built, any exogenous shock can turn ups into downs and vice versa. And it is precisely exogenous shocks (e.g. what will the Iranians do tomorrow, what will Bush do, what will the big boys do?) that counting and its big brother econometrics cannot handle at all! But the world is full of these. How many of the news items in today's newspaper have you predicted?

To be sure, many turn out to be irrelevant, but not all. And once they have happened, it's too late for the model! Now, there are people who evidently know in advance things that are not in the public domain. The Iranians know what they'll do to the sailors tomorrow, but most of us don't.

Suppose they are each given $1 mil in gold and sent home first class tomorrow after seeing the Iran nuclear sites destroyed tonight. One suspects the market might react and counting will have proven utterly useless. Suppose, on the other hand, the Iranian Navy, having proven that it is superior to the Royal Navy, decides to blockade all oil exports from the Gulf. Hmmmm…



 Imagine if you will a very bad year in the stock market with a substantial rise in interest rates. Imagine, too, the elders of the stock market having to go to the Palindrome en masse to beg him to buy back his tremendous line of shorts stock, and begging the bearish insurance company, conglomerate hard landing guy, or media forensic accountant, to say a few bullish things to prevent stock from falling to zero.

That situation sounds somewhat similar to the present except it was 1907 not 2007. In 1907 the S&P fell 40%, from ten to six, and the elders went to Boy Wonder, Jesse Livermore to buy back his shorts. Also, interest rates went to 200% rather than the five percent inversion of today.

I felt that a study of the backdrop and concerns and intricacies of how investors tried to make money in the aftermath of that environment might teach us some lessons about how to navigate 2007. It also might provide some food for thought on what we've learned in 100 years. I turned for guidance, therefore, to the Ticker Magazine of 1908. It was a 50-page monthly edited by Richard Wyckoff, similar in its concerns, articles, and advertisers to many we have today, like Stocks and Commodities, Active Trader, or Futures.

The first issue could have been written today. Except that like most things written 100 years ago, it seems to be focused on a much higher common denominator, i.e., the literate investor population of their day. I find all their articles just as timely today as when they were written, and often their insights seem much more useful than comparable journals of today.

The first issue starts out with an excellent article, Mistakes of Investors. The mistakes are divided into excusable mistakes and inexcusable ones. The excusable ones are what we would call those that occur from the vagaries of change, where the investor has taken all precautions and done his due diligence. "If his reasoning has been wrong, or if unforeseen events bring disaster, it is a misfortune. Not so, however, with "willful mistakes."

Here's Cushing's classification of of willful mistakes to avoid.

  1. Avoid inside information.
  2. Never make an investment on enthusiasm or excitement.
  3. Use your own judgment.
  4. Pay for info rather than getting it for free.
  5. Consider earning value and market value. The man who buys real estate looks to the enhancement of value more than to earnings.
  6. Don't lose confidence. The investor hears rumors of impending disaster, which, if he would reflect upon, he would see would have no effect on his security. This applies to bank runs.
  7. Stay away from names. (Even then there were touts and promoters.) No high sounding titles can make it a success if it lacks the true qualities of success itself.
  8. Don't put too much reliance on advertisements, especially red paints.
  9. The losses through mining investments (not tech) are greatest. Beware of promoters who have no reputation to lose.
  10. The greatest mistake is one of pessimism and doubt. Never let your mind fall into that chasm. Do not think because you have lost money in one investment that all are unsafe.

The most interesting article to me in the first issue was by our old friend Roger Babson, written in 1908 about bank loans. He says that when the proportion of loans to investments gets too high it's bearish and when it's too low, it's bullish, but on a time series basis for all banks, and cross-sectionally between banks within a year. He gives yearly figures from 1860 to 1906 to verify his point and then shows how the panics of 1873, 1894, 1890, 1893, 1898, and 1903, were accurately forecast by the ratio.

The key ratio he uses is 50% loans to assets, which was "In 1873, the ratio of loans to resources first exceeded 50%. Consequently a panic occurred by the spring. Another panic occurred in 1903. Again the western farmer came to the rescue and owing to bountiful crops, the recovery continued until 1897 when interest rates exceeded 2200% a year."

Thus, Babson preceded Boltan Tremblay, Colonel Ayres, the bank credit analyst, the fake doctor, and many other greats in relying upon these credit ratios more than 100 years ago. It's overdue for a test again today.

A final article in the first issue is archetypical of articles of today. A retired engineer has a mathematical way of predicting swings in markets, and shows with a chart how his method caught "the immediate trend of each market, and the beginning and end of the longer price movements, and whether stocks are being accumulated or distributed based on a balance between the volume of price movements and volume of transactions."

He catches the full movement by "eschewing selling on strong rally, and bucking an upward trend, but instead waits until the rally has run its course and the downward movement has actually begun." In that modality, let your profits run. He seems to have captured in 1907, exactly the essence of the main methods of trading futures of today, including the methods used by most CTAs and most of the books written about trading.

In addition to these articles, an excellent article on bucket shops, the harms of short selling restrictions, how a floor trader makes money, and ticker talk rounds out the issue.

I'll augment this with further insights from the subsequent issues, as they're too good to miss.

Peter Earle writes:

After a couple of years (1907 - 1911 or so), Richard Wyckoff's Ticker became the Magazine of Wall Street, which was published until 1970. In all fairness, and perhaps unsurprisingly, in its last 20 or 30 years it was but a shadow of its former self.

Wyckoff lost control of the Magazine of Wall Street in the midst of a messy divorce from his former secretary (and by that time editor of the MoWS), Cecelia, in the late 1920s. After moping about and writing for a few years, he started a small bi-monthly magazine called Stock Market Technique which ran for the last three years of his life, ending in 1935.

They are extremely rare in their original unbound format.

Bruno Ombreux writes:

The inventory at Global Investor Bookshop offers a good flavor of the market in the early 20th century, and it's true it has not changed. Some articles and some books have been reprinted.



 I have come to the conclusion that trades are soufflés and trading is cooking soufflés.

A soufflé is a quick and simple dish to make, yet it can flop easily. With hungry guests waiting at the dinner table, there is little margin for error, since soufflés have to be served hot from the oven. A messed soufflé cannot be served. Failure is public. Because of this, cooking soufflés is stressful, but it is also a source of pride, a statement of confidence in one's cooking skills.

All soufflés contain thoroughly whisked egg whites, with a variety of ingredients. Flour, butter and milk often figure in the recipe. But one can also use anything from chocolate to cheese, mashed potatoes or cauliflower. The mixture is put into the oven, at a specific temperature for a specific time. Since there are no two identical ovens, the cook must know his oven, just as the trader must know his market's idiosyncrasies.

Then it is alea jacta est followed by intense watching. If one doesn't wait long enough, the soufflé will not rise. If one waits too long, the soufflé will rise then collapse. Timing is key. Trades are the same. A trade must last long enough but not too long.

Like finance, cooking is replete with superstitions. The scientific method has been applied to cooking only recently and rarely. For centuries, people thought that soufflés rose from air bubble dilatation in the egg whites, but molecular gastronomy has shown that this is not the case. Soufflés rise because of vaporization of the water contained in the preparation.



Since it is rainy outside, I am reading a good book: Common Errors in Statistics and How to Avoid Them, by Phillip Good and James Hardin. 



 I am both an investor and trader. But looking at my results I should probably only be an investor. It is not easy to trade with a full-time job on the side.

As an investor I am 100% long with my stocks. I will stay 100% long no matter what. I can sell a stock, but only if I am able to find a better one to replace it. I am not going to sell because of the overall market. Actually, I could sell if it goes up 130% like Shanghai last year. But I am never going to sell because it has been going down.

Today, my investments are down 2% from 12/31/2006, and down 10% from February intraday peak equity. I don't care the slightest bit. They could go down 30% and I wouldn't care either.

I am not crazy. There is a very good reason for this stubbornness.

I started investing seriously in stocks in 1996. Since then there has been a crisis in 1997, another one in 1998, and one of the biggest bear markets in history in 2000-2002. I was investing with a mix of stock picking, market timing, style timing, and small/big timing. Believe it or not my market timing allowed me to sell at all the intermediate tops in 1997, in 1998, and in March 2000. It allowed me to avoid the bulk of the bear market in 2000-2002. I came back too early in August 2002, sold in September, came back at the exact bottom in March 2003!

With this nearly perfect timing, you would think I have impressive compounded returns. That couldn't be further from the true. At the end in 2005, I did a complete audit of my 10-year record. It was prompted, among other things, by some things I read on the Spec List, mostly from the Chair but not only from him. So thank you guys for your down-to-earth audit-prompting approach.

Results of the 10-year audit:

Market timing resulted in dramatically lower volatility and draw-downs than the market; but who cares? It resulted in only a 2% over-performance compared to the index. In terms of absolute returns, beating the index by only 2% is ridiculous. It is incredible that even though I caught most major tops and bottoms in 10 years, I only over-performed by 2%. Even more sobering is that if I had kept the first 10 stocks I ever bought and never sold them, forgot them and never done anything else, my over-performance would have been 4%.

How could this happen? Well, that's very easy:

First, I caught all the actual tops, but also about 10 of them which never turned out to be tops. The market continued higher and I missed part of the move. Second, even when the top was an actual top and I was flat, it created the problem of knowing when to get back in, which in most cases occurred a bit too late. Third, buying and selling too much is created a lot of friction in the form of commissions. Over 10 years, the amount paid in commissions can be really impressive.

Based on this I decided to be always 100% long. I am not timing the market, styles, or anything any longer. I still hope to continue beating the market by a couple percent a year from stock-picking (probably more beta than alpha). I don't care if the results are more volatile. This is largely compensated by a huge decrease in workload and worry. Freed time can be dedicated to more useful pursuits, like learning to trade.

Jaime Klein writes:

I have, well, had, two now only one extremely financially talented relatives. The late one, when told I was going into the financial business, laughed rather rudely, I thought. And noting so many of my family members were already in that line of work, he asked me who was going to bring home the bacon. Well, he said, seeing as you're determined, I'd give you this bit of advice: Never buy a stock if in your lifetime you don't see it returning your original investment to you annually in dividends. And if they're any good, they only pay two percent.

Absurdly enough, his own results were so far beyond this as to make this counsel seem the most conservative expectation possible. He was probably 30 years ahead of the sage into Coca Cola, which he obtained by selling Minute Maid to them for stock. He never sold it except to buy the occasional Goya or Renoir, or make a charitable donation to Harvard or MIT.

I was aware of only two other plays: one was a quick flip which his partner told me netted over 100X in less than three years. The other was selling United Fruit, which I imagine he paid near nothing for, to Eli Black, right at the top back in the conglomerate heat of the '60s. I can't remember much about the foolish and ill-fated acquisitor except that he defenestrated himself shortly thereafter, taking his briefcase along with him.

Anyway, it's been my pleasure, while unfortunately lacking in outstanding talent myself, to have met so many ingenious and interesting people in my all too brief 65 years. One of these days I'm hoping I'll learn something from them. But in the meanwhile, it's always fascinating, albeit particularly in the political and religious arenas sometimes quite alarming, to see how clever so many people are.

From Scott Brooks:

Volatility is a terrible measure of risk. There is no risk on the upside of volatility. The goal should be to reduce all down side volatility, thus my patented investment strategy of buy low and sell high (Green List/Red List post from several months ago).

In all seriousness, I am fixated on the discovery of ways to mitigate downside volatility while participating in most of the upside of volatility. But since I'm far from the smartest person on this list and have been told in no uncertain terms that it can't be done, I feel like I'm fighting an uphill battle. Still, who knows, maybe there is a way!

I've never been one to give up just because others say it can't be done. If I listened to others (like my guidance counselors), I'd probably be laying carpet back in Maplewood, going to the corner bar, watching COPS every night, and aspiring only to be the "Maplewoods, King of White Trash."

From Craig Mee:

I accept these results, however…

Plenty of you know a lot more about stocks then I do. But I would like to offer here that a two percent increase in returns and with this, the opportunity to be out of the market in major declines, represents to me some nice sleepy nights.

With a bit of fine-tuning maybe marks can be picked slightly better on entering and exiting longer term positions. But on that black swan event, when something may drive the market into a huge selling spiral, I believe for me at least it may be worth that extra agro.

From Kim Zussman:

Similar but less quantitative self-assessments:

1. At least in US, taxes bite deeply into putative alpha (or masquerading beta) if you trade vs buy and hold.

2. Concur that most effect was lowering volatility. You will get lower volatility with stocks<100%, and pretty much always lower returns. Looking back, you will regret not being 100% stocks, but during the ride you live happier <<100%. Thinking about a big down year as a future possibility feels a lot different than having one.*

3. Besides drift, the reason buy and hold works is that there is too much temptation for the vast majority of people to time the market. It is unnatural not to check your investments, and not to be tempted to act on them. People don't like it when their million $ port becomes worth $800,000, and sell before "losing it all". Then it turns around and people don't like missing up 30% years, and buy back in. The hope-panic-irony cycle makes the market rise over time only for those not riding the emotocycle.

* The abstraction of future pain and foolish willingness to fall in love is nicely summarized by the late Sam Kinnison.

Jack Tierney adds:

I was invited to a dinner party but expected very little. The guests were getting thin on top and hefty through the middle. Our host was dressed in colors that defy the known spectrum and civility was to be shown the greatest horse's rectum. So we mingled and we spoke and mentioned our positions. I mooted that I was all in cash and was swarmed by five physicians. "Perhaps an evil humor attacked him on his flight or maybe he's an infidel who has yet to see the light." Their concern was very real and they needed to be consoled so I admitted that in addition I owned a little gold. Screams and wails followed and the panic gained momentum.

To quell the crowd I shouted, "Wait, I also own argentum." Now that they were fully aware of these judgmental flaws they ripped away my velvet gloves and exposed my hairy paws. They marched me toward the door when the host yelled out to quit, "Why this poor benighted soul has never heard of drift."

So began my lessons and I've brought them to the south, a bearish thought may cross your mind but never cross your mouth.

Abe Dunkelheit adds:

 Bruno's post was very interesting. I made exactly the same observation. Market timing lowers volatility but doesn't guarantee any substantial out performance. And yes, one's first ideas tend to be much better researched than all these other in and out decisions. Never to sell them would have turned out the best in my personal case also.

And there seem to be people who don't make any professional impression and live a very retired life who tend to buy and hold and accumulate incredible returns without doing much.

I know about a guy in Switzerland who was retired and did it with wine. He bought all these Chateau Mouton Rothschild wines for USD 500 a bottle 10 years ago and they now go for USD 10,000 at auction because rap stars and Russian mafia are pushing prices up. I only know about this guy because I was one of the sellers. I had bought my bottles for USD 300 and thought a cool 60% gain in less than two years could not be wrong. He had an incredible cellar with all these wines, but his house and car and his whole appearance were very modest.

Another example I know about is a guy who was jobless and lived on social security, but had saved several hundred thousand euros [back then deutschmarks] and invested them through the accounts of his children. He put it all into Deutsche Telecom at the IPO and cashed in a 600% profit during the Internet boom. That was his one and only investment.



 I would like to offer some simple thoughts on non-linear relationships. The usual way to study non-linear correlations is to transform one or more of the variables in question. For example if we have a reason to believe that the underlying process is multiplicative then we can use a log function to model our data. When we do a correlation or regression of y~x we can just take the transformed variables ln(y)~ln(x) as our new data set. We are still doing a linear correlation or a linear regression but now we are doing it on the transformed variables.

Ideally we would know the form of the non-linear relationship from some theory. Absent that we could use a general functional form such as the polynomials. So our transform could be something like X^2, X^3, or X^4. Using one of these terms is usually pretty safe. But combining them in a multiple regression can be problematic. The reason is that the terms x^2 and x^3 are about 67% correlated. Using highly correlated variables to model or predict some third variable is a bad idea because you cannot trust the statistics you get.

One way around that is to use orthogonal polynomials or functions. We have previously discussed Fourier transforms and Chebychev polynomials. Both of these classes are orthogonal which also means that we can fit a few terms and add or delete terms at will. The fitted coefficients will not change if we truncate or add to the series. Each term is guaranteed to be linearly independent of the others.

Bruno Ombreux asks:

Using one of these terms is usually pretty safe. But combining them in a multiple regression can be problematic. The reason is that the terms x^2 and x^3 are about 67% correlated. Using highly correlated variables to model or predict some third variable is a bad idea because you cannot trust the statistics you get.

I have a question.

One of the reasons for adding regressors is to take into account all possible reasons behind a move in the variable we are trying to explain. However, multicollinearity being prevalent in finance, it is a source of headaches.

If we could randomize and/or design experience plans for empirical studies, as we do in biology, we could get rid of part of the problem.

Is it possible to randomize ex post? Let's say I what to study Y = aX+ b + e. If instead of taking the full history of observed (Y,X), I am taking a random sample of (Y,X), it creates some kind of post-randomization, which should reduces the impact of other factors.

Does it make sense? Of course, we would lose all the information contained in the non-sampled (Y,X). That means even less data to work with, which is not nice with ever-changing cycles.

Are there books about this type of technique? I have never heard about it so maybe it doesn't exist.

Rich Ghazarian mentions:

And of course if you want a more powerful model, you fit a Copula to your processes and now you are in a more realistic Dependence Structure. Engle has a nice paper on Dynamic Conditional Correlation that may interest Dependence modelers on the list. The use of Excel correlation, pearson correlation, linear correlation … these must be the biggest flaws in quant finance today. 

Jeremy Smith adds: 

With linear functions we can compute the Eigenvectors to get an orthogonal representation. One problem that gets in the way of nonlinear models is that it isn't clear what is the appropriate "distance" measurement. You need a formal metric of distance to model, compare, or optimize anything. How far apart are these points?

With linear axes, distance is determined by Pythagoras. But what is suggested for the underlying measure of distance if the axes aren't linear?

These remarks about correlation resonate with me, especially in the case of the stock market.

From Vincent Andres:

If you did replace your original axis X and Y by new axis X'=fx(X) and Y'=fy(Y) this is a transformation of the kind P=(x,y) -> P'=f(P)=(x',y')=(fx(x), fy(y)).

This transformation can be reverted without worry. P'=(x',y') -> P=(x,y) where x and y are the antecedents of x' and y' thru the reciprocal functions fx^-1 and fy^-1.

A "natural" suggested distance measure in this new universe is thus : dist(P1, P2) = dist(ant(P1), ant(P2)) ant = antecedent.

This works for all functions fx and fy being monotonous, e.g., (ln(x), x^2, etc) because there is a strict bijection between the two universes. It could even do something for a more large class of functions.

Sorry for the difficult notations, but I hope the idea is clear.



 The question is: how to take money out of the pockets of VWAP traders? This has been bothering me for quite some time and I haven't found an easy answer yet.

An old-school broker would slice a big order, perhaps 1/3 at the open, 1/3 at the close, working the remaining third at his discretion over the day. It was possible to make a few bucks intraday by fading him at the open. Since most VWAP programs are slicing according to the U-shape of volume, they are not really changing the daily volume distribution.

So that didn't change. What changed is that the book and tape have become useless. Instead of having big standing limit orders sitting like fish in a barrel, or market orders on the tape, we have hundreds of tiny little orders that are impossible for a human to follow. The Level 2 book, and the tape, are useless now, except in very tiny issues.

So: how to spot the buyers?

Very tough. I have noticed that sometimes when VWAP orders kick in they smooth the price like crazy. One could imagine an intraday smoothness detector to spot big buyers. I have also noticed that some of those programs are operating at fixed-time intervals, such as every half hour. So we should test if price behavior at the half hour is predictive of anything following.

Going beyond VWAP and into algorithmic trading, there is a recent thread with quality posts on Wilmott.

Mark M McNabb adds:

Having read some literature on VWAP I don't see much benefit for most traders, as the inferences in the canned packages seem subject to the same heuristic biases as the investment community's at large. These packages may be good for positioning large orders across markets, but the long and short of it: they are still subject to an outlook and bias. 



 Sarkozy appears on track to win the French elections in April. This is something I have been tracking for the past three years, since Sarkozy distanced himself from Chirac and started vocally turning away from the traditional European socialist model of political economy and towards free markets etc.

Could the French electorate finally be willing to push back against socialism? Will they finally begin to sell off state-controlled companies? Is there concern for Airbus's continued state support in Europe? Is this an indicator of a broader trend in Europe?

The questions and potential ramifications of a Sarkozy win in France are staggering and endless.

George Zachar responds:

Sarkozy has made lots of recent statements that leave him firmly in the "democratic socialist" part of the political spectrum. He's not overtly hard-left like Royal, but his election would hardly be a mandate for what Americans would term free markets.

Roger Arnold replies:

Relative to US standards you're right. But he's also running for office and needs consensus. The fact that he is even in the running, let alone in the lead, is an indication of a change in sentiment in France by the electorate with respect to the trajectory their current policies have them on.

The fact that his lead is increasing is an indication of his migration to the middle as the election approaches.

Bruno Ombreux writes:

I am watching this by necessity since I am in France. Sarkozy has not won yet. I think it will be a very close call.

The guy has courage. He is running his campaign on the theme the "party is over, time to get back to work." Problem is that the French people have had their minds washed by the leftist media and school system since 1968. So a big part of the electorate might not be prepared to give up socialism.

And Sarkozy, like all French politicians, remains a statist. I support him, however, because he is the less worse choice. And if he is elected and can get the country back to work, even a bit, that would be great.

The biggest change could be seen abroad, not domestically — in foreign policy, since Sarkozy is pro-USA and pro-Israel.



 If a market trades with regular volume of 20 lots a side, and trades fairly rapid fire throughout the day, you can bet your kids college fund on the fact that if a market then goes 2 lots on the bid and 80 on the offer and holds there for 4 x plus of a time period greater than the normal time of an execution without the small volume being given, that offer will be lifted pronto.

You can almost count it 1 … 2 … 3 … mine.

Victor Niederhoffer writes:

 This is a most interesting and creative observation that suggests many fruitful extensions. It may be the best way to show that volume matters - of course it's the opposite way from what is usually thought. Still, Justin Mamis had similar speculative insights.

Hanny Saad writes:

Very well stated and I agree with you [Craig]. Do you care to take a stab at what you think might be happening behind the scenes? 

Craig Mee replies:

There're probably a few options here, but certainly at times you do have one or two key buyers or sellers driving the market. When this happens, the small retail traders will lean against solid offers time and again. This will continue until a situation develops where the market has traded through several levels and where no more small sellers are there to be found, yet the one or two hungry sharks are still circling. Thus you get the situation, whereby we have only two on the bid (all retail punters are now already short or have been burned trying to lean against offers) and 80 on the offer. Thereby no one hits the bid, market holds, 1/2/3/ bang… the big fish can get size on and lift the offer….

So, in a nutshell, one reason this happens, I believe is due to one or two large corporations shipping in volume for a hedge (though I'm sure the list could offer other reasons). And if you have big enough pockets it can often pay to fade this move. Its not, however, always a smart move, unless you have a direct line to God.

Bruno Ombreux writes:

"[A]nd if you have big enough pockets in can often pay to fade this move, however its not always a smart move…"

That's one thing I've seen a few times when I was a professional oil trader. It is entirely anecdotic and not testable, but relevant to the subject. The nice thing about being a professional, is that one has people on the floor, and they comment on what's happening like, "Goldman on the bid for 10,000 lots."

Now, in the 1990s, 10,000 lots WTI was big size. And if it were Goldman, and not their commodity subsidiary J. Aron, it meant they were acting as brokers for a fund that was an outsider. When professional traders tried to move size, they rarely showed it that way. They worked it all day long, preferably through several floor brokers, to hide one's intentions.

What usually happened in bygone days, is that a lot of small people tried to front-run the 10,000 lots, on the theory that a big buy order is like a free call option. They probably expected it to be taken piecemeal, if it were to be taken, which would leave time to sell and get out if the size showed signs of being eaten away. And if the market went the other way instead of filling the order, it was free money.

But instead of the order being eaten piecemeal, we often got information like, "Hess just booked Goldman." Which meant a commercial filled the 10,000 lots in one go, probably for hedging purposes.

The market moved toward size and the result was that all the frontrunners saw nothing below, and got into a panic. It cascaded as they hurried up out of their longs and the market collapsed. It can be really fun if it happens a few minutes before the close.



Hunter and hunted or predator-prey relations are pervasive in the animal world. We're accustomed to observing and reading popular summaries and videos of the dynamics and techniques of survival for such pairs as lion & gazelle, wolf & squirrel, fox & lynx, coyote & seal, osprey & smelt, pike & minnow, and spider & fly. Such studies have been extended to romance and health among humans. Predator-prey relations are also common in markets. For example, the relation between market maker & day trader, dealer & ephemeral trader, flexible & inflexible, large trader & small trader, informed & uninformed, vig taker & vig payer.

Many studies in the field are based on the Lotka-Volterra model. This is a set of simultaneous differential equations relating to the rate of growth of the predator and prey populations to each other. A typical set of equations relating rabbit growth to fox growth states that dr/dt = ar-brf and df/dt = ebrf-cf where a is the natural growth rate of the rabbits, c is the death rate of the foxes, b is the death rate of the rabbits whenever they meet a fox and e is the proportional gain in growth that a fox gets from eating a rabbit.

Such equations do capture the main idea that as the rabbit population increases, the foxes gain in number because rabbits are easier to find and eat, and this provides a homeostatic mechanism to stabilize the rabbit population. Similarly, as the rabbit population declines, the number of foxes decreases because they have less food, and this helps increase the rabbit population which in turn tends to increase the fox population. As might be guessed, small changes in the assumptions of the model, such as time delays, lead to widely divergent behavior involving cyclicalities, instabilities and sharp changes in the dynamics that do not correspond to what we observe in most real-life populations.

A similar critique could be made of the two other standard methods of studying predator-prey relations, which are the functional response curve and the optimal foraging theory. The basic regularities there are that the costs and benefits of gaining prey vis a vis future reproductive success determine the extent and energy with which the predator seeks the prey. The key dependent variable is how much the predator eats as a function of the difficulty of converting the prey into food. An increase in the search time, handling time, or consumption time, reduces the predator's desire to eat. Certainly this leads to insights.  The problem here is that all these parameters are subject to estimation, and they are interrelated and subject to different hypotheses as to their function.

A good book for studying these techniques at an elementary level is John Alcock's Animal Behavior, and a good summary of the ecological approach to these dynamics can be found here.

Methods of studying the factors that enable predators to be successful have always been important to me as I, like other numerous individuals not at the top of the food chain, are often prey to much larger predators. I have often wanted to learn how to avoid capture, and even considered the possibility of sometimes turning the table on the predators and bagging them once or twice just to make the game a little more even sided. Thus, when I came across a cover story in Outdoor Life titled "Predators' Deadly Tricks," which describes how hunters go about capturing the most elusive predators in real life such as the coyote, the bobcat, and the mountain lion, I was very attentive and decided that I should try to devise principles from the practical and theoretical literature that might help other prey like me in their incessant battle with those who would devour them. 

  1. Signaling is key.The signals that the prey send out to show that they are not easy to digest prevent the predator from even considering attacking, and this saves much energy for the escape. Colors and scents indicate that the prey contain poisons. Stotting, the jumping behavior of gazelles when about to be chased by a cheetah, indicates that they are very mobile and not worth eating. Indeed the essence of the article is that the best way to attract a predator is with an electronic duplication of the distress call of its enemy. Amazingly, the coyote will often show himself within one minute of hearing the rabbit's call, especially when it's made with a "Foxpro FX5 that has a 200 sound capacity, one gigabyte of memory, recall buttons to switch between sounds, remote control functions, and a 700-yard range. Less than a minute later (after the call), a pair of coyotes charged in and we handily dispatched them." Market prey often indicate that they are ready, willing and able to defend themselves by the placement of limit orders in large size, but cancel if they are near just to prevent the larger predator on the other side from even thinking of going after them. The talk with your counterparts is how much more is available for adding to my line when you well know that one more grain of salt would be enough to topple you over.
  2. Vigilance is essential. The herding animals all find that 100 pairs of eyes with 50 always awake are enough to warn them of danger. Noses are always sniffing, ears are listening, and the antennae are always feeling. Indeed, some ducks can sleep with one eye open so as to never be victimized by a surprise attack. The hunter uses a telescope so that he can always perfectly see the adversary. He never lets the prey's vigilance work to his advantage by approaching stealthily, parking his equipment a mile away from where he's going to hunt, and setting up in a blind with proper camouflage. The prey in the market doesn't leave the market for a moment, as that might be the time that the enemy attacks.He cancels all orders when they don't get filled so that a surprise news announcement that's worth a limit move won't catch him just a few ticks from the last price. He has his computer set to wake him, which buzzes around in his private area so he never sleeps through a dangerous situation or lets the predator devour him totally.
  3. Deception is essential. My goodness, the moth blends in with the bark and orients with the grain of what he's sleeping on. The flies disguise themselves to look like bees, and the octopus can change 100 colors in one second. The spider uses a million deceptive lures to entice the fly into its web. The golden orb weaving spider spins a web that's so enticing that even when a bee breaks free, it will dive right back into it after it has escaped. (I am reminded here of the system player who, after a very bad trade on one side, doubles up on the other side for the next trade.) The chapter on deception in Education of a Speculator details other areas of deception in the world. "Quality camouflage is a must; select the pattern that most closely matches the foliage and landscape." Whatever you do, don't make any news. As a prey trader, I don't even like to type out that I'm thinking of exiting a trade, for fear that a predator might have my screen bugged or that the keystrokes are programmed to signal my intention. I never let the other side know what my stop point is because I know that it will always be hit. If I'm really hurting, I'll try to act 5,000 times stronger than I am, and I won't even begin to reduce my position by one contract for fear that my camouflage will be found out.
  4. Proper equipment is a must. Predators are constantly sharpening their claws and teeth. Prey must always practice escape maneuvers. Over many generations, most prey have adopted advanced techniques of escape that include the full range of methods used by individuals in their cohort from the beginning of time to elude capture, be it poison, scent, or cry. Their bodies are perfectly suited to the escape in size, color, speed and strength. The properly equipped hunter, in addition to his Swaroski binoculars and Foxpro FX5 caller, currently has a Gerber Epoch Pack, a Stoney Point bipod, Cabela coverup pants, and, of course, the obligatory Ruger bullets in a Browning rifle, a Bushnell scope, motion decoys, and a set of shooting stocks.
  5. If all else fails, try the unusual. Be prepared to shout if the predator attacks. The proper equipment for the trader starts with a proper price feed, perhaps one that's within a foot of the source of the prices so as not to lose out by the speed of light that it might take to get to you one-thousandth of a light second away.  Next, one should have a computer that's always set to trading and that isn't interfered with by email. Finally, have an office where no one can distract you from the job of survival with the cares of the world or a bill from the Service.
  6. Never give up. The cries of animals often save them from death. If nothing else, they serve to alert family members. The squirting of poison and the enlargement of the body is a common tactic of the caterpillar, and the gyrations of the weasel in extremis are often enough to ward off death. The hunter is told to scream if a predator attacks him and to have a spare set of guns and knives. As a trader, I try to follow the rules of a good competitor in sports who never gives away the last point of a game if there is still an iota of energy left in his body. There is always someone you can call on to help you fight back. On occasion, I've even asked a'la the Boy Wonder for the other half to help me out in a time of crisis, and so far the trust funds are still intact.
  7. I would recommend studying the literature on predator-prey relations by reading a few good books, following up on some of the hundreds of thousands of citations on the search engines, reading the Outside Magazine article in the December-January double issue and then trying to apply these techniques to make yourself impossible to detect, fruitless to waste energy on, and impossible to digest when caught. If all else fails, fight to the death.

J. Klein adds:

One Predator - One Prey; if it was ever so easy. 

It is more like Many Predators - Many Preys - Many Parasites.  Symbiosis. Competition among different parasites - how to maximize exploitation without killing the organism parasited. How to use a competing predator to one's benefit. Mixed situations: One is a predator and a prey at the same time but to different kind of critters.  How a steady state equilibrium evolves. 

In my opinion, however, we humans have already won nature's battle and rule the ecology to our benefit. We easily see through the animal world's tricks and catch them as we want. But the market is wholly made up by humans, who presumably have all been exposed for generations to nature's tricks and have become resistant to them. Situations like those that nature presents to us are no longer relevant, and we have moved to a higher level. It is a different game here.

Since we are part of the game, it is very difficult to see what is going on and much more how to manage it.  It is said that even the big winners know how they did it and why they succeeded. It seems to me that those winning have more useful memory, are able to calculate more precisely, see the present and the future more clearly, can formulate better plans, and execute more rapidly and precisely. In the market, nature's tricks don't work any more. This is a play of pure and cold intelligence.

Scott Brooks comments:

I've thought about this predator/prey relationship for many, many hours as I was sitting in a deer stand and I have several thoughts on this issue. I'll share some in this post. 

One of the biggest things to recognize in a predator/prey relationship is the opportunity that exists. One of the biggest things that we need to look at is the difference between instinct and reason. Whether prey or predator, if you are instinctual, you are acting out of some deep seeded genetic conditioning that causes you to run when faced with adversity. 

Think about it. If there are seven lions chasing a herd of 200 gazelles and the gazelles had the ability to reason, they would say, "Lets stick together and as a group go over there and trample those seven lions to death." The 200 gazelles would win that battle, and probably over time could condition the instinctual predator lions to leave them alone. The cost of messing with those gazelles is just too high. 

Think of an instinctual predator like a bear. Almost any bear could take a human if they wanted too, especially the bigger varieties like Grizzlies. Humans are simply not equipped to deal with them physically. But for the most part, we've conditioned bears to stay away from and fear us. That's only because we have the capacity to think and reason at a level that the Grizzly doesn't. We've figured out a long time ago that taking some animal gut and stringing it on one bent stick, and then taking another straight stick and putting a sharp tip on it, gave us the advantage. Then along comes names like Remington, Browning, Winchester, Anshultz, Benalli, etc. and the odds are stacked in our favor. 

When I played poker back in the 80's, I looked for certain types of players to be at a table before I would play. They were the prey. They weren't thinkers. They were gamblers. They let the cards fall as they may and "hoped" that things would go their way. But they had no real system or methodology to identify when to hold'em and when to fold'em. Most of them could not name three cards that had been played and subsequently folded (I'm talking seven card stud). So they had no idea what cards were still available to be played or not. I can't even count all the times when I could tell what hand someone was trying to build or bluff me into thinking they had and yet had no idea that the key card was already burned in the deck because someone had folded earlier. I guess I was a counter of sorts even back then. I'm not sure that qualifies me as a counter yet, maybe it just makes me someone who paid attention and kept track of things. 

These "gamblers" were hopeless gazelles at the table. I'm not saying that to be braggadocious. They simply didn't know what they were doing … they were nearly instinctual prey. They "needed" to win. They were always one card away from catching a break. They relied on luck. The reality for these guys was that the only way they could truly win was to quit and stop playing. Otherwise, ruin awaited them all. 

Those are the guys that I played against. I did not play against other good players. If there was more than one other good player at the table, I would find another game. I had nothing to prove by beating another good player. I was there for one reason and one reason only: to win money

For the same reason that lions don't usually attack other lions to eat, I was not interested in paying the price associated with trying to win money from other good players. The cost and risk/reward was just too high. 

To apply this to the markets, it is important to figure out where the instinctual investors are playing and those that don't have a thinking system, and use that to one's advantage. 

What are the masses going to do when "X" event happens? What is their likely "non-thinking" irrational emotion based response ("quick, run, the lions are coming"). 

Unfortunately, as I've said before, the masses left the markets after 2000, 2001, and 2002. They were burned so badly, and fear chased them away from what was very likely the greatest buying opportunity of their lives. It was like gazelles drinking from a stream and some of them getting snatched by an alligator. It seems to me that after a few have been snatched, that's the time to go get their drink … the alligators have enough food to last awhile now … and if nothing else, there is a few less alligators now patrolling the shores for food. The odds of success have gone up for the gazelle … but that's when they leave in fear. 

So I will be that thinking predator. I will only fight battles that I know I can win. My goal is simple. To make money! That's it. I've got no ego in this and no axe to grind. I'm not going to challenge Prof. McDonnell in the world of options, or Prof. Haave in the world of commodities, or George Zachar in the arena of bonds or Vic in the world of index futures. They are simply more skilled and knowledgeable than I am in those arenas. I could be a predator in those worlds, but I would be like the Grizzly bear, and they would be the thinking human up on the ridge 200 yards away pointing a Win, and a 300 Mag at my vitals. That's a battle I can't win. 

But there are things that I'm good, and there are arenas I can battle in. Since I only want to make money, I will only play in the arenas with the best risk/reward ratio for my success, and I will stick to those arenas (but I'll still learn the other arenas … and who knows, I may show up there one day and dip my toe in … but only when I think I'm ready … and then only with a small amount of money to make sure that I'm really ready). 

So, Phil, Gordon, George and Vic, be careful, I may show up in your arena one day … and I'm a good stalker who knows all about how to properly deceive with camouflage …

Tim Humbert comments:

Over Christmas I heard a wonderful recipe for pike:

-preparation: gut and de-scale, rub rock salt and pepper onto flesh, squeeze some lemon juice, insert some herbs into fish, wrap in aluminum foil and cook for 30 minutes

-consumption: throw pike in the bin and eat the foil

Rick Foust adds:

The largest predators (e.g. lions) are much smaller than the largest grazers (e.g. elephants). The largest grazers have much longer life spans than the largest predators despite having inferior camouflage. Certain large houses come to mind.

Small grazing animals (e.g. rabbits) do not survive long despite having excellent camouflage. Their numbers are maintained by fertility (replenishment). New, poorly bankrolled traders come to mind.

Bruno offers:

Professor Sorin Solomon, of the Racah Institute of Physics, has produced some very interesting market models based on Lotka-Volterra. Here is his homepage.

He showed that a generalized Lotka-Volterra model for the market yields a truncated levy distribution for index returns!

See for instance his 1998 paper: "Stochastic Lotka-Volterra systems of competing auto-catalytic agents lead generically to truncated pareto power, wealth distribution, truncated levy distribution of market returns, clustered volatility, booms and crashes."

There are simpler explanations for TLFs, such as a random-walk with time increments that are variable rather than fixed, just like with real-world transactions … but I thought this was topical.

There could be one way to check the above, and that is the impact of random time between transactions. On Euronext, we've got a mechanism for trading very small stocks. It is called "fixing." One could compare behavior of such stocks to behavior of other stocks that trade continuously. One could also check the behavior of stocks that have moved from fixing to continuous trading or the behavior of the whole French market as it moved from all stocks fixing to most stocks continuous in the mid-eighties. There's also a possible comparison between London Gold fixing and NY COMEX.

Todd Tracy comments:

Market Set Ups

While reading Victor and Laurel's article on Predator-Prey Relations, my mind exploded with visuals: foxes hiding in the bushes waiting to pounce, predictive and instinctual reactions to events, finding myself trapped in currency positions, panic driven searches for exit strategies. I realized that I am the prey. I am the new blood that greases the gears. I am the greedy trader who walks into the trap set by smarter, quicker and more thoroughly financed predators. As with much of the information gleaned from Daily Speculations, I found corollaries not just in the markets but also to life.

But wait, I've been here before. Where have I seen these deceptive techniques in use? Spy fiction. Yes, I have read all the Greene's, the Amblers', the LeCarre's, the Clancy's, the Forsyth's, the Flemming's, the Weisman's, the MacLean's, the Harris', the Buckley Jr.'s and a lot of the Ludlum's. The spy, leaving a trail, using cut outs, drops, proprietary tools and the most diabolically elaborate set ups imaginable. Institutionalized deception, deception as a way of life, and tradecraft so efficient as to make the prey oblivious to the fact that they have even been caught.

War is serious business whether or not it be cold, which brings me to the non-fiction. The Secret History of the KGB, the History of the Mossad, the development of the Office of Strategic Services, The Wall Jumper, the techniques of SMERSH, Stalinism, Churchill's autobiographical books and one of the greatest historical accounts on the subject, A Man Called Intrepid by William Stevenson. Control will leave no stone unturned to reveal facts. Control will sacrifice lives to perpetrate false information.

Why should the markets be any different? It's scary to think that once I feel like I'm playing the charts like a marionette, it is I whose strings are being pulled. I am a novice speculator, but my eyes are widening. If only I had Victor's booklist before I read all those novels. All is not lost however because I am learning to tie strings from my life experience to the experience of the markets.



Continuing my study of truncated Levy flights, I have found some paper coming up with the best explanation so far of why variance is necessarily finite. Physically, variance cannot be infinite because there are only a finite number of observations. That's so simple and so grounded in common sense that I am wondering why no one came up with it sooner.

I would like to make another remark. Since, as the Chair and others observed, the market can be quite jumpy in the short term, but converges to normal in the long term, say yearly returns, the central question is: how long before the market converges to normal?

This is somewhat opposite to the Mandelbrotians' worry: how long before a 10-sigma event?

I would be grateful if anyone could point me to good papers about measuring convergence speed.



When magic of the markets is felt every moment, why is there no organized market for magic?

For New Years Eve, one chose to be at the Mela restaurant, (Mela a word from the Indian vernacular means the village fair). Among a host of activities from a village fair, the restaurant specializes in bringing a personal magic show to your table for a small fee, and the question arose right there at the dinner table as to why is there no organized market, not even a national or trans-national company that specializes in retail or wholesale magic?

There are several national and international companies with listed stock in the arena of restaurants, hotels, movie making, movie screening, bowling alleys, vacation organizing, vacation sharing, culture companies, etc., but there is not a single listed stock or organized magic company. Why?

Here are some possible explanations:

Many more ideas come to mind, but then the thoughts have stayed lingering around this one point about being personal. All other human endeavours in the arena of entertainment and services that have been able to overcome the personal factor and lend themselves to being productized, standardized, predictable, mass-emulated, mass-transported, mass-communicated etc. have come to evolve into giga-corporations. Individualistic personal pursuits of acting, dramatizing and magic have failed to turn the magic of the markets to their advantage.

So, is the magic really in the crowds rather than in the magic itself. What important lessons could one derive from the failure of magic to draw the magic of the markets to its advantages?

Easan Katir adds:

This weekend I had a front-row center seat amidst a sold-out house at the Geffen Theater in L.A., to view up close a talented sleight-of-hand master, Ricky Jay and his 52 Assistants, directed by David Mamet. Consequently, I have been contemplating similar corollaries between the conjuror’s art and the trader’s art. Certainly there is plenty of misdirection and deception in both arenas. There is also plenty of explanation to convince one that the impossible is normal. Mr Jay produced winning poker hands, and explained that a card cheat must not only give himself a good hand, but give the suckers good enough hands to inspire them to stay in the game.

Steve Ellison offers:

An important parallel between magic and the markets is the role of patter in distracting customers’ attention from the sleight of hand. A thing to which a magician is drawing the audience’s attention is almost certainly not the main event. The weekly enumeration of reasons to be bearish is an example of market patter.

Laurence Glazier comments:

Magic is also a matter of political or sociological point of view. Is our very existence magic, or the random walk of chemicals? If I construct a chord progression which moves the e-motions, is it science or something more? The magician who bends forks and keys - the process often continuing after after he has ceased touching them - wil never convince the “component parts” of science, and likewise neither would those who have vibhuti.

I am not sure that music works well in the market - where it is there the market - a la Adorno - may affect it adversely, and similar considerations may apply to real magic. If life is to be magical, it must have magical qualities. It is easier for children to see them, though, so let’s stay young.

Andres Vincent counters:

Forgive me for disagreeing, but DNA strands, crystal organization, life itself, a snow flake, clouds, animal life, glass, light, rainbows, electromagnetism, classical mechanics, relativity, etc., etc.. The whole universe is magical, so to see magic there is no need to hallucinate. Just read the book of nature. But to appreciate this beauty its complexity must be (at least a bit) understood, i.e. we have to observe, to work, to learn — in other words, try to become adults.

If adults stay young, and that’s unfortunately the case of the majority, the only magic provided today is overconsumption and/or religion, i.e. deceptions.

Bruno Ombreux mentions:

I would add geology and botany to your list. I got undergraduate classes in both of those, an it is incredible what learning about these subjects does for you.

After studying geology, for instance, one sees the world in a different way. Walking in the countryside — you don’t see the normal countryside any more. You can see how landscapes came to be, you can see millions of years of evolution, movement, shocks, erosion, chemical reactions. And you don’t see rocks anymore, you see names.

You can call a stone by its true name, that is magic. It actually kills all the poetry of a walk in the countryside though, so I am glad I forgot my geology classes.



An interesting undergraduate paper, is looking at the New-York phone directory. And… No, it is not about the Benford law. It is about subsequences.

The same approach can be applied to markets. For this, we will consider random mappings between two ordered finite sets: time [1,2,3…..t] and ranked asset returns [r1, r2, r3….rt].

The longest subsequence in asset returns should converge to a Tracy-Widom distribution. It is a beautiful distribution that seems rather ubiquitous. Unfortunately, it is relatively new also, with few implementations. There is some code in S-Plus, but it is using a S-Plus function, ivp.ab, with no obvious equivalent in R. If someone knows of an R module solving ODEs at point B, knowing the solution at point A, please let me know.

So we won’t be checking the asymptotic behavior of any asset. Instead, we will content ourselves with the study of the longest subsequence in subsets of length 15, for which exact frequencies are provided in the New-York directory article. It is easy to extend the study to other lengths, by generating random sequences in R.

We’re using the daily DJIA since 1896 for illustrative purposes, but as explained often on DailySpec, this doesn’t make sense. It would be better to look at more recent returns, preferably intraday, because many observations are required.

Intuitively, longest increasing subsequence behavior could be useful to know. For instance, if it is longer than dictated by randomness, it means that big drops are followed by smaller drops and big rises are followed by bigger rises, than would occur in a random walk. This evokes a “U” shape where it would make sense to buy sharp drops.

Please note that consecutive drops or rises don’t necessarily occur the following day. Subsequences are not defined by consecutive returns. They are an entirely different concept from the one of runs. Instead, they are defined by returns distributed all over the interval. We are looking at some form of market local curvature. But it is a distributed curvature, not a continuous one. This kind of stuff is not captured by usual tests and certainly not by the eyeball.

A chart of realized frequencies against exact frequencies seems to indicate that actual longest subsequences are longer than theoretical ones. At the 5% level, a Kolmogorov-Smirnov goodness-of-fit test rejects the null hypothesis in favor of non-randomness.

Two-sample Kolmogorov-Smirnov test:

data: LN0 and Sexact
D^+ = 0.4667, p-value = 0.03813
alternative hypothesis: greater

Buying sharp drops is a good thing … But that’s for 15-day subsets only and needs more work.

The R code is appended. Peer-reviews are needed due to my propensity to counting mistakes.

# #   Longest increasing subsequences #   # Patience sorting    patience <- function(x,s=NULL)    {    s[1] <- x[1]        for (i in 2:length(x))        {            for (j in 1:length(s))            {if (x[i] <= s[j]) {s[j] <- x[i]; break}}        if (x[i] > s[length(s)]) {s[j+1] <- x[i]}        }    return(s);    }  # data loading    S15exact <- scan("S15exact.txt")    testdata <- read.table("testdata.txt",sep=",",header=T)    testdata <- diff(log(testdata$close))     # to change if need be, eg replacing S15exact by random numbers frequencies    data <- testdata    Sexact <- S15exact  # data ranking, data split, Q samples of length N    N <- 15    Q <- trunc(length(data)/N)     data <- data[1:(Q*N)]    data <- rank(data)    datasplit <- 1:Q    datasplit <- rep(datasplit, each=N)    samples <- split(data,datasplit)  # increasing subsequences     LN <- numeric()    for (i in 1:Q)        {        samples[[i]] <- patience(samples[[i]])        LN[[i]] <- length(samples[[i]])        }    LN <- tapply(LN, factor(LN),sum)  # adding increasing subsequences with zero observed frequencies # there must be a more elegant way     LN0 <- tapply(rep(0,N), factor(1:N),sum)    for (i in 1:dim(LN0))        {        for (j in 1:dim(LN))        {        if (names(LN0)[i] == names(LN)[j]) {LN0[[i]] = LN[[j]]}        }        }    LN0 <- array(LN0)/sum(LN0)  # graphs     plot(LN0, type ="l", col="brown")    lines(Sexact, col="red")  # Kolmogorov-Smirnov  kstest <- ks.test(LN0,Sexact,alternative = c("greater")) kstest

Bruno later adds:

I did a bit more research on Tracy-Widom this week-end. Among a great many other things, it is also measuring the probability of explosion for a random-walk with drift. This makes a lot of sense intuitively. Explosion, the right hand side bar in a “U” shape, and longest increasing subsequences are more or less the same thing.

The fact that it measures probability of explosion for a random walk ‘with drift’ is reassuring. I was a bit worried by the possibility to be capturing only positive drift, and not adding any new information, since subsequence computation is drift-independent.



Most professionals would frown upon such a piece of software as WealthLab, but it is very cheap with good backtest functionality. This is my own opinion, but I think that backtests are useful, even if no substitute for rigorous statistical analysis.

What’s nice is that they have a utility for using R inside WealthLab, and someone on their R forum posted code to communicate the other way, from WealthLab to R. All these efforts are yielding a decent low cost exploratory analysis and backtest solution for the small guy.



I recently stumbled upon Ehrenfest Urns. The results from their simulations look like tick charts with no drift.

And I am wondering if things like advances-declines, ticky, basically anything in the market that is based on a sum of binary states, could not be modeled as an Ehrenfest process.

Going further, an Orstein-Uhlenbeck process seems better suited to a market drifting up against a wall of fear (fear is the viscosity), than a standard random walk. Why don’t people use these?

I know this sounds like a lot of name-dropping, but this is merely to point out that the science of finance seems very backward. There is still a lot of progress to be made.



I use an intuitive and unscientific rule of thumb derived from a law of cybernetics.

To forecast/control a system of degree N, one needs a system of at least degree N+1. Positing an arbitrary hierarchy of markets systems: years > months > weeks > days, means that to forecast one day ahead one needs to look at weekly anomalies.

That is sets of five trading days. Since in a parametric setting one needs at least 20 to 30 data to converge to a normal law, the minimum length of data is 5*20 to 5*30, that is 100 to 150 days, to operate at the daily frequency.

Paolo Pezzuti replies:

My opinion on this issue is that the length of data should not be defined as a fixed number (e.g. 150 or 200). Data selection to run the tests should reflect criteria of behavior observed in relationship with the scope of your test objectives. In the everchanging cycle process you might recognize that a cycle has changed because of increased volatility or directionality or what else you have identified as your guiding parameters of the market “personality”. In this case your length could vary a lot. For example, if you assume that a new paradigm began in 2003 with a low volatility environment, etc. and this is relevant for the type of assumption that you want to demonstrate with your testing activity than you could use a 600-days data test set.



1. A book recommendation from John Lamberg: Five Hundred and Seven Mechanical Movements by Henry T. Brown

2. Bruno Ombreux recommends the academic paper Dangers of Data-Driven Inference, “a good paper touching on a subject oft mentioned by the Chairman”



There are many angles to the markets.

There are Gann angles, which of course make no sense, because angles on a chart are depending on axis units.

There is also the Cauchy distribution, whose fat tails are scaring away those who can remember 1987 but have never traded smallcaps or electricity. This distribution can be generated from angles, made by someone shooting randomly at a distant target. Where "randomly" means uniform. Hence fat tails are the property of some distribution of angles.

More interestingly, there seem to be a lot of statistical tests developed for circular data; that is angles. I found out about them in 100 statistical tests by Gopal K. Kanji. Just for fun, I gave a try to the V-test, or Modified Rayleigh. It is a test for randomness, checking whether observed angles tend to cluster around a given angle.

The data is JPY/USD monthly returns since 1965. One problem surfaced though — how to transform returns into angles?

I chose to project them on a vertical axis, in some reminiscence of the Cauchy target experiment. As a result, time is factored out of the study. It could turn it into what I think they call an "axial study", in which all angles need to be doubled in the computations. This could be a mistake, but it does not affect the results. The conclusion is the same whether the angles are doubled or not.

And the conclusion is that we reject the null hypothesis that angles are random around the zero-line, at the 0.0001 significance level. Rayleigh's V is 5.255. There is some element of non-randomness in monthly JPY/USD, which still need to be identified.

Here is the code for the test, except that V's significance had to be checked in a table, unavailable in any R package.

YEN <- read.table("MonthlyYenUSD.txt", header=TRUE)
Close <- 100*diff(log(YEN$Last))
Angle <- 2*atan(Close) #Axial data

xbar <- sum(cos(Angle))/length(Angle)
ybar <- sum(sin(Angle))/length(Angle)
r <- sqrt(xbar*xbar+ybar*ybar)
phi <- atan(ybar/xbar) # xbar>0, else add Pi radians
theta0 <-0 # theoretical angle direction i.e. null hypothesis
nu <- r*cos(phi-theta0)
V <- nu*sqrt(2*length(Angle))

Gibbons Burke adds:

I am no Gann fan, but Gann angles escape the aspect ratio problem you mention because Gann plotted his charts on paper of a constant scale, so that one increment of time was always kept constant in physical distance relative to the units of price on the y-axis. Typically it was 1 point in price = 1 unit of time. In this way, a 45 degree line always related to a rate of ascent or descent of, say, one point per day. So there was some internal consistency. Or, as Shakespeare put it "Though this be madness, yet there is method in't."

Early technical analysis software packages like CompuTrac let you draw angled lines which were completely arbitrary because of the vertical range in the time period plotted in the chart, but they were there because the TAG group threw just about anything into the program that users asked for. But they also had methods of drawing Gann angle lines with the consistent aspect ratio ability.

Bruno replies:

To answer my own post, I tried as much as I could to find randomness with circular tests, but could not. The reason is certainly that returns data is not circular. The way to make it circular is to introduce time. Divide the circle in 12 for monthly returns.

Then returns have to be in a third dimension. Fortunately, we have got spherical statistics!

Dr. Phillip J. McDonnell adds:

With all due respect, Gann realized his error only after he had been publishing for some time. It was a retrospective fix. But it is an inadequate fix.

When a stock splits 2:1 any normal chart is now no longer a 1 point to 1 time unit ratio. When Microsoft paid its 10% dividend is the new ratio now .90 to 1.00 or should it be 1.10 to 1. Gann is moot on the question.

When a stock pays dividends should the price be adjusted? Should the dividends just be ignored with the attendant error in rate of return?

What about weekly charts - is the time scale 5 units or 7 or just 1 (a week)?

What about monthly? Is it 1 unit, 21 units or 30 or the actual number of trading or calendar days?

With many angles, many time scales and dubious rules it should be quite easy to find many examples that come 'close' to turning points in the markets. In fact it is probably quite difficult to find any failures. This is especially true if one is allowed to define 'close' as whatever one needs to make the current data fit.

Even though the arcane mysticism of Gann is suspect, Bruno's ideas on angles may have some merit and should not be lumped into the same basket. The Cauchy distribution induced by the angle model can be problematic. During the 90's several advances were made by Zar and others in the statistics of angles and tests thereon. That area is relatively new but very workable.



The beauty of Judo is that it is a very subtle sport yet a very lethal one. In fact Judo means "gentle way" in Japanese. One soon finds out that there is not much gentle about it. It is still a martial art.

Judo is an isolated sport, often a lonely one. It is just you and your opponent and the battleground is the center of a mat in full view of the audience. There are no teammates, nobody blocking for you, and nobody hitting behind you. And no excuses. Plus, you are always in direct contact with your opponent. There is literally no let up and no time outs. It is combat at the highest level. Balance, strength and explosiveness are critical skills. And you have to be constantly aware of your opponent because one misstep one false move can instantly end a match.

Judo emphasizes fighting (randori) as its main form of training. Half the combat time is spent fighting on the ground, called ne-waza and the other half standing up, called tachi-waza. Actual fighting, albeit within safety rules, is considered to be much more effective than only practicing techniques, since fighting full-strength develops the muscles and cardio-vascular system on the physical side of things, and it develops strategy and reaction time on the mental side of things.

Judo's balance between both the standing and ground phases of combat gives judoka the ability to take down opponents who are standing up and then pin and submit them on the ground. A Judoka can also force an opponent to submit through a chokehold or a joint manipulation such as an arm bar. Thus multidimensional skills are essential to develop to becoming an accomplished judoka.

The skills in being a successful Judo champion and a successful speculator are very similar. Both require a great deal of discipline, awareness, and specific mastery of skills and techniques. Plus strategy and attention to detail are prime requisites for success. In addition accountability is paramount as there is none to blame for failure than oneself. In the end it is the combat, the battle, that determines the ultimate victor. There is no statistical bias no French judge who grades the combatant lower through political favoritism. The contest is decided on the mat. And the results are final. The vanquished accepts this and validates it through a final acknowledgment of bowing to an opponent who has bested him in the arena.

Bruno Ombreux offers some similarities between Judo and speculation:

Hard work. All the skills you mentioned can only be developed through hard work and practice. Provided weights and technical levels are not too far apart, the judoka who trains 4 times a week will beat hands-down the one who trains only twice.

Successful speculation is hard work.

Bruises. Pain in a full-contact martial art as judo, goes beyond muscle ache normally associated with sport practice. Shiai - the competitive form of randori - is very intense. First degree burns on the neck, resulting from judogi friction, are common. (judoki is the outfit worn by judoka). Passing out from strangulation happens too. Broken members also. My personal souvenirs from judo are a broken nose and a permanently paralyzed big toe.

Speculation is a source of bruises in the form of losses. One has to pay his dues.

Specialization. Every judoka has a "special". This is a throw that he excels at, often because his morphology is well suited to this particular move. For instance, morote-seoi-nage, a shoulder throw, is said to be favored by smaller players, because it is easier for them to get inside and under the opponent gravity center. The judoka trains his "special" more than any other move. He is going to use it a lot in competition. The idea is to excel at one thing rather than be average at many.

This analogy with speculation needs not be explained after Chair's post on Specialization and the Division of Labor.

Focus on the wrong the methods. For years, proponents of strikes-based martial arts like karate or kick boxing, proclaimed their superiority over wrestling styles, like judo, "a mere sport for schoolchildren". This changed with the introduction of "Mixed Martial Arts" fights. Evidence surfaced that in real-life, unlike in kung-fu movies, one-on-one fights with no limiting rules most often end on the ground after the exchange of a few blows, a grapple and a throw. The most efficient fighting style was Brazilian ju-jitsu, because of its emphasis on ground combat. But judoka and wrestlers also performed very satisfactorily.

In speculation, people focus on the wrong methods, as explained again, by the Chair in the first-half of Practical Speculation. Then reality kicks in.

Impossible mastery. It is impossible to master judo. It is an endless study. That is what the "do" in ju-do stands for. It is a "way". An endless voyage whose final destination is never reached.

This one is easy. Speculation too is a "do". An endless study in which perfection is unattainable, and the best one can hope for is improvement.

Principles and tradition. The other side of judo beside combat, kata, is a codified series of moves, almost like classical ballets. The first kata focus on basic throws, on mechanics. The last kata, which are learnt by very few people, focus on entirely different things.

Itsutsu no-kata, taught to 6th dan black belts, features only 5 throws which are supposed to contain the essence of judo. Those throws do not even have a name, yet they embody the principles of judo. They are simple, fluid, beautiful.

Koshiki no kata, taught to 7th dan, is an ancient kata from the Kito-ruy school of ju-jitsu, which was attended by Jigoro Kano. It is a throwback to medieval Japan's fields of battle, a series of throws designed for combat in samurai armor. Ideally, this kata should be performed in such armors.

Who knows what Jigoro Kano meant by the inclusion of these two final kata in his curriculum? It is perhaps that, in the end, one should focus on first principles and a study of tradition.

That is not too different from the site, focusing on principles is learning how to fish, reading old books is integrating ancient wisdom.

Education. I could go on and on drawing other parallels between the site and martial art education. Teaching by showing. Use of arcane yet limpid language, "when the yellow bird sings, the sun sets on the jade mountain, little Grasshopper". err…. Too many Hong-Kong movies. Haiku and people taking their shoes off before entering the dojo…

But that's not about judo anymore and I must trade the last half hour.

« go back


Resources & Links