lets Try

November 2, 2020 | Leave a Comment

Victor Niederhoffer  writes: 

Tony Bobulinski held presser claiming Joe Biden knew about Hunter's business deals

Bill Egan writes: 

Bill Rafter has a very nice model.

Yes, rolling windows mean you cannot use standard inference.

So what?

Use the levels and turning points and curve shapes. Apply visual pattern recognition.

Look back over 20-30 years and see how it performs. For example, high volatility vs low volatility years can cause differences.

You are missing out on $$$ by insisting on using only methods where inference works.



lets try

November 2, 2020 | Leave a Comment

Victor Niederhoffer  writes: 

Tony Bobulinski held presser claiming Joe Biden knew about Hunter's business deals

Bill Egan writes: 

Bill Rafter has a very nice model.

Yes, rolling windows mean you cannot use standard inference.

So what?

Use the levels and turning points and curve shapes. Apply visual pattern recognition.

Look back over 20-30 years and see how it performs. For example, high volatility vs low volatility years can cause differences.

You are missing out on $$$ by insisting on using only methods where inference works.



 My wife wrote:

Does this count as an Economics lesson? I pay 1 cent per dandelion picked from our yard (organic control). Two children secretly agreed to leave a breeding stock. (A third child ratted them out, but I did nothing as there were more children not party to the agreement than party.) One of the children who made the agreement secretly picked all the dandelions that had been set aside as breeding stock and turned them for cash. The other wigged out.

Jeff Watson writes: 

Very smart kids who deserve a reward. But I advise you to gently punish the one who queered the deal. Nobody ever liked or respected a tattletale.



David MacKay says:

"Principal Component Analysis" is a dimensionally invalid method that gives people a delusion that they are doing something useful with their data. If you change the units that one of the variables is measured in, it will change all the "principal components"! It's for that reason that I made no mention of PCA in my book. I am not a slavish conformist, regurgitating whatever other people think should be taught. I think before I teach.

Bill Egan responds:

Well, Prof. MacKay is wrong. In fact, I have made predictive models that have worked for years in real-world corporate environments that were based on PCA. Worse yet for the good Prof., all work done in the predictive modeling of optical spectra for the last 40 years or so has involved PCA or a related method.

PCA transforms your data into a set of uncorrelated, unit length vectors. The first of these new vectors contains the most variation in the original data. The next vector explains as much of the remaining variation in the original data as possible, and so on. Each new vector is a linear combination of the original data into the new vector. The method is reproducible and quite numerically stable if you use singular value decomposition as the algorithm.

PCA is a very useful way to reduce the dimensionality of a data set, say one that has many variables, to a smaller set of uncorrelated variables you can work with. To be fair, the new variables do not necessarily have physical meaning, but they often do, and it always pays to look at the weights applied to the original variables (called loadings in some of the literature).

Well, Prof. MacKay is wrong. In fact, I have made predictive models that have worked for years in real-world corporate environments that were based on PCA. Worse yet for the good Prof., all work done in the predictive modeling of optical spectra for the last 40 years or so has involved PCA or a related method.

PCA transforms your data into a set of uncorrelated, unit length vectors. The first of these new vectors contains the most variation in the original data. The next vector explains as much of the remaining variation in the original data as possible, and so on. Each new vector is a linear combination of the original data into the new vector. The method is reproducible and quite numerically stable if you use singular value decomposition as the algorithm.

PCA is a very useful way to reduce the dimensionality of a data set, say one that has many variables, to a smaller set of uncorrelated variables you can work with. To be fair, the new variables do not necessarily have physical meaning, but they often do, and it always pays to look at the weights applied to the original variables (called loadings in some of the literature).



 Cold reading has much in common with market charlatans:

"There seem to be three common factors in these kinds of readings. One factor involves fishing for details. The psychic says something at once vague and suggestive, e.g., "I'm getting a strong feeling about January here." If the subject responds, positively or negatively, the psychic's next move is to play off the response. E.g., if the subject says, "I was born in January" or my mother died in January" then the psychic says something like "Yes, I can see that," anything to reinforce the idea that the psychic was more precise that he or she really was. If the subject responds negatively, e.g., "I can't think of anything particularly special about January," the psychic might reply, "Yes, I see that you've suppressed a memory about it. You don't want to be reminded of it. Something painful in January. Yes, I feel it. It's in the lower back [fishing]…oh, now it's in the heart [fishing]…umm, there seems to be a sharp pain in the head [fishing]…or the neck [fishing]." If the subject gives no response, the psychic can leave the area, having firmly implanted in everybody's mind that the psychic really did 'see' something but the subject's suppression of the event hinders both the psychic and the subject from realizing the specifics of it. If the subject gives a positive response to any of the fishing expeditions, the psychic follows up with more of "I see that very clearly, now. Yes, the feeling in the heart is getting stronger."

Jeff Watson writes:

Here's a great how-to" book on cold reading.

Bill Egan writes:

A complementary resource I recommend is "The Definitive Book of Body Language" by Allan and Barbara Pease. Always watch peoples' body language and compare it to their words, and watch how both change over time. For example, when the fraud thinks he has you, there is often a split second where he will shift his body position and display a chilling facial expression like a fox looking at a chicken. That half-a-second is real important to you.

Jim Sogi writes:

Trial lawyers look for cues in the jury's race, clothes, hair styles, books or magazines, shoes, apparent class, education, prior experiences who they speak with, their background information on their questionnaires to get a read on how they might decide a case. Trial consultants use broader data on how similar groups might react to similar situation. During Voir Dire, a short question and answer period, the lawyer can ask the prospective juror some questions that might shed light on the juror's prejudices that would justify being removed from the panel or dispose the juror against the lawyer's client. Again, all forms of cold reading.

A fun game I like to play while people watching in restaurants, or on the street is to look at people and try to figure out without anything more than watching from a distance, where they are from, what they do, what the relationships are between members of the group, what they might be like. Family groups on vacation are a pretty easy read as well as their internal family dynamic. Old couples are straight forward. Groups of young people tend to send strong signals. Groups of business men, groups of tourists, newlyweds all have characteristic mannerisms. The next level to try discern their relationship, what they are like and get an idea about them from only external signals.



 I've had a terrible head cold for the past week, and it's made me think about what a weak approach the world takes to the common cold. Doing a little wiki search you'll find that people are out of commission for something like 2 weeks per year. Tens of millions get poured into some cancer drugs that are viewed as successes because they increase life expectancies from one month to three months. But when you go to the pharmacy for your cold, 80% of what you see is junk, and what's not junk is just barely. (I'm making up a lot of the actual numbers in this post, but you get the idea.)

Here's a rundown on some of the pharmacy items:

Long-last 12-hour nasal decongestants, inhaled (like Afrin): This stuff works for awhile and provides actual relief, but 1) it doesn't work for 12 hours–more like three, and 2) they tell you you can only use it twice per day and up to a maximum of three days. OK, well I'll just have to make sure my cold only lasts three days! If you consider how people treat the warnings about drugs like crystal meth, I would imagine that a lot of people use Afrin for longer than three days and more than twice per day and don't get badly hurt. The CVS pharmacist kind of hinted that that was the case. But on the other hand, I don't want to get a permanent stopped-up nose and Afrin addiction.

Short-last nasal decongestants, inhaled (like Neo-Synephrine): These are supposed to work for 4 hours, but of course they don't. There are scattered hints that they don't pose much dependency risk, but the label says otherwise–use no more than once every 4 hours and not for more than 3 days.

Oral pill nasal decongestants — phenylepedrine — These don't do diddly. I discovered this on my own, but later the pharmacist told me that everybody pretty much knew it.

Oral pill nasal decongestant — pseudo-ephedrine — For these, you have to go to the pharmacist and show your drivers license so that they can check that you're not making crystal meth. Usually I don't want to go to that kind of trouble, but word on the street is that phenylephrine, which is the new pseudo-pseudo-ephedrine, like the one that Mother gave Alice, doesn't do anything at all–you have to get the REAL pseudo-ephedrine. Anyway, I got some 12-hour slow-release capsules of pseudo-ephedrine. They seemed to have some slightly helpful effect, but not nearly enough to give comfort.

"Nite-time" stuff — There are literally dozens of varieties of this at CVS including multiple store-brand versions of the same thing. They're all equal to Tylenol+Phenylephrine (useless) + Dextromethorphan HBr + Chlorpheniramine Maleate. The Dextro… is described as a "Cough Suppressant" and "Chlor…" as an antihistamine. I know what Tylenol and Phenylephrine are. I don't really understand the last two drugs, but I think their real purpose is to put you to SLEEP. That's not the worst thing in the world, but they only last for about 4 hours or so. So then I wake up at 3am and want some more, but I worry about taking more because I've been reading that it's easy to overdose on Tylenol and mess up your liver.

Various Zinc stuff, acidophilus, Vitamin C — I already pretty much know that Vitamin C doesn't work, since I already take a lot of it, having read Linus Pauling's book years ago, but I still get plenty of colds, and they last a good, long time. It's possible that some of the other stuff could work. The typical story is that one study showed good results in 1996, but it had some kind of flaw in its setup. Of the remaining studies about half showed something good and half got nulls. Well gosh, 1996 was 16 years ago, and we're talking about alleviating the common cold, which keeps the entire world out of commission for two weeks per year. Why in the heck doesn't somebody do the definitive study? Meanwhile, I have the option of paying the toll to what I suspect are charlatans.

I read about a real company called Biota in Australia that supposedly has something that pretty much cures the common cold, though it's not on the market yet, and it will be very expensive and perhaps only available to asthmatics [I will apply to become one]. However, in the best of circumstances I can't imagine the FDA approving something like that in less than two decades because it will be argued that since nobody dies from the common cold, it's fine for us to just suffer.

I'd be very interested to hear helpful tips.

Leo Jia writes:

Prolonged exposure to negative psychological states such as fear, tension, anxiety and etc, which seem to be inherent but unconscious to most traders, can make one's immune system weak. The immune system is key in fighting cold and other abnormalities in the body. Best things to me that help strengthen the immune system and alleviate negative senses are physical exercises combined with meditation, Yoga or Zen practices. For me personally, playing the violin helps a lot also as the dedicated playing puts one into a concentrated mental state that can be close to meditation.

Victor Niederhoffer writes: 

To what extent do we catch most of our colds from the classmates of our kids at school or our coworkers at work? Is one of the great advantages of home schooling aside from the fact that the kids don't have to spend every weekend with a wasteful birthday party, that they are healthier and don't catch colds as much? And similarly for work at home. 

Bill Egan writes: 

We homeschool six children. The kids tend to be less sick than the kids of my colleagues at work. Ours still manage to contract a sufficient number of plagues from other kids in our homeschool network, the YMCA, choir, etc.



 I assigned my 13-year old to read Letters from a Self-Made Merchant to his Son and write me a page on what he learned. Here is his report:

Graham’s letters to Pierrepoint were intended to teach him things, but the letters have been educational to me as well—and very likely to many others. For one thing, I have learned that if someone makes the same mistake multiple times he can’t keep making excuses and that if you’re going to make a fool of yourself, you should at least try to make sure you’re being a different sort of a fool every time. I could use this to avoid problems. Then there’s that some people can only see those above them, and some can only see those below them, but a good man can see both ends at once. It’s easier to climb if you help those above and below you, because they might help you up. This could be useful in many aspects of life. Finally, there’s that you shouldn’t start talking before you know what you’re going to say and that you shouldn’t keep talking beyond when you’ve said what you were going to say. You may have noticed that this letter is shorter than you might have intended, but that is because I applied the third thing I mentioned to this letter.



 What is the significance that when Sokol brought the Lubrizol deal he showed Buffet projections, but "these projections went unexamined" by Buffett. Despite the Talmudic education I have received, I claim that when someone serves as right hand man for 10 years, and then suddenly something like this trading comes to light, it is not a single instance. Much of the posturing must be designed to cast a positive spin on this.

Bill Egan writes: 

Sokol's lawyer's latest includes:

"It is alarming that Mr. Buffett would be advised to so completely flip-flop and resort to transparent scapegoatism. After 11 years of dedicated and hugely successful service to various Berkshire Hathaway subsidiaries, Mr. Sokol would have expected to be treated fairly. That would have been in Berkshire’s interest."

Will Buffett relearn wisdom of Ben Franklin vis-a-vis Vic's comment below? "Three can keep a secret if two are dead."

"That would have been in Berkshire’s interest…"




 Traders who communicate make more money. A great post by Jonathan Lehrer. 

"Herds on the Street: Why messaging traders are like scared fish" .

Ken Drees comments: 

I think of schooling more related to crowd behavior and not specs– but maybe this author thinks small minded traders are like bait fish in that they react and must constantly tweet, chirp, burp, ping, belch, touch fin, and keep swimming as a group so that when the big gaping maw makes a pass one would be in the 80% not affected and the poor sorry bud from the other end of school gets chomped.

This constant talking and interaction saps my energy– I use the chatter a a contrary filter and in some cases just turn it all off.

Contrast Livermore in his silent trading room to the adolescent instant messaging of the trading crowd.



 In a nice article about the failure of Hewlett Packard's directors, written from a liberal perspective as are 99% of the stories from b (which caused me to cancel all my subscriptions there, thereby saving much contemplated expense, but probably disrupting the rhythm), he refers to duos that have been successful: Jobs Wozniak, Filo Yang, Page Brin, Hewlett Packard. I know of a number of 2 person partnerships that are successful, but have always felt that 3 person partnerships are very unstable and unhealthy, as was mine when I started with NCZ. I have always felt that the reason is that it's too easy for any two to form a coalition against the third. Have others here found the 3 person triangles very dysfunctional, and is there an economic reason aside from the all too prevalent attempts to better themselves at the expense of another that lies within the human heart? What are the market implications of such?

Bill Egan writes:

My experience is also that two person partnerships work much better, and adding three or more people leads to a mess.

I believe there are three reasons for this. For two people:

1. You have time to try to understand the other person's viewpoint.

2. Combinatorics are in your favor. With two people, there are only four
possible positions to discuss.

3. No politics because no one can get an ally.

With three or more people:

1. You have less time to try to understand the other peoples' viewpoints, which creates more opportunity for misunderstanding and miscommunication.

2. Combinatorics are not in your favor. With two people, there are four possible positions. With three people, there are eight possible positions on any given issue.

3. Politics can get ugly because person one can get an ally (person two) against person three, etc.

Ken Drees writes: 

Treasure of the Sierra Madre comes to mind.

Trader Craft writes:

In gravitational physics, two body systems are orderly and predictable. Once you get to three bodies, the system becomes chaotic.

Stefan Jovanovich writes: 

The Founders' direct experience with bicameral and unicameral legislatures led them to oppose both Franklin and Hamilton's preference for a single body. What the Founders did not anticipate was that the Federal judiciary would become a co-equal 3rd branch. IMNSHO, the instabilities of our system have their source in that unexpected development. For some of us, James Marshall is anything but a hero.



 This is a funny rag on 50 Cent: "50 Cent's Investment Library"

But once done laughing, you might enjoy the excellent book by Robert Greene with 50 Cent's name on it, The 50th Law.

Here is a quote from the book:

"The greatest fear people have is that of being themselves. They want to be 50 Cent or someone else. They do what everyone else does even if it doesn't fit where and who they are. But you get nowhere that way; your energy is weak and no one pays attention to you. You're running away from the one thing that you own - what makes you different. I lost that fear, and once I felt the power that I had by showing the world I didn't care about being like other people, I could never go back."

- 50 Cent

J.T Holley writes: 

I get the joke, but he actually chose the name as a metaphor– "Change". He took it upon himself to make something of himself other than what he was accustomed to seeing in Queens.

I think the Wall Streeters who bash him are just a little bit on the "sour grapes" side. He moved from Queens to Farmington. Those MBA's probably don't like that and neither do all the other critics who reside elsewhere. Ironically, the Farmington mansion was owned by Mike Tyson whose former bodyguard put 9 shots into .50 cents body. Just a theory, but I think the house was purchased as a personal mark of his own overcoming.

His success as an entrepreneur is something that isn't appreciated and should be looked at in my opinion quantitatively by his checking account and its sustainability thus far. He is an entrepreneur. Took the effort and time to do such. Get's paid for his services. He has his critics and they don't like it. Go figure.

I like when he was working on the movie and soundtrack to "Get Rich or Die Tryin'" he was asked when he found time to sleep. He was either working on the film or the soundtrack and people took notice. His response, to paraphrase, was "Sleep is for people that are broke, I don't sleep. I have a small window to make my dreams reality".

Also when shown or quoted 50 almost never seems like a braggart. He appears humbled and utilizes his time to profit.



Much of what medical researchers conclude in their studies is misleading, exaggerated, or flat-out wrong. So why are doctors– to a striking extent– still drawing upon misinformation in their everyday practice? Dr. John Ioannidis has spent his career challenging his peers by exposing their bad science.

From the article "Lies, Damned Lies, and Medical Science" in the Atlantic.

Craig Mee writes:

Thanks Bill, outstanding read. Everyone should read that including the whole family. It does most reality tv shows and glossy mags out of a job….that's how enjoyable it is. Test and retest, especially the original basic findings seems to be one of the main messages (which dailyspec emphasises often)…and everyone has got their own agenda mixed up in everything all the time.

Replacing quants and traders for reasearchers and physicians in this passage brings some interesting thoughts, and for the passage: "there's simply too much complexity in patient treatment", think individual markets.

"Researchers and physicians often don't understand each other; they speak different languages," he says. Knowing that some of his researchers are spending more than half their time seeing patients makes him feel the team is better positioned to bridge that gap; their experience informs the team's research with firsthand knowledge, and helps the team shape its papers in a way more likely to hit home with physicians. It's not that he envisions doctors making all their decisions based solely on solid evidence—there's simply too much complexity in patient treatment to pin down every situation with a great study. "Doctors need to rely on instinct and judgment to make choices," he says. "But these choices should be as informed as possible by the evidence. And if the evidence isn't good, doctors should know that, too. And so should patients." 

Victor Niederhoffer comments:  

I have always said that aside from the licensing of Drs. , the insistence on double blind studies needed for approval is one of the greatest reducers of life expectancy, and of course, maintainers of anti competitiveness, and of course, improper use of statistics in the real world aside from our own field.



 Watching (and trading) GBPUSD today brings up a few interesting oberservations. As cloudy skies enveloped Tokyo and the Nikkei started to get hosed, S&P futures fell easily to carve through the bid. Gold struggling, Crude off ok, specie longs on the QE train booking. Will it turn the USD position?

GBPUSD…bid early… easing easing, bang, through 158.60, and straight to 1.5790 …stops …trendline break…not sure but Europe had been in for 3 hrs, and had good time to look at day's Asian events before acceleration took hold.

Who knows whether technicals are worthy, or if it's just a momentum play, and a flip of the coin…but the break on the day seemed long overdue.

Bill Egan writes:

Gold and dollar even more extended now in model space with values consistent with a reversal; by analogy a couple of springs pulled too far…looking forward to seeing whether this will be a mild pause or rush back. 



I recently watched an excellent talk at TED by Prof. Laurie Santos at Yale on how monkeys deal with risk just like people.



Errors in statistics are usefully classified as type 1 and type 2. A type 1 is a false positive or undue credulity and a type 2 error is a false negative or false skepticism. The greater you try to reduce the level of error in one the greater the likelihood of error in the other.

                                          Don't reject             reject

no effect hypothesis true       correct                type 1 error              

no effect hypothesis false      type 2 error          correct

A useful way of considering the decision making is above. Consider for example the no effect hypothesis that a pill is not healthy. if it's not healthy and you say it's healthy you make a type 1. If it's healthy and you don't say it is healthy you make a type 2.

A certain agency that regulates drugs is famous for only considering the type 1 errors, making sure with endless and ruinous double blinds that type 1 errors are minimized to the excessive making of type 2 errors and keeping off magic bullets that would extend life span and health enormously.

There are many areas where these trade-offs between errors occur. For example in spam filters. You can reject good things, that's type 1. You can accept bad things– that's type 2.

Our own field often has trade-offs like this. The hypothesis that a system or set point for a trade is random is a good null hypothesis. If you accept the system, you're just incurring churning for a worthless randomness. If you don't accept the system, and it's good, why then you've lost some good money.

The decision to expand your business or trading is another area that crops up frequently. If you expand it you might get in over the head. If you dont expand it, you might miss the gold. The movement into a new field, or the engagement of an employee or employer is another frequent trade off of type 1 and type 2, gullible reaching versus excessive caution that frequently arises.

The usual way to trade off between the two types of errors is to consider the cost of both errors, and to balance your decisions based on the relative costs. Considerations relative to randomness, and variability must also be considered. Also, the myriad psychological biases that lead us to place too much reliance on avoiding the two types of errors that the cognitives have contrived with their silly experiments on college students et al.

What other trade-offs of type 1 versus type 2 do you see that mite be of use to market people or others and what better way to consider gullibility versus skepticism do you see?

Alan Millhone writes:

This weekend I will travel to Grove City, Pa for a yearly Checker Tournament. While playing I will have choices to make. Sometimes there's only one way to move. Often times there's more than one way to move or jump. Checker players and Market players need to evaluate all moves or trades before executing. In Checkers if you touch a piece you have to move that piece– often with disastrous results. If you trade on line you need to consider your trade carefully before hitting "send". Tom said, "Move in haste— repent in leisure". 

Victor Niederhoffer adds:

Sharif KhanThe trade-off in errors in games like checkers and chess would be someone offers you a seeming advantage. Your null hypothesis is that it's not worth accepting. If you take the gambit or seeming opportunity when it's really no good you're making a type 1 error.

In checkers I've found that no opportunity that looks good, no opportunity to set a trap for example, is worthwhile against a good player, as good players never make mistakes. You were too gullible. If you don't take the opportunity when it would have been good, you're making a type 2 error. You were too skeptical. I find that in checkers the type 1 errors are much more costly than the type 2 errors, but in chess I don't know enough to say. But among the good players, I think they often are too cautious or too skeptical if they wish to win a world championships. They are too likely to go for the draws. In general, I would say if you want to be the best you have to be ready to make the type 2 errors to a greater extent. But then you always risk going belly up.

The situations are not without personal applicability to myself. It's easier in squash. I played an errorless game. Never made a type 1 error of going for broke with very risky shots. Well, it wasn't that bad. i went for about 5 years without losing a game in a match or so. But it wasn't good enough to beat the infernal Sharif Khan as much as I should. I should have played a much more errorful game, being willing to accept the risky shots and confrontations and hitting it on the rise and changing my infernal errorless slice backhand to a top spin so I could belt the ball through the Khans the way the Cubans who played Jai Lai could. In other words, I didn't make as many type 2 errors as I should have. 

Anatoly Veltman comments:

I remember grandpa coaching me at 5 or so: "always believe a man." If it turns out to be a lie, you'll find ways to extricate. But if you distrusted without good reason to begin with, you risk losing a friend– and that's an ultimate loss.

Jim Sogi adds:

Why do smart people make either type 1 or 2 mistakes? Presumably, and by definition, it is not because of stupidity, so some heuristic must be at play. In type 1, the fear and result is that you look and feel stupid. In type 2, there is less risk of looking and feeling stupid, but you end up being frustrated by the loss of opportunity. The joke around here is "which is worse" –you present a bad and a ridiculously bad alternative. Weighing the cost benefit is faulty because of proven heuristics are lopsided towards avoidance rather than gain. Add in marginal utility considerations and the difficulty is even harder. 

Bill Egan comments:

I will add a twist to the type 1 error problem. I have seen certain extremely intelligent people simply be unable to conceive they might be wrong. This is a problem that gradually gets worse. They have been right 99% of the time because they are so smart, and as time passes, they make bigger and bigger bets because, 'hey, I've been right.' This isn't quite hubris or arrogance because they really are that good. Finally the odds catch up with them.

Roger Longman writes:

Roger LongmanVic,

Love this.

So seems to me the real challenge is figuring the cost of a type 1 or 2 decision.

In the case of the FDA, the costs of a type 2 error are dramatically higher than the cost of a type 1. Those costs aren't financial, or not primarily financial. There's the substantial humiliation cost, for example, of approving a drug that turns out to have some important side-effect (or a side-effect more important to a group of influential people than its benefit to a group of less influential people) and being dragged in front of a congressional committee (Charles Grassley of Iowa has been the grandmaster of this, but he's got plenty of competition). There's the bureaucratic cost of being passed over for a more visible job, or an interesting review opportunity, if your name is associated with a controversial decision.

All of which is to say: of course the FDA errs more on type 2's. And the growing pharmacopeia (much of which is generic and therefore low- cost) only encourages this bias towards type 2 errors, particularly in regard to follow-on drugs (i.e., new molecular entities in the same class as approved drugs). If — the FDA figures, albeit not publicly
– by instinct, as it were — there is already a good drug that helps a majority of people why take the chance that a second drug in the same class will provide more incremental benefit than incremental risk, which — as I note above — comes with disproportionate institutional costs?

There's also an inherent problem with drug development divorced from serious comparative effectiveness (the current system of purchasing drugs, based as much on rebates retained by payers as medical and economic value provided to patients and employers, actually discourages the kind of comparisons common in most sectors of the consumer economy).

Victor Niederhoffer comments:

Longman was editor of Windhover Information Ventures Biomed and is very knowledgeable about the FDA. I wish he had been at Prof Tabarrok 's talk at junta where the positive case for the FDA going out of business was limned with statistics and current studies on deaths caused by lack of approval.



FeynmanWhy are there tides? How do the planets move? Just a couple questions you will get the answers to in the first of Feynman's wonderful lectures

Prof. Richard Feynman gave a series of lectures on physics at Cornell in 1964, which are called the Messenger lectures. They are now available free on the web courtesy of Bill Gates. Very engaging, only a few equations (at least in the first one that I have watched), so suitable for a general audience teen or older. (These are different than the Feynman lectures on physics, which are a three volume book set with all the equations based on the physics course he taught freshman at CalTech).

For those unfamiliar with Feynman, he played the bongos, drew nude art, enjoyed safecracking, helped create the atomic bomb at Los Alamos as part of the Manhattan Project, won the Nobel Prize in Physics in 1965, and was revered as a teacher of physics.

Chris Cooper comments:

Feynman taught Physics 1 and 2 only for the two years it took to complete the series of lectures which became the three-book text for those courses in the following years when I took them. I still have my copies because even at that age I realized that they were something special. As a senior, star-struck, I took an advanced course in quantum mechanics from Prof. Feynman. Although we had two texts to cover the subject, I was amazed to come to class and watch him explain the topic of the day from an entirely different angle, time after time. That was the class where I best learned my most important lesson– the people who do this are really smart, smarter than I, and they also work much harder than I was willing to. I flunked miserably, confirming my abandonment of dreams of a career in high-energy physics. In retrospect, Prof. Feynman ended up teaching me the most important lesson that I needed to learn at that time, though he never even knew who I was.



 I just found a good book. "Pattern Recognition, 4th ed." by Theodoridis and Koutroumbas. Authors are at the University of Athens and the National Observatory of Athens.The text begins with Bayes classifiers, then covers linear classifiers, nonlinear classifiers, feature generation and selection, and ends with a large section on clustering. Clear writing and Matlab code at the ends of chapters. Introductory level but with a very wide coverage of algorithms.

A companion book has some more Matlab code and examples. The code examples will help you understand the algorithms. More code at the website, too.

My obligatory warning– please learn learn about linear regression methods at the level of chapters 1-12 of Neter et al. "Applied Linear Statistical Models, 4th ed." before you tackle any of these books. Neter will teach you the loop and matrix forms of the algorithms for regression, plus the basic theories, all of which will make any pattern recognition book much easier to understand, including this one.

P.S. the 2nd edition of Hastie et al.'s "The Elements of Statistical Learning" is now available for free as a pdf.

P.P.S. A third good book is Bishop's "Pattern Recognition and Machine Learning." Between these three books you should be able to find something useful for a pattern recognition problem.



 I read and hear many people saying 'what a waste' today about the tsunami precautions in Hawaii. It reveals an interesting thought pattern that is a trap for traders and everyone confronted with imperfect prediction systems.

Are insurance premiums a waste? Is that fire extinguisher a waste? Doubt it, provided you did not overpay. We call this the Rule of 1. The Rule of 1 = the number of times a model (or modeler) can be wrong before people stop believing them.

Human brains have a horrible built-in concept of probability. We reward single successes and punish single failures when we should not. One day of good or bad luck tells little. Just because you did not die this month doesn't mean you can cancel the term life insurance for next month, which you originally bought to provide for your wife and kids in case you did die.

Traders need to estimate risk and act on that estimate, not ignore risk. No such thing as a perfect model. It (and you) will be wrong from time to time no matter how good it is (or you are). Frame the thinking as "What is the cost of the risk if it happens?" Heading for the hills after a huge earthquake is cheap insurance. The cost is of the risk is dying. A bad trade to stay on the beach. The beach will be there tomorrow. Other bad trades or too much leverage can cost you all your capital. The market will be there tomorrow, too.

Another rule for the reader. The Rule of 3. Suppose you test or look for something multiple times and do not find it. Can you estimate the probability it is there, even if you did not see it when you looked n times? The textbook example is: suppose you are told you need a serious and difficult to perform operation, and you are told the surgeon has performed the operation 20 times and no one died. What is the chance you will die? The Rule of 3 states that the 95% upper bound for no events is 3/n. So you still have a 3/20 or 15% chance of dying if you have the operation. (Van Belle, Statistical Rules of Thumb, 1st ed, pages 49-50)



at Iowa State UniversityHere are a few good statistics books that I have been using:

Sheshkin - Handbook of Parametric and Nonparametric Statistical Procedures

I have been using the 2nd and 3rd editions, although there is now a 4th edition out. Nice discussions and summaries with examples. Like a set of cookbook encyclopedia entries. Easy to use but not for a beginner (if you are, I echo Vic's recommendation to buy Snedecor - Statistical Methods, 8th ed).

Siegel - Nonparametric Statistics for the Behavioral Sciences, 2nd ed.(1988)

Very well written, with many clear examples and guidance on appropriate uses of various tests. A good book for a beginner. 2nd ed. is hard to find, and has lots of extras, but the the 1st edition (1956) is readily available for <$20 and considered a classic.

Conover - Practical Nonparametric Statistics, 3rd ed. Good and contains many tests. The prose is not as nice as Siegel but it has good examples so you can easily figure out what he is doing.

Martinez - Computational Statistics Handbook with Matlab, 2nd ed. Lots of good ideas and examples with code.

I am offering up nonparametric statistics books because I have been dealing with some ugly real life data and nonparametric methods can often handle that better. Real life data is often not normally distributed which is a requirement for many of the usual tests to work. The combination of nonparametric methods and simulations allowed by the power of modern computers is very, very handy for model building and answering hard questions like "did this really work or are we fooling ourselves?". Or worse questions like, "Here is the data. What happened?"



 Ford E-350 Super Duty 12-passenger vanI've noticed that every time I have sold a car, the dealer told me how flooded the market is with my make and model, and that the prices are much lower than I would expect. I take the dealer's words with a grain of salt, but am still not going to beat him in his own market. The equities and futures markets are much the same (and although this is anecdotal), it seems whenever I need to pitch out a position, there is a lot of whatever I own offered for sale at that moment. Conversely, whenever I wish to buy something, there is never enough around. I usually end up paying too much and selling too cheap.

Jeff Watson, surfer, speculator, poker player and art connoisseur, blogs as MasterOfTheUniverse.

Bill Egan writes:

You need a new van. A large one. By the end of February.

Ford E-350 Super Duty 12-passenger vans look good.

Prices for 2009 models

new MSRP: $36,325

new Invoice: $32,184

new Edmund's "What Others are Paying": $30,459


Edmunds used retail price: $25,808

Kelley Blue Book used retail price: $24,710


What is the distribution of true prices? Attached is the histogram of prices of 168 used 2009 Ford E-350 vans listed on within 500 miles of my house. Yes, the same basic vehicle is selling for $16,477 to $32,995. This took about ten minutes for me to make. (Yes, some options did vary, and there were a few misclassified vehicles, but overall this is really useful)

It is even faster to look at the most useful percentiles. Just sort by price on Then go to the last page to get the total number of vehicles that actually have a price. In this case there were 168 vans with prices. Scrolling through the four web pages and quickly counting allows you to figure out the percentiles. For example, the midpoint or median is at the 84th van (168/2). That price is $21,495. The 25th percentile is at the 42nd van (168/4) priced at $19,998.

Used list price at the closest Ford dealer: $21,695 (just over 50th percentile)

Used "best price" at closest Ford dealer after we had a chat about other dealers' prices: $19,495 (18th percentile).

For a fleet vehicle manufactured in March, 2009, auctioned in October, 2009, clean CarFax, with 13,200 miles and almost all the options.

Kim Zussman replies:

Don't you think the distribution of prices attributes less to mispricing than +/- valuation factors estimated by sellers — mileage, condition, options? They ought to know the going prices, and attempt to price theirs competitively modified by their knowledge of condition.

As with romance, if condition is held constant then price should vary based on ignorance and desperation.

Gordon Haave writes:

Car selling is all about one thing: price discrimination. The good salespeople do one thing: size up the maximum that you are willing to pay and make sure that is how much you pay. They do this through a number of techniques but most notably the questions about what you do for a living, how much your current car payment is, etc.

All the confusion around pricing and monthly payments and such is just to give them wiggle room to be able to charge you the most you are willing to pay.

Alan Millhone adds:

I pass a used car lot daily in Belpre and have been noticing the strategy this fellow uses to push his lot vehicles. He calls 89,000 miles LOW mileage ! One car he lists on window as a LOCALLY owned car — So? Occasionally he puts a HOLD sign on the windshield — who cares? Across the street is a strip plaza that a man owns that I know. On the front that is not used you can set your vehicle or boat or trailer and pay him 20.00 a month for that parking spot. Lots of traffic flows by each day. You list details on your vehicle and make your own deal. One new car dealer of KIA has his own clunker program and will give you 5,000.00 on your trade. My best friend traded in his car and a van he had towed in and the dealer allowed him 4,000.00 each towards a new KIA auto. I note gas in my area creeping up again so the economy cars are selling well.




While Belichick's decision is certainly controversial, especially in hindsight, I would not be surprised if objective statistical analysis showed that in general teams punt too often on 4th and short yardage.

Taking the usual kick and runback, perhaps the defensive team takes over an average of 30 yards down the field. If the offensive team can make the short yardage, say, 75% of the time, in general that seems a worthwhile tradeoff. Of course, Belichick was ahead, with two minutes to go, on his own 29 yard line, so not the most favorable case for it.

I recently heard of a small college team where the coach nearly always goes for it on 4th down, and nearly always on kickoffs kicks an offside kick to try to recover the ball. I bet he comes out ahead, although his tradeoffs are more favorable than in the NFL where kickers kick much farther, less chance of a kick's being blocked, etc.

I know nothing about football, but sounds a strategy worth trying more, especially if gives the defensive team a lot more uncertainty.

Bill Egan comments:

Belicheck's decision is a lovely example of bad risk control. He risked everything on one play.There are those who might argue that risk management should of course be a matter of necessary discipline. And even though it may behoove one's survival interests to never wager everything on one unknowable outcome, ratcheting up the risk parameters when one clearly has the upper hand, should never be ruled out. That said, even though one may steadfastly never risk anymore than x percent on any one outcome, it would hardly be unduly irresponsible to risk somewhat more than x to wager that Belicheck will not do that again anytime soon.

Why, because there's a considerable qualitative element at work here.

He wasn't the inevitable victim of the inevitable 100-year flood, some of which will inevitably if soggily cluster. No, he was the victim of not having any faith in his defensive employees.

And he knows full-well that the flesh-eaters in the media would reduce him to dry bones but quick if he did it that egregiously again, and it didn't work again.

Therefore, it would be relatively safe to dutifully wager that it won't happen under those same circumstances anytime soon, and it would hardly be imprudent to risk more than the ordinary x.

(Provided that's in the prospectus. Never mess with the lawyers, they're a lot cagier than football coaches. And the odds are always in their favor.)



 The Definitive Book of Body Language by Allan and Barbara Pease is a great book. It has clear and useful descriptions of body language signals from head to toe that will help you understand the intentions and views of people you are interacting with. Go buy!

The authors advise that this is not magic, and start early with three rules: 1) read gestures in clusters (no one signal tells all), 2) look for congruence (does speech agree with body language? if not, go with body language), 3) read gestures in context (crossed arms signal closed mind, but not if the person is just cold).

Useful business examples include: dominance via handshakes, types of fake smiles, closed mind signals, power gaze for intimidation, and the effects of seating arrangements at tables on meeting outcomes. Men are less innately skilled at reading body language, and the text includes a section on women's courtship signals.

Read chapter 19 first. It is a set of drawings designed to test you skill at reading body language. Then read the rest of the book and try chapter 19 again.



 Given the current mortgage rates and the fall of the housing market, I want to purchase my first home. Since I am stationed at Fort Hood in Texas, I have been doing heavy research in the Killeen / Harker Heights area. I thought I would ask for some advice. I spoke with Tim Melvin about this earlier, and he mentioned that I should never pay more than 10 times the annual rental rate of comparable houses. Does anyone else have any other good valuation metrics like this or have any knowledge / advice that would help me out as a first time homebuyer?

Legacy Daily replies:

I have found 10x to be used in two cases:

1. High house prices relative to rent — get one to cool off and think more clearly about an investment and do additional homework 2. Low house prices relative to rent - get one to jump in without thinking clearly on a "bargain" investment without doing any additional homework 

Some initial questions worth clarifying:

1. Is this a home or a leveraged investment? a. home — ignore rules like this and find the best place to live, raise a family, pursue happiness… b. leveraged investment — do enough homework to be confident enough about the decision to ignore all general rules.

Assuming investment:

2. What is the holding horizon? What future plans could interfere with that holding horizon? 3. What is the appreciation potential for the country, state, county, city, town, neighborhood, subdivision, this property…? I have not yet been able to come up with sufficient justification to buy for income alone when it comes to residential real estate. 4. What segment of rental market would the property (subdivision, neighborhood, town, etc.) attract? Is that the segment one wants to serve? Real estate agent needed to rent? 5. How predictable is the income stream? How would economic booms/busts affect it?
6. What are the worst case scenarios? What could go wrong?
7. Financial analysis — P&L, tax impact, financing options, downpayment flexibility (very illiquid), initial estimated repairs, etc. 8. Legal analysis — zoning issues, easements, property title issues, locality department issues, neighbor issues, etc. etc.

Couple additional points:

1. Decent real estate attorney representing one's interests can save from numerous headaches (especially true in foreclosure/short sale cases). 2. Avoiding a buyer's broker saves one money, gives additional negotiating room, makes the seller's broker more willing to work extra hard for the deal. 3. Inspections are money well spent, even if one does not end up buying the property. 4. The market is generally very efficient (yes even during this recession). Why has the property one's considering not sold yet? etc.

I hope you find this useful.

Jim Rogers writes:

The rule of thumb I've heard used is 1% of sales price should be equal to or less than comparable monthly rent (that's a little more aggressive than Tim Melvin's measure, especially when you factor in the mortgage tax shield). I'd say, use either and stick to your guns.

Sam Marx replies:

Don't trust what the real estate broker says about a house's value or price. Do your own research.

Try to find prices of recent sales of similar houses in same neighborhood.

Check with the local banks to see what houses they now own and what are their asking prices.

If you can go to foreclosure sales, do it, not to buy a house but to get an idea of what the market in houses is and remember those prices when negotiating with a broker.

I don't recommend buying at a foreclosure unless you're experienced at it.

Don't be shy about making offers 25-30% below asking price when dealing with a broker.

Watch for estate sales, the heirs are motivated sellers.

I don't know your area, maybe it's reached a bottom, but in FL, housing prices are still too high. The stock of St. Joe Land (JOE), FL's largest landowner, was 69 a few years ago — now it's 15.

Phil McDonnell advises:

 Buying a first home can be a frightening prospect. It should start with a realistic look at your needs. How many bedrooms and baths do you need now and in the future? If your life involves one or more women strongly consider the extra bath. If you have the skills a fixer upper my be of interest.

I frequently advise my Realtor wife on the statistical aspects of our local real estate market. Pricing in this market is especially tricky. It is a declining market but that also means buyers have much more negotiating leverage. To measure your local market ask a local Realtor for the latest stats on number of homes on the market and number of sales in the last few months in your area of interest. For a normal market this is about a four month supply of homes at the current monthly sales rate. In this market it is running about 10 months of inventory per home sold. Hence the declining prices as sellers compete. One should consider staying out of the market until the inventory show signs of declining. However do not be fooled by a one month decline in local inventory. Buyers in the Seattle area are negotiating prices an average of 4% below asking. Get the similar number in your area.

As a buyer in this market it is best to view the prices as a price distribution. Suppose we have ten houses in your area. But only 1 will sell in the area in the next month. Clearly it is most likely to be the one that offers the best value on a relative basis. The other nine are over priced for these market conditions. By staying on the market for another month they will probably lose something like 1% in value per month.

There is an old saying in real estate. One should buy the least expensive house in the neighborhood. Generally this is true. After numerous regressions on homes it can be said that among comparables the most important single factor is square foot of the house. For the best resale find out which area has the best schools. Even if you do not have kids the people who ultimately buy your home may have them and it will help resale in the long run.

Check out all the government mortgage deals and tax subsidies. They are offering a tax credit of up to $8,000 for first time buyers. 30 year fixed rates are below 5%. The military may offer even better deals. Remember the $8,000 credit is only paid the following year via a refund so you do not have it to use as a down payment. It is more beneficial the smaller the house you buy. I saw a recent home sold for something like $80,000 in Killeen. The $8k represents 10% on that home, but only 5% on a $160k home.

Dr. McDonnell is the author of Optimal Portfolio Modeling, Wiley, 2008

Henry Gifford adds:

Home prices, in general, are still falling in the US, therefore waiting will probably bring lower prices.

As property prices fluctuate, one sign of high prices is easy loans. Times when prices are better tend to be times when loans are hard to get, with of course reasons for this relationship. But, as an affiliate of the military, there are sometimes special deals available to you that are not available to other people, which means you can be one of the few buyers out there at a good time to buy. Some of these loan deals only exist on paper now, as the price limits and interest rates make them impractical, therefore nobody talks about them, but because they are government programs which get updated slowly, and usually out of sync with the market, they can be really good deals at times. Therefore there may come a time when you can get both a good price and a good loan.

Buying near a military base involves risk of base closure (I owned a whole bunch of houses near a base that closed) or downsizing, and since you're in Texas where there is lots of land, upsizing the base won't put much pressure on prices - people will simply build more houses. Perhaps you can ask around inside the gates to get a feel for this.

Buying and selling property involves large costs for brokers, taxes, title insurance, etc., which penalize short term ownership, meanwhile you can get transferred to another base at a moment's notice, which puts you in the position of being in a hurry to sell. If, instead, you buy a commercial property, you can own it as long as you live, with far less management headache, which makes owning it while living elsewhere more realistic than renting a house to someone.

Phil McDonnell responds:

I think the truth in this statement is based on a defect in the way people perceive value. Suppose the average home in a neighborhood sells for $500k but yours is worth $400k. Then if the average goes up to $600k the innumerate masses will think that all homes have gone up $100k not the 20% they really should have. When they do this the $400k home appreciates by 25% not 20%. In other words people add when they should multiply by a percent increase factor.

Dr. McDonnell is the author of Optimal Portfolio Modeling, Wiley, 2008

David Hillman writes:

Another part of that defect is focusing on the value of the improvements v. the value of the land.

Some years back, a close friend bought a lousy house on a great piece of property in the best neighborhood. Even though it was a prestigious address in a 'branded' area, he got a deal on the property because the house was so undesirable. The plan all along was to demo the house and built a new one to suit, which is exactly what he did. He had realized the land was worth perhaps 90% of the true total value of the property before the new construction.

Many county auditors, etc. have searchable tax records online with the assessed values of land/ improvements parsed out. One might use that to figure a reasonable estimate of market value of land v. improvements. Don't forget the old saws apply….'land, they're not making any more of it'….and….'location, location, location.'

Bill Egan writes:

In the last 10 years, I have bought three homes and sold two. Did not plan to, but that's the way it worked out due to job changes. Sold both houses in < 1 week for a profit despite forced timing. We were not in subprimeville, either, and the last sale was 2001 before the real estate madness.

My wife and I kept resale value in mind because you never know what can happen to you. We made sure we bought homes that were average to excellent on the following criteria:

  1. School quality
  2. Exterior appearance and interior layout — good and normal
  3. Quiet, safe neighborhood that looks good
  4. Reasonable size (3/2 or larger)
  5. Likely demand due to commuting routes/distance to jobs

For example, I was working at a biotech in NJ from 1999-2001. We bought a 3/2.5 in a newer development, nice neighborhood in Burlington County, right next to an average-quality elementary school. However, the area was less horridly expensive than the homes closer to Princeton, where I commuted to. There was strong demand from people priced out of the homes closer to NYC/Princeton.

Rich Bubb replies:

1.  look at the neighbors. C-L-O-S-E-L-Y… look at the state of their domiciles (even getting "invited-in" for a look see if at all possible), and the state of the upkeeping… especially the immediate next door folk. You might end up living next door to your own personal nightmare. Believe me, it is Not Enjoyable. Even after almost 20 years. Thankfully everyone else on the entire block is somewhat more sane and respectful of their neighbors than my nextdoor nightmare. Or to put it another way: you might get the best deal that no one else could stand…

2. if you really know somebody in the real estate biz (my sister is an agent), have them look around for you. she got her daughter's family a fabulous deal in a great neighborhood. Or to put it another way: sometimes real professionals Do Know what they're doing.

3. look long at the deal, bid low for the deal (Game Theory might help a little here, here is a cool intro), then be prepared to walk away… even if not doing the deal means you'll have to go back and start the whole search-etc process all over again, and don't put pressure on yourself or let anyone pressure you into buying. My wife was not prepared to walk away from her last car purchase. She still got a good vehicle, but she could've strengthened her bargaining position by uttering the words, "Let me think about it." And then purposefully heading for the door. We went outside and argued between ourselves about leaving. She *wanted the vehicle*. It cost her almost $5k more than I wanted her to pay.

4. Consider the cost of long term ownership. I mean, Really figure it out… what's the cost of x, and y, and z, and can you afford it if those costs all hit at once.

5. Tangentially to #1 above, if there'll be kids living next door… would you:

(a) invite them in?, or

(b) chase them away?, or

(c) start scouting for really out-of-the-way burial sites?, or

(d) let them borrow your most deadly power tools?

Just mentioning this as my siblings and I were the 'b-c-d' and almost always the Never-more-than-once 'a'. And the neighborhood's less-than-model parents would often let their barbarians-in-training train at our place… Or to put it another way: your neighbors' kids might have fiends, er friends, worse than they already are…

Hmmm, karma might really exist…

Russ Herrold adds:

A anonymous blogger, 'Benjamin.Publicus' on Thomas Paine's blog  had this this observation:

… The author lives in a community that is (or was) at the epicenter of the mortgage crisis. The developer aggressively marketed the homes to young, first time home buyers, many of whom renters. No money down, own instead of rent, mortgage payments the same as the rent, etc, etc. The development was started in 2001, so the first wave of 5 year ARM's hit in 2006.

…and it goes on from there.

I have spoken to that author (and a couple others) about contributing to DailySpec, but he has been busy.

Dr. Herrold is Principal of Owl River Company, a high-end Unix consultancy

Rich Bubb adds:

As mentioned previously, my sister is a real estate agent. following are her comments on home shopping & buying.

Get a Real Estate Agent to represent YOU as a BUYER. Sign a contract as such. Tell them what YOU want.

There are surely things important to you that you would like to have in one of the biggest investment decisions you will make.

TAKE NOTES of likes/dis-likes of each home you view. re: Basement, Garage, Four Bedroom, Square Footage, LOCATION. I stress location because it can make or break the satifaction of your purchase.

Drive through the neighborhoods you are considering at different times of the day to see what the atmosphere is.Pay attention to the neighbors up keeping of their property. Schools?, established neighborhood?, new additions? child / adult ratio?
Comparison shop, don't just jump at the first home you look at just because you can afford it. Ask your agent to provide you with a CMA (a market analisis of a surrounding area - 5 mile radius ).

Get pre-approval from your lender, look at homes a bit higher than your range and offer LESS - the worst that can happen is, they will say NO or counter-offer and you may wind up with a nicer quality home.

BE Strong in making the decisions of your offers. Be prepared to give and take.

Then BE PATIENT thru the purchase process which seems like it takes forever because we are a see it, buy it, want it now, kind of people. It is a process that is in place to protect you. re: CLEAR TITLE

Again, don't just settle for a home, get as close to what you want as possible.



 I found myself lying awake in my bed last night thinking about the Nobel Prize Winner. No! Not like that….but about what he said in Stockholm last week. Expected Utility Optimization. What he said is that the goal of asset allocation should be optimizing the expected utility for the actual investor in question, and that the mean variance model should just be looked upon as a special case. And of course he is right. I mean, by the way he sets it up, he is right by definition. But….I am thinking how it would play out in the real world. In my fantasy, a consultant would sit down with an investor, asking questions to find out his preferences. Of course this is already happening in a general sense but here it would end in a very specific investor utility function). Then the asset allocation would be done based on the utility function.

I am thinking that what will be overlayed on the usual return/risk models, are constraints (e.g cutting off tail risk, smoothing out fluctuations and what have you) and while the model presumably maximises return given a risk level and those added constraints; if we add constraints there must be risk premia transferred to someone else? By definition, since the investor specified his utility function (and given that the formulas and models held up and he got "what he wanted") he is better off than before, but so must someone else be?

I am not sure this new allocation model will start a revolution in the way asset allocation is done. I think however that finding situations where other investors are up against constraints, could help open up possibilities and profits. In the micro realm, many traders prefer to cut off the risk of gaps against them, by not holding overnight. This might open up possibilities for traders well capitalised and with good stomach, to do just that (this must be tested). Other suggestions are welcome.

Adi Schnytzer critiques:

AdiIt never ceases to amaze me that people who know markets and work in them don't realise that we don't know the probability that anything will happen tomorrow unless we are in a fair casino. So the idea that anyone can maximize expected utility is nonesense since you don't know the probabilities. I am currently working on developing a risk index as a follow-up to such an index developed recently by Aumann. He cutely argues that even though we don't often know the probabilities to assign to events, it's important that, in principle at least, we have an index. Well, I've been looking for real life examples of his index (and my follow-up) in stock and derivative markets, and simply cannot find one. As a top bookie once said to me: "If I only knew the winning probabilities of the horses, I wouldn't need to know winners; I'd be making a fortune anyway." Spot on.

Jim Sogi adds:

Martin talked about "…cutting off tail risk".

The thesis that outliers shape the future is intriguing, but also that the risk cannot be eliminated. The idea that one can cut left tail risk is an illusion that in itself creates a greater risk. As Phil says, it also cuts right tail return.

Jeff Watson concurs:

Risk can be quantified, assumed, bought, sold, transferred, created, subordinated, reassigned, split, delayed, diluted,  fragmented, hedged against, and layed off……. Risk can respond to some methods, but it is still risk, and is near impossible to eliminate.

Speaking of planning in general, Stefan Jovanovich adds:

I have quoted this before, but it seems worth repeating, if only to add a mite to Adi's wisdom. Planning in business is all very well, but the trouble is that your plan's assumptions always turn out to be works of fiction. As John Wannamaker said, "I know half the money I spend on advertising is wasted. If someone would tell me which half, I would very much appreciate it."

Vince Fulco concurs:

This quote has always seemed appropriate… 

Moltke's famous statement that "No campaign plan survives first contact with the enemy" is a classic reflection of Clausewitz's insistence on the roles of chance, friction, "fog," and uncertainty in war. The idea that actual war includes "friction" which deranges, to a greater or lesser degree, all prior arrangements, has become common currency in other fields as well (e.g., business strategy, sports). [Wikipedia].

Russ Humbert warns:

One of the hardest things to get people to see is that most people/businesses have a long term utility function but operate as if all risk is short term volatility.  For example, I work for a company that has a niche market and is privately held. The owner wants to pass this business on to his great-grand kids so each will be as well off as he is now.  He has only teen kids now. This niche has very little volatility of earnings and good ROEs. But this just encourages piling on the same long term risk, to minimize the short term risk.  That is: grow the core business, not diversify. We already have the leading player in this niche.  Barriers of entry: a learning curve, requires some marketing  nimbleness, and need for stable size and reputation.   However, long term this has  no good ending. Best case we double our market share and flatline growth. But many worse cases.  Bigger, deeper pocket competitor or many, learns our niche attracted by the ROE and stable vol. We are regulated out of the market. Products slowly go obsolete, replaced by Government safety net. We lose our reputation, etc.  See this in spades throughout the fallen out of favor or failed businesses, due to subprime mess.  Low vol high ROE business, until….  For the speculator this would be like choosing a strategy that 95% time gives "Alpha" in a beta model based on quarterly results of recent history.  But all the "alpha" is hidden because, 5% time it causes you to go broke or close to it.  It just hasn't happen yet, or recently.   Basically volatility as a risk measure can hide long term complacency defeating most utility functions.

Going back to the military aspect Bill Egan adds:

An interesting aspect of the fog of war is the common mistake of not reevaluating the plan often. A major cause of this error is that people confuse perseverence towards a goal (a good thing) with sticking to the particular plan they are using at the moment to achieve that goal. Criticism of the plan and proposing actual changes to deal with new information or uncertainty are considered as defeatism or disloyalty and the operationally fluid are smacked down. The no longer relevant plan is then ridden on to failure to a loud chorus of "yes, sir! yes, sir! three bags full, sir!" A pleasant sight if it is your opponent doing this but awful if it is your leadership. I have fond memories of serving as a company commander under a battalion commander who always asked us to tell him if he wasn't making sense and meant it. Good man. 

Phil McDonnell  enlightens:

PhilThere are many deep questions in Mr. Lindkvist's ruminations on Expected Utility Optimization.

My first comment would be that there are at least two distinct classes of utility function. The first class might be what can be called the Ad Hoc Class. This would include the questionnaire method of approximating one's utility function.

Other methods might be classified as normative, as in what one should ideally want to use for a utility function. As a well known example we have the Sharpe Ratio. This is based upon the normative idea that one should maximize expected return but with a quadratic penalty for increased volatility which is treated as a surrogate for risk.

The idea of using a square root function as a weighting for betting returns actually goes back several centuries to Cramer, a mathematician. His friend and frequent correspondent Daniel Bernoulli countered with the idea of a logarithmic weighting function, which is also what I espouse with extensions. Bernoulli's ideas were not translated into English until the 1950s and thus were lost to Western thinking until very recently.

Dr. McDonnell is the author of Optimal Portfolio Modeling, Wiley, 2008



J SogiKaizen is a Japanese concept which means continuous incremental improvement. This is a process in contrast to and in disproof of the idea that outliers alone form history. It is the tortoise and hare issue. Incremental and thus compounded gains allowed Toyota to become the largest car maker in the world. This kind of steady gains over time arguably has been responsible for more and greater changes than the discontinuities. In markets it is the idea that it is hard to beat a buy and hold over the centuries. The slow advance of human records over years is another example of incremental improvement. For traders a steady improvements in skills, new and changing techniques to adapt to ever changing market cycles and a steady return is a good alternative to a boom bust methodology.

The larger brokers such as Lehman use up to 25-35x leverage. Banks are leveraged up to 20x with less than 5% capital. The common historical variance in any particular market should lead to some big swings in equity. The real estate market would similarly be leveraged at least 5-10x or more with a 10-20% down. This level is like the "Mexican option". The levels seem to be coming down now. This is affecting market action.

Scott Brooks dissents:

Scott BOutliers may not "form" history, but they do lead it. The outliers in history are the ones that led the way to new and innovative change. Whether you look at outliers like Alexander the Great, Hitler, Churchill, Washington, Charlemagne, or Ford, Rockefeller, Edison, Morgan, or Shakespeare, Van Gogh, Picasso, Beethoven, The Beatles. It's the great ones that lead the direction or show the way. Society as a whole makes a decision, in the form of many individual decisions, to follow the lead of those that are paving the way.

There is no question that society as a whole benefits and moves forward in a buy and hold methodology. But the great ones lead the way and change the course of mankind….whether on purpose, or by accident.

Are the moves of these great leaders a mere function of chance? Yes and no. Greatness is predictable, it's just not known where it will come from or the impact it will have. Just as one can't predict the outcome of a horse race of a coin flip with 100% accuracy, we do know that the coin will land with some result, and that some horse will win the race. These horses are the one's that shape history and direction, just as the great (or infamous) leader have over time….which has lead to the incremental progress of the human race. But the fact that some men have greatness is less a function of chance and more a function of an ongoing decision process. They each, literally, decide to be/do something and work towards that goal. There is no chance in a pursuit of greatness or a goal. There is no greatness in a winning a lottery, only chance. Achievement of greatness is not about the result achieved, it's about the process one followed to achieve that greatness.

I believe the same is true from trading. We all can make progress if we're willing to work towards that goal and learn from our past experiences and build on those experiences as well as the experiences of others that we learn from. Greatness comes from process we learn over time and in the falsification of beliefs that we hold to move towards a higher truth.

Average and ordinary people don't want to believe in something new. They want to continue to believe that the earth is flat or the that universe revolves around the sun. It takes a great man to falsify those beliefs, and move the whole of mankind in the direction of greater enlightenment.

Riz Din concludes:

The interplay of outliers and incremental change over the long course of history seems like a very natural process, although sometimes I pray for the outlier to arrive quickly and disrupt the state of affairs because incremental change has led to bloatware, bloated institutions, etc. that are riddled with inefficiencies. When the incremental path is followed, things get embedded in such a way that there is no way of overhauling the system and it takes a competitor to do what is necessary and start over from scratch.

Here is an interesting Economist article from my archives on the topic, it is from the excellent, Millenium edition (Dec, 1999), and discusses living standards over the past thousand years in the context outliers leading the way, incremental change, and our benefiting of compound return of growth/living standards.

Take a look at the chart.

In our life-times we see steady year on year growth and improvements in conditions and view this as normal. However, the author of the article notes, "material prosperity has risen more in the past 250 years than in the previous 10,000. And so conditioned to growth have people become that most westerners now expect their standard of living to improve automatically year by year; if it does not, something is wrong. This taking for granted what would once have seemed miraculous is the measure of the change."

Jim Sogi tries to get the last word in:

The Economist article Riz cites falls for the recency effect, that recent events are more important. Not so. The invention of language, the wheel, fire, tools, iron, writing, and printing presses probably surpass recent inventions in creating better prosperity advances and change. 



Researcher: Basic Greenhouse Equations "Totally Wrong"
Michael Asher - March 6, 2008 11:02 AM

New derivation of equations governing the greenhouse effect reveals "runaway warming" impossible

How did modern researchers make such a mistake? They relied upon equations derived over 80 years ago, equations which left off one term from the final solution.

Miskolczi's story reads like a book. Looking at a series of differential equations for the greenhouse effect, he noticed the solution — originally done in 1922 by Arthur Milne, but still used by climate researchers today — ignored boundary conditions by assuming an "infinitely thick" atmosphere. Similar assumptions are common when solving differential equations; they simplify the calculations and often result in a result that still very closely matches reality. But not always.

So Miskolczi re-derived the solution, this time using the proper boundary conditions for an atmosphere that is not infinite. His result included a new term, which acts as a negative feedback to counter the positive forcing. At low levels, the new term means a small difference … but as greenhouse gases rise, the negative feedback predominates, forcing values back down.



Gregg wrote a nice comment: "This NYT editorial story is not good for drug stocks which depend on revenues from cholesterol lowering drugs. That is good and bad. Good from a moral-philosophic point of view, bad for my portfolio."

Drug stocks will become much more volatile in the next 5 years. Many billion dollar patents are expiring and the pipelines are not there to replace them. Big pharma will downsize more and more in US/EU, and throw money at all biologicals as well as small molecules designed for cancer to fill pipelines, and shift research projects (and esp. regulatory testing) for small molecules to cheaper chemists in China/India. Why?

Labor costs. Wu Xi in China (WX on NYSE) a great example of the last. WX is also getting into cheaper regulatory/development testing. Many big pharmas opening R&D centers in China, too (Shanghai). Biologics are harder to make but easier to develop (usually less toxic). Small molecules for cancer are more likely to get approved. Big pharma has made billions on small molecule drugs for chronic conditions. Double whammy here - a) many good drugs coming off patent for those conditions, so unless the new drug is much superior, it has no chance cost-wise, b) FDA is now much tougher on the safety of drugs for chronic conditions, especially if the condition is not life-threatening. Cancer is still hard to treat and life-threatening.

VCs seem to be putting less money into traditional US small molecule startups. If I were a VC, I would move money to China/India, but also look for plays for pipeline fillers. Big pharma execs have to be seen "doing something" or lose their jobs.

Their real problem is the length of the R&D cycle. Research averages 5 years, clinical testing in humans averages 7 years, and FDA approval time is ~14 months. So 13 year cycle.

Fail rates are huge. Take 10 research projects. Each makes 2,000 molecules on average to produce a clinical candidate (drug to test in man). One of those ten projects will produce a drug. So about 1 in 20,000 molecules becomes a drug after 13 years. Why the fail rate is so huge is another essay.

Consequences are that big pharma execs reorganize alot and buy pipeline fillers while cutting costs and praying the pipeline gets better randomly (and they fire the research head, too). Changes made back in research may or may not affects the success rates. Doesn't matter much, either, from the CEO viewpoint. Even if a research process change/new technology tripled the success rate, you couldn't tell easily. 3x better means 1 in 6,667 molecules succeeding after 13 years… So big pharma execs "do something now" even if it has little chance of fundamentally changing the odds in order to save their own necks. Likely more shenanigans like hiding of bad data on Vioxx and new Vytorin ethics scandal.

This means lots of volatility. The problem is the high fail rates mean it is tough to tell which small companies are any good. Interesting times ahead.



Foam RocketsWe have started son Will, 9, doing science with Dad on Saturday mornings (and Josh, 6, tags along) while Mom takes John, 4, and Anna, 2, to music class.

Last week, we learned about averages and how to compute them. Small boys like weaponry, so Dad had th idea to take Grandad's gift of air pressure fired rockets (little foam rockets you attach to a bulb which you squeeze hard) down to the garage. We fired 5 rockets per boy out of the doorway into the driveway, measured the distance in inches, and computed the average. Wide variation was noted and will be the topic of "standard deviation" class later. First we had to get some good data. Josh was ecstatic to shoot farther than Will, but even he thought it interesting that sometimes Will outshot him and vice versa.
Today, Will fired up the Electronic Snap Circuit kit. Project #1 was a little boring but a good start — make a circuit and use a switch to turn on a light bulb.

Project #2 made up for that. This one has the kid make a circuit with a switch and a motor. The motor has a 3-bladed propellor attachment. If you run the motor long enough, the propellor comes off and zooms across the dining room like a flying saucer. Great howls of laughter with dancing for joy and then repeats.

All good scientists keep some sort of notebook. Will currently has to write down date, title, the data he collected (rocket inches, for examples), and a couple sentences describing what he did and what happened. This part is boring, so I'm going easy on it at first, but he is doing well.



 Herman Weyl's Symmetry, is a small book, modest in approach, but has huge ideas addressing core issues in philosophy, math and markets. He addresses symmetry from a generalized qualitative method, gives fascinating specific examples from biology, phylogeny, ontogeny, crystals, bee honeycombs, Egyptian, Greek, Persian, Sumerian art, math, physics and then generalizes to a an astonishing mathematical model right in line with the speculative approach with a multitude of trading applications. One of most fundamental ideas is that symmetry is more than just an aesthetic quality, but one of the core laws of the universe and leading to predictive application.

Symmetry arises from the very mapping of space and time and lies at the core of relativistic thought. Time is symmetry in the past and future revolving around the here and now relative to light and the point of observation. It is present in all higher life as bilateral symmetry. Life has laevo and dextro forms, a left and right. A regular symmetry in time is called rhythm in music. Art forms are filled with examples of symmetrical patterns in friezes, architecture, vases, floor tiles, in art. Symmetry is the basis of beauty and aesthetics. Curiously, in all life, my daughter tells me, DNA is left handed.

The quantitative model defines symmetries by defining the groups and examining the congruencies or mirror images in 2d or crystalline structures in 3d and reality in 4d or higher dimensions. The simple form of the bilateral congruencies, which applies to market common two axis lattice approaches is AB-BC: AC. By examining the mirror images, one can define the axis of rotation with obvious market application at 1/2a. Weyl examines transformations of the groups and claims that there are limited essentially different congruencies, which for markets can basically be described as patterns. The underlying characteristics can be rigorously defined.

In one of the most fascinating sections of the book, Weyl discusses the problem that the Pythagoreans guarded as a great secret and divides the schools of geometers from the algebraists. The essential problem is that algebra cannot solve the relation of a diagonal with its squares due to the irrationality of the square root of two, or curves due to pi, but solutions to that and many other problems can be solved with a ruler and compass. Weyl introduces Cartesian manipulation to solve the Pythagorean dilemma that was solved in in Universal Math using simple arithmetic and avoiding the irrational number problem.

Market symmetry is found in its basic microstructure of bid and ask. There is essential symmetry in the market negative correlation. The key is to find the axis of symmetry. Here is the key to investing, trading and identifying changes in cycles as well.

Weyl's method of defining auto morphisms Symmetry underlies Chair's examples such as Lobogola, cane investing, release, consummation, penumbras. No less important are the asymmetries of drift, vig, information. The market desires and demands symmetry. It is not satisfied until the moves are consummated. There is the symmetry of equal length of moves in the waves of various durations and in the retracements. Sometimes simple symmetries provide the greatest profits without resort to math but by use of two fingers.

Bill Egan notes: 

I have a different example for you. Drugs are asymmetric; highly symmetric molecules generally do not work well in optimization programs in drug discovery. I refer to the actual 2-D/3-D shape of the molecule. 



"Reasonable people adapt themselves to the world. Unreasonable people attempt to adapt the world to themselves. All progress, therefore, depends on unreasonable people." - Shaw

Frame of reference is critical. We may go beyond the idea of the quote above by considering the viewpoints and knowledge of those involved. I am talking about more than the old lion phenomenon, where the old lions hate their younger successors and rehash the ideas of the past.

What if the problem can be solved through application of concepts foreign to the people in the discipline? Crackpots fail because they search for their new ideas too far into the wilderness. The Chair is one who exemplifies the search for useful new concepts in the knowledge of other disciplines. I recently built an effective combined experimental and computational model for a toxicology problem that the toxicologists said couldn't possibly work. They owe me a beer because their frame of reference (standard toxicology) was wrong for the problem, when my frame of reference (spectroscopy and computational modeling) easily explained what was happening.



 I have been thinking a lot about the important problem of establishing cause and effect, since Nigel Davies brought up the topic earlier in the week.

It seems like the more complicated systems become, the more difficult it is to establish cause and effect relationships between different phenomena. Physicists often say that two phenomena are "associated", when they don't dare establish a cause and effect relationship.

But the science that really has a problem with this is biology, which studies extremely complicated systems (remember that there are more cells in your body than there are stars in the Galaxy, and each cell is quite complex). Perhaps more biological thinking would do us good, rather than the physics-like thinking often used in market modeling.

In the germ theory of disease, for example, it is very difficult to establish the mechanism by which the germ actually causes the disease. And the mere presence of the germ in large numbers is not enough to establish cause and effect. The germ may be an opportunist, taking advantage of the diseased state to increase its numbers.

Anyway, around the turn of the last century an attempt was made to make criteria for establishing this kind of cause and effect. One scheme was called Koch's postulates. Here they are in their original form:

Koch's postulates are:

1. The microorganism must be found in all organisms suffering from the disease, but not in healthy organisms.

2. The microorganism must be isolated from a diseased organism and grown in pure culture.

3. The cultured microorganism should cause disease when introduced into a healthy organism.

4. The microorganism must be reisolated from the inoculated, diseased experimental host and identified as being identical to the original specific causative agent.

I'm sure my fellow Specs can find holes in this scheme. But can we establish our own postulates for cause and effect in markets? Of course we can't "introduce" and "culture" market phenomena (much as we would like to). But maybe some other kind of formal scheme would provide food for thought.

Nigel Davies adds: 

This seems like such a good analogy. For example there are recurrent conditions in which the sufferer has occasional outbreaks of the disease.

In considering outbreaks of a market disease there might need to be the presence of germs plus some trigger factor such as 'stress'. Even then there are no guarantees.

Bill Egan writes:

Tylenol (acetaminophen or paracetamol) is the #1 cause of fatal liver failure. It produces minor effects on the liver at the recommended dosage levels (4 grams per day in adults). Fatal liver damage (zone 3 necrosis) can happen with as little as 10 grams per day. The safety margin is a mere 2.5x between pain relief and death.

Most drugs are metabolized by enzymes in your liver, so that they can be excreted more easily. Acetaminophen is not an exception. Five percent of acetaminophen is converted to a reactive metabolite that will bind to liver cells and kill them. This is normally prevented by the body's anti-oxidant defense system, a molecule called glutathione, which binds to reactive molecules and thus prevents them from causing damage. Should your glutathione levels be low (not eating right, alcoholic, sick, etc.) you will be more susceptable to damage caused by the reactive metabolite of acetaminophen because your glutathion levels will be lower than normal. Should you ingest a bit too much acetaminophen, you rapidly exceed even the capacity of the normal levels of glutathione to prevent its damage. It is like a step function.

What market mechanisms act to prevent routine damage? What is the tolerance limit beyond which they are overwhelmed? (This is a violation of Koch's postulate #1 but it is chemical not biological.)



 How To Get Rich is a fun book to read. Unlike the usual book and seminar hucksters, Felix Dennis really is rich (est. 585 mil pounds). Dennis has a very clear and friendly writing style. It feels like he is talking to you when you read the book. He tells a number of good stories about how he made different piles of money, including how he bought The Week in 1973 when he was totally broke and hacking a Bruce Lee biography to pay the bills. This was just at the moment Lee died suspiciously, which jump-started all sorts of money making opportunities. There is wisdom here in Dennis' tales of success and failure; practical comments on negotiation, delegation, hiring, raising capital, poetry, and also dire warnings.

"Now comes the hard part. Before we really get started on getting started, I ask you to consider carefully the short list below. It is by no means comprehensive, nor will it be the last list in this book, but should you find yourself unable to measure up to even one of these initial demands (and I mean just one), then my suggestion is that you close this book and give it to a friend, or an enemy

- depending on the degree to which you enjoy ironical gestures.

- If you are unwilling to fail, sometimes publicly, and even catastrophically, you stand very little chance of ever getting rich.

- If you care what the neighbors think, you will never get rich.

- If you cannot bear the thought of causing worry to your family, spouse, or lover while you plough a lonely, dangerous road rather than taking the safe option of a regular job, you will never get rich.

- If you have artistic inclinations and fear that the search for wealth will coarsen such talents or degrade them, you will never get rich. (Because your fear, in this instance, is well justified.)

- If you are not prepared to work longer hours than almost anyone you know, despite the jibes of colleagues and friends, you are unlikely to get rich.

- If you cannot convince yourself that you are 'good enough' to be rich, you will never get rich.

- If you cannot treat your quest to get rich as a game, you will never be rich.

- If you cannot face up to your fear of failure, you will never be rich."

Concluding his tale of acquiring The Week and slowly making it work, Dennis says,

"Trust your instincts. Do not be a slave to them, but when your instincts are screaming, Go! Go! Go! Then it's time for you to decide whether you really want to be rich or not. You cannot do this in a deliberate, considered manner. You can't get rich by painting by numbers. You can only do it by becoming a predator, by waiting patiently, by remaining alert and constantly sniffing the air and by bringing massive, murderous force to bear upon your prey when you pounce."



After reading the paper the Chair found, my memory has been jogged. Introduction to Statistical Quality Control, 5th ed., Douglas Montgomery, pages 95-6 discusses the use of the range to estimate the standard deviation.

An unbiased estimator of the standard deviation s of a normal distribution is s(hat) = R/d2
R = range
d2=variable depending on n

So the factor 1.6926 is really d2 for n=3 (the # of GPS measurements Schwarz used).

For n=1 to 10, Appendix Table VI on page 725 of Montgomery gives:
n d2
2 1.128
3 1.693
4 2.059
5 2.326
6 2.534
7 2.704
8 2.847
9 2.970
10 3.078

Montgomery notes that the range method works very well (retains high efficiency) for small samples sizes (n <= 6).

Victor Niederhoffer writes:

An interesting article on
ranges shows that a good estimate of the standard deviation from a normal
distribution is range/1.7. Sequential estimates of the standard deviation from
the range, for example:

              date    range   stand dev

                4 02     10        6
                3 30     22      14
                3 29    14         8
                3 28    12         7
                3 27     8          5
                3 26    15         9 

For S&P futures this might provide a good template for thinking about short-term volatility. 

Bruno Ombreux adds:

Here is one of the early articles on the ratio of range to standard deviation, featuring tables for the ratio. Of course, today one can use resampling methods to get these kinds of ratios, even from non-normal populations.



 One of the reasons humans are still competitive with computers in chess is that we are aware of patterns that don't compute. Take, for example, nature of pawn structure. One can count individual pawn weaknesses but it's very hard to find an algorithm by which the harmony between pieces and pawns can be assessed. The human mind, however, is quite capable of this.

Might it not be the same with markets, that there are patterns which can't be effectively coded and others which can? As a very simple exercise one might try to count the number of waves that tend to accompany a decline from highs or see whether an n or u formation is being created. Seems to me that it's very, very difficult to do this with numbers; but the human eye is reasonably adept.

The problem of course is that without a clear computable definition of what one is looking for there will be too much that is open to "interpretation," so the results could hardly be relied up. So what is the solution?

I've been thinking about a possible way round this but please excuse me if it is scientifically unsound. What about having generic patterns that contain multiple computable definitions? For example one might have major categories like "panic" or "breakout," but then multiple and detailed definitions of what these are, just to be sure that the computer will recognize them but not for something else. Then when it comes to the stats the generic categories are tested rather than the details.

Just a thought.

Vincent Andres adds:

Another example: it's very easy and fast for a human eye to detect if points are aligned; it's quite a long calculus for a computer.

"So what is the solution?"

This is a deep question. One answer is to stop reasoning/computing with "crisp" sets. With crisp boundaries you have indeed threshold effects that make the reasoning/computing discontinuous and unstable. One way of doing that is using "fuzzy" sets. With fuzzy sets, set limits are no more crisp, but continuous. So working on them is more stable and more continuous. Nice applications are for instance in control.

Fuzzy sets are an interesting tool when it comes to trying to represent knowledge and work with it.

It's not a miraculous tool. Yes it is (or was) a buzzword. You can do the best and the worse with it. And it was done and it is done. Like with neural nets, genetic algorithms, etc, etc. Like with statistics, probability theory, etc. But it's a nice (and very mathematical) topic.

From Steve Leslie:

I like your analogy to visual patterns that don't compute. There are similar parallels in poker.

A computer can give exact statistics of making a hand and pot odds, etc. It can also calculate tendencies with a player. However poker is a game of imperfect information therefore much is subject to interpretation.

Now then:

Crandall Addington is one of the great poker players of all time and a true character. I saw him on TV 25 years ago playing in the old World Series of Poker with a $10,000 buy in. This was when no limit hold-em was essentially an obscure game and $10,000 was a lot of money. He was wearing a Mink Stetson. This was before PETA for sure.

He said that limit poker is a science but no-limit poker is an art.

Limit or structured poker contrary to popular belief contains little bluffing. Most of the hands are played straightforward. There are many multi-way pots and almost all hands go to a showdown.

No-limit hold-em is entirely different. Statistics and straightforward play will only take you so far. It is much more a game of playing the table and the opponents. A feel for the game, understanding its ebb and flow, and evaluating the dynamics of the players are critical. The best no limit poker players know when to be tight, when to turn aggressive, when to bluff, and when to truly gamble. This is where experience is essential.

Similarities occur in trading stocks and futures.

There are the fundamentalists. People like the Buffett of 20 years ago, who was a protégé of Benjamin Graham. Martin Whitman and others. They can be the value players and the grinders. They see big picture things and exploit opportunities but only when the balances are tilted in their favor.

There are certainly the quants, people like Mr. Symonds who obviously have a created a superior mousetrap. But of course, they are neither talking nor sharing what they have found to be successful. There are some others such as D.E.Shaw. Once again they are extremely secretive and are constantly working on their algorithms that identify patterns. Many of the employees are PhD's in computer sciences, mathematics, and music. They are the equivalent of the Rand Institute. Guys and gals who sit in seclusion and are constantly perfecting their own "black box."

Then there are those who trade on a combination of statistics and feel. They tend to be excellent at the "feel of the game" and reading the opponents. The Chair is one of the best of these. One of the finest traders in the world who worked for one of the great traders in Soros. Robert Prechter has had significant success trading off of Elliot Wave patterns.

Then there are the floor traders. They are very intuitive and great readers of the market. They get the first look at where the orders are being placed and who is placing them. In Education of a Speculator, Victor describes in detail how one of Soros's traders would enter the elevator to the floor and the bids would change. It became a game of the cat and the mouse.

In summary: There are opportunities for each of these to profit from the market. As each of the above have demonstrated in their abilities to make money time and time again. It then boils down to what kind of game are you are in and an understanding the rules.

From Bill Egan:

Plotting the data different ways pays off all the time. I earned a US patent because I examined bi-plots of ~50 variables and saw something interesting. Further investigation showed a sensible relationship to the physical mechanism I was interested in modeling, and I quickly built a model that has worked for eight years now.

I always use bi-plots. Once I have a feel for the data and can throw out some variables, I will color points in bi-plots by a third variable. I use this to highlight known extreme values, events, or odd experimental results. It often reveals useful patterns to the careful eye. Histograms of the distribution, data percentiles (percentile function in Matlab), and empirical cdfs are also handy. Multi-modal distributions are often interesting and show up in a histogram.

Software like SpotFire makes this very easy, and includes ways to size and shape data points by other variables (although it isn't cheap to buy). You can certainly do this sort of thing with a bit of work in R or Matlab or S+.

Another trick of the trade is to compute correlations among your variables. You can almost always remove a variable that is r^2 0.9+ with another variable. This will cut down on the amount visualization you need to do.

Further thought from Nigel Davies:

What if the most subtle and powerful engine for pattern recognition and synthesis is in fact the human brain? In this case shouldn't we be training ourselves rather than our computers?

Probably the search for patterns does this anyway but this would seem to be another benefit of the Chair's recommendation to hand count.



 Models with adjustable lookback periods, e.g., the length of a moving average, require a tedious if simple sanity check. Test all periods. Plot the results (final equity value or whatever other measure you are using) vs. period. Which are most profitable? Is a contiguous block of periods profitable?

For example, perhaps only shorter periods are profitable and longer periods lose money. Or are only a few scattered periods profitable? The last effect suggests the model is nonsense. If most periods are profitable, or if all periods between 1-30 days are profitable, that makes some sense. If the 24 and 66-day periods are profitable, but no others are, what real effect could be present to justify their profitability?

Many people use a 200-day moving average to estimate long-term trends. The model hypothesis is that long-term trends matter. I would not consider using a 200-day moving average to be data snooping or data mining. To me, data snooping is testing all possible periods and using the best one without doing any analysis like the one I suggested above. Poor analysis is testing only one period, even if it is your hypothesis, without checking the alternatives at all.



Song: Won’t Stop
Artist: Bob Seger

Album: Face the Promise 2006 

You can cry if you want to

You can rage at the night

You can blame all your wounds on the world if you like

you can drink from the bottle

no ice and no glass

you can lie in the morning and say it's your last

but you won't stop there

no you won't stop there

you can tell yourself different but you won't stop there

you can study the ancients

you can learn every fact

you can follow the cycles that leave and come back

how everything changes

it's better with us

one day you're a comet

the next day you're dust

but you won't stop there

no you won't stop there

there's always the future and you won't stop there

tyrants and kings do their usual things

and you try to stay out of their way

follow the truth and you'll find what you need everyday

there's always tomorrow

always a chance

that you can stand in the spotlight

and not have to dance

you can find something solid

stronger than steel

and it might touch your heart

cause it just might be real

but you won't stop there

no you won't stop there

it's not in your nature and you won't stop there

no you won't stop there

no you won't stop there

there's always tomorrow so you won't stop there



The normal distribution is a poor fit to the daily percentage returns of the S&P 500 from 1950-2005. The lognormal distribution is a poor fit to single period continuously compounded returns for the S&P 500, which means that future prices are not lognormally distributed. However, sums of continuously compounded returns are much more normal in their distribution, as would be expected based on the central limit theorem. A t-distribution with location/scale parameters is shown to be an excellent fit to the daily percentage returns of the S&P 500 Index. [Read more here]


Resources & Links