# Sampling Without Replacement, from Victor Niederhoffer

January 11, 2009 |

Sometimes one sees almost every conceivable variations of prices during the year. It's like an evil genius had a bag of tricks and never had to repeat one exactly the same way. I wonder if it's possible to turn this around and assume that there is a finite number of tricks, possible variations that will occur and then predict that the ones not used yet will eventually be used. This becomes particularly relevant for people who look for repetitions of past patterns and in days like this find that there is nothing similar to it in history. Regrettably, that is true almost every day. There should be some creative ways of testing this.

We could look at market entropy, in an information theoretical sense:

Code every possible pattern in bit form, eg 1110011000111

Measure entropy.

See if the market is maximizing it, this would be the "Second Principle of Market Dynamics".

We could also have a Prigogine's Theorem analog i.e the market is forming patterns that will minimize its entropy production.

As an analogous situation, I believe that slot machines, keno games, etc. have their results or numbers selected by random number generator formulas. I always thought that if one is an expert on the existing formulas or was able to generate a random number generator formula based on a series of outcomes then he could beat the game.

I realize that the casino could easily thwart this in keno but it would take additional work on their part for the slots.

In the stock & futures market, I understand there is some pattern recognition software now available. I have no experience with it.

## James Sogi writes:

Maybe sampling something simple like variance over the last couple days might give one a clue. Volatility clusters, and lack of volatility clusters, and variance of volatility within those clusters or length of the clusters, or the survival rates. Again the replacement issue and the assumption of independence clash. The replacement assumes independence, but a cluster model assumes some correlation.

A related (and very important) topic is the number of stable patterns achievable by a set of interconnected nodes. On this topic, a worthwhile read is Stuart Kauffman. Kauffman's work is rather well presented in a chapter of Deep Simplicity: Bringing Order to Chaos and Complexity , John Gribbin, Random House. 2005. ISBN 1-4000-6256-X.

Name

Email

Website

1. James M. Schroeder on January 9, 2009 10:51 am

I really like this as a way of thinking about the market, but even if one could list all of the market's tricks there would still be two main problems I can think of.

1. Maybe some of the tricks are present multiple times in the bag.
2. The bag must from time to time be refilled and shaken up, and figuring out how often this happens would be crucial.

Another perhaps more sinister problem would arise if one of the tricks in the bag was "repeat the last trick".

2. douglas roberts dimick on January 9, 2009 12:31 pm

Why “turn this around and assume that there is a finite number that will occur and then predict” for purposes of testing or modeling? Instead or a prospective assimilation, why not relate “variations of prices” to quantified time and mass correlations for testing (or indicating) functional states of (non)directional patterns? With or without replacement, statistical sampling of market input presents the constant (or repeating anomaly) of patterning (or trend or directional) variants due to numerically defined universe and population “formations.” Therefore, it (being that state-input-output patterning) cannot be reduced to a mathematic calculation (or quantification) without first articulating a metacircular (state-input-state) formulation of those static conditions (or the stochastic ecology) of both current and past price action – notably, the beginning and end of any given state, as so rules-based defined. dr

3. douglas roberts dimick on January 10, 2009 2:53 am

My research and developing QR (or quantitative relativity) found that applications of cluster and randomness theorems may encode (or embed) emotive imposition of patterns, even when we quantify such formulations, often being guised as science.

In a recent post reflecting upon the Madoff event, consider these selected excerpts…

“Dopamine neurons weren’t designed to deal with the unpredictable oscillations of Wall Street… following our primitive reward circuits. Unfortunately, the same circuits that are so good at tracking squirts of apple juice will fail completely in these utterly unpredictable situations.”

“People enjoy investing in the market and gambling in a casino for the same reason that they see Snoopy in the clouds,” says Read Montague. “When the brain is exposed to anything random, like a slot machine or the shape of a cloud, it automatically imposes a pattern onto the noise. But that isn’t Snoopy, and you haven’t found the secret pattern in the stock market.”

“That’s why a randomly selected stock portfolio will, over the long run, beat… the vast majority of mutual funds in any given year.”

“Since the market is a random walk with an upward slope,… [i]nvestors who do nothing to their stock portfolio - they don’t buy or sell a single stock - outperform the average “active” investor by nearly 10 percent.”

“Wall Street has always searched for the secret algorithm of financial success, but the secret is that there is no secret. The world is more random than we can imagine. That’s what our emotions can’t understand.”

Hedge Fund Fraud
Category: Brain & Behavior
Posted on: December 15, 2008 9:59 AM, by Jonah Lehrer http://scienceblogs.com/cortex/2008/12/hedge_fund_fraud.php

My call for a paradigm shift in program trading architecture is due to exactly that… proffered risk management schemes based on algo-quant constructs that are claimed to all but eliminate yet in truth encode the “human” (or emotive) element at precisely the key, central load-bearing truss of commonly engineered trading and portfolio management design: market states.

Sure, a computer executes order protocols. Who programs the protocols? Moreover, who programs the input-output derived theorems governing the (non)correlated (see Madoff video at youtube.com) pattern recognition systems?

Be it with random walks or clustering to identify patterning “formations” (or states) of price action, such patterns are ostensibly of a discrete, mathematically contrived ecology, which may or may not be repeated or so recognized by or during any given, subsequent (securities, sector, industry, or exchange specific) market transaction, isolated or sequential in nature or design.

Why? The conditions precedent to the formation of any natural or artificial correlation of state-input-state elements necessitated for any such given algorithmic formulation have been subjectively (i.e., human observed) or perhaps even “scientifically” memorialized — which indicates a “best guess” sequence of analyzing and distinguishing. Just because one or a gaggle of PhD’s so articulate a given price action phenomenon does not mean that “the market” either so realized or consciously patterned that exact (exchange) formatting of behavior.

Perhaps the comment regarding market entropy offers the citation of any number of market instituted or systemic events a la the second law as to universal entropy increasing over time, to wit: currentsession(0) currentsession(0)[1]. How may we quantify any given analog relative to time and mass before, during, and after this coded sequencing of mass and time?

Does market psychology cease to exist during the sequential interim? Would application of wave theory here require extension or recalibration? How are these first two queries correlative to entropy?

As an academic inquiry, one may appreciate the utility. However, from the perspective of risk management, what is the efficacy for such a pursuit albeit understanding the degree of metacircular deviations (and thus variants) occurring within algo-quant modeling?

dr

4. douglas roberts dimick on January 12, 2009 8:50 am

{ WEBMASTER NOTE: SORRY… HERE IS COMPLETE ENTRY… PICTURE OF BOOKCOVER MAY BE FOUND AT http://s220.photobucket.com/albums/dd138/douglasrobertsdimick/ }

During the translating and editing of my book here in China, my two diary stories (one in the preface and another at the close of the book) were considered confusing and perhaps best excluded from the final version. Here I offer the preface…

Preface: Two Sides of a One-Party Transaction?

The Art of Conversion: defined by skill and craft during a process of transference (or transcendence) involving the nature and structure of capital – financial, corporate, political, even social, or a combination thereof – of which is attainable and, therefore, to be gained, lost, or hedged, whereby value is created or transferred from an opportunity (e.g., the supply and demand of a product or a service) perceived or recognized as resulting in a net gain, to include an exchange process itself.

Reflecting this idea of value creation (or transference), here I offer the first entry of my diary in China…

March 30, 2006

There is 30% battery remaining on my laptop. Near 7pm, Thursday.

I am at Gate25C awaiting my connecting flight from Beijing to Wuhan.

Arrival from Houston at 2:40 to Beijing Airport and the subsequent five hour stopover has been exhilarating. Though I lived in Korea for two one-year tours during my military service, here, now in China is a foreign awakening, as so began…

Upon clearing customs, I walked out into the public lobby of the two-story terminal. Immediately, directly ahead, I spot an escalator with signs indicating taxis, hotels, and food. There I meet Wang Tai.

He sports a dark red hotel uniform and says he is a “hotel officer” of the Blue Sky Hotel. When I told him that I wanted a hotel room for a few hours to shower and change after my 16 hour trip from Houston, Texas, he said “OK” and to follow him.

“What kind of hotel do you want,” he asked? “Top, medium?”

I send a signal – “How much?”

We went to a motel some ten minutes from the airport. I got a room. We bought 5 cans of his favorite Chinese beer from a local store. We called Fen. I took a shower, changed, and then we headed back to the airport.

He and a hometown friend, who also works at the airport for Blue Sky, accompany me to a restaurant on the balcony, second level of the terminal. Having an hour until my connecting flight, I insist that I buy them a drink for their kindness, so we each enjoy a margarita and a Chinese beer. We smoked a couple of W’s friend’s Chinese cigarettes and listened to my self-produced CD mix of music on my Gateway notebook (made in China) – selfproduced, first my top country-western favorites then a sampling of my Miami-to-New York club mix.

On the plane now, awaiting take-off to Wuhan Airport. Only 22% battery remaining.

There, reflecting, I think how humorous… I am in the country for 30 minutes and have already had my first run-in with the traffic police – surprise, surprise…

W and I had embarked in a taxi for his company’s motel. The taxi driver exits the airport ramp. Suddenly, seconds into our ride, a police road block appears around the bend at the opening of the terminal’s underground exit ramp.

Our driver slows almost to a dead stop, only to then accelerate in an attempt to pass the police. It appears that he is running their blockade.

A policeman jumps in front of us. Still our driver maintains his improbable evasion.

The officer then stomps his left foot onto the front bumper of our car. Four other officers immediate swoop over to surround the taxi.

We are stopped. The police are shouting at us, causing our driver to turn off his motor just before he is yanked out of the car.

Everyone is barking at an elevated pitch, everyone except my new friend W and me. It is at this point, sitting in the back seat with W riding shotgun, that I recognized a street-wise respect for my new acquaintance.

When I decided to come to China some months prior to arriving, I made a deal with myself for what I wanted to accomplish by living here. Two years since arriving, I can now say that I have gained far more from my original, self-dealing bargain. To the extent that markets may measured by time and money, from a personal standpoint, time has become the real windfall from my investment of both (time and money) here.

For I have developed a profound understanding as an America of the times in which my grandparents lived; they survived the Great Depression. My parents, Donald L. and Patricia E. Dimick, sacrificed and contributed during World War II as my grandparents had done this as well as during the “first” Great War.

I, as the product of these two generations, have had no such comparative, real world experience as to the suffering and struggle to be found in (when not directly presented with) this world of humanity. Prior to living in China, perhaps the only comparative instance when I sacrificed for a greater good (other than family) was four years of voluntary military service with two one-year tours in the Republic of Korea during the 1980’s.

Now with two years of teaching, consulting, and traveling in China, I may attest to at least having had a front-row seat to a post-game show of a critical phase of nation building, being social and economic reconstruction (i.e., post Cultural Revolution). Sure, I had witnessed the struggle of life and death during my two years stationed in Korea; a little different, though, as I was stationed with tens of thousand of fellow, allied troops.

No American military here. Only Red Army soldiers and police who may be seen practically everywhere… on street corners, in the stores, pictured on billboards and in the newspapers, even formationed before, during, and after certain television programs.

Prior to March 30, 2006, I had not lived in a communist country. Enough said…

Thus, what I have learned since my arrival is that China today may be partially characterized as a reflection of America’s industrialized yesteryears. Without entering here into a social-economic analysis (as such treatment requires its own book), I surmise that this parallel of time and distance between the two nations is that of a human phenomenon… what we may label as being the establishment of social order to achieve economic development.

As an American, as a political scientist with a juris doctorate, I find it all too easy to recite the litany of contemporary issues at hand which “we” cite in offense with the emergence of China as a developing nation. Note: some of these charges so alleged are made not in contempt (but for historical context) of this centralized collage of provincial governments so managing some 56 ethnic minorities.

Does economic fracturing of social harmony (as some here in China and other developing and emerging nations so define it) appear as a natural progression of economic liberties?

How may state planners balance coordination of central policies with corporate leaders while promoting value-creation within market sectors reliant upon foreign (or domestic) investment and production enterprises?

For instance, the freedom to purchase an automobile: is it actually an economic liberty or a privilege such as an entitlement granted by the state? Moreover, may the privatization of a state-owned firm be the natural evolution of a socialist system of governance?

In either case, given the purchase or sale of an automobile or the financing and or investment in the transfer of a corporate entity, a central issue appears to focus on the nature and structure relative to both transactions?

Now some twenty-years as a student of investment banking, I posit that we arrive at the topic at hand… the Art of Conversion. “Sha-ma” (or my phonetic pinion word for the Chinese character meaning “What”) you may say?

I cannot provide Chinese students any greater insight into China’s requirements now placed upon itself to develop management techniques and styles beyond that already proffered by Jack Welsh, former CEO of General Electric, in his book titled Winning. Here Mr. Welsh frames and addresses such issues most convincingly, as might a corporate officer so worthy when confronting “the big picture” with each balancesheet-like item of any contemporary issue of the day.

Nor may I offer further assistance to the “louyui” (or pinion word for the Chinese character meaning “foreigner”) who comes to China seeking to invest in and profit from any allegedly “mutually beneficial relationship” so devised and constructed based on the schematics of this political economy. Authors to include George Zhibin Gu and Winston Ma provide both breadth and foresight into the biomechanics constituting China’s evolutionary world of investment banking.

What I attempt here is what my adopted Chinese brother, Brother Tu, so accomplished when he first invited me to his family home in the town of Jhan Sha just outside the city of Wuhan in Hubei Province upon meeting his mother, now my adopted mother, and his family — to include his wife, Ms. Wu, whom he met when he made a bank deposit with her as the bank’s window teller. A recognized central China entrepreneur and private lender, Brother Tu impressed me with a desire to convert his cache of financial and social success into an exchange process with me as a foreign guest in his country.

Since, Brother Tu and I have invested more than time into understanding each other. We have become vested in each others, own, individual, personal-belief systems. Moreover, by doing so, at least I have come to gain a better sense of my own (American) family’s history. So profiting, though, I thereby realized a loss of self, a sense of self that was rather ego centric as an American.

Henceforth, I have converted my thinking, which in turn causes me to now often hedge my actions and reactions. Whereas, before I was sometimes linear, sometimes circular in my vesting of thoughts relative to any cause and effect, I no longer assume or presume to know the nature and structure of a given transaction – to include the sale or purchase of an automobile or the stock transfer of ownership in a corporate entity.

Be it (a) conducting due diligence on a Chinese holding company seeking to target selective foreign markets or (b) considering potential transaction-operation alignment scenarios for a commercial investigation and security program, I seek to identify the threshold of a given investment or transaction.

What does that mean exactly?

For no particular reason of which I am aware, from time to time, I laugh to myself, reflecting upon those first minutes of my arrival here in China. That remembrance causes me to sometimes move, sometimes be motionless with the weighing of each contingency.

What was the critical decision making process underlying the taxi driver’s attempted running of the police blockade? Was it based on “guanxi” (or the pinion word for Chinese character meaning “relationship”) in that the driver recognized the police officer? Did the policeman fail to recognize the driver by virtue of the fact that the officer’s superiors were present? Maybe they had never met before?

Or was the driver hedging in that the fine for operating an unregistered taxi presented a greater cost of doing business than evading the actual charge itself upon citation? Perhaps visa-a-versa presuming arrest, as the police did detain our driver in the back of a “paddywagon” (or what I learned as the word meaning “police vehicle for the detainment of suspects”)?

And how did my new friend, the hotel valet, come to demonstrate a greater capacity for corporate leadership than some company officers and directors whom I have had the occasion to meet?

And what of me, the foreigner, who happily transferred to my “new friend” the sum of \$100 (US) – then valued in exchange of approximately 800 RMB, which is about what the average income (non taxable) that a construction laborer in Wuhan may receive. I viewed the exchange for a profitable learning experience, yet many of my Chinese and foreign friends still criticize me for committing such a subprime-like excess liquidization of Beijing hotel valets?

And, finally, there is the state… Of all the given hours of any of the given days, why then? Was airport security so armed with profiling criteria determining when a road blockade was most likely to help rid society of offending transport-service-sector personnel so violating applicable laws and regulations?

The threshold then? From this reflective due diligence of my first moments in China so resulting in one of my first commercial exchanges in country, I gained a glimpse of the levels of reciprocity operating within this society. Moreover, by actually engaging in such daily economics, I experienced a simplistic form of the art of conversion as practiced within the hospitality (and law enforcement) sectors of China’s economy.

There, at these moments in the life of a that process of conversion (from opportunity into value), may one discover when and how to hedge – as my hotel valet associate did so adeptly when talking us out of further implication warranting police involvement. Therewith may we witness the threshold of a deal.

In retrospect, … I was inbound,… the taxi driver was outbound,… the government took its share,… and my new friend, the hotel valet, whom directed me to the preferred vehicle for the ensuing transaction in the first place, whereby after I gladly paid — no, actually insisted that I pay him – more than eight-times the fee for which he asked, well… my new friend, “Big W,” the hotel valet, now he would make a fine investment banker, whose skillful transformation of opportunity into value as described herein provides a testament to the art of conversion.

dr

5. Dating A Taurus Man on February 19, 2009 6:35 am

yes, it’s cool, and useful for me