Oct

5

CME MDP Data, from Jim Sogi

October 5, 2015 |

The CME site says, "The CME Group Market Data Platform (MDP) disseminates event-based bid, ask, trade, and statistical data for CME Group markets and also provides recovery and support services for market data processing. MDP 3.0 includes the introduction of Simple Binary Encoding (SBE) and Event Driven Messaging to the CME Group Market Data Platform. Simple Binary Encoding (SBE) is based on simple primitive encoding, and is optimized for low bandwidth, low latency, and direct data access."

This sounds like a compression algo. Music is decoded into MP3, to lower bandwidth. I heard an interesting article on NPR about someone who analyzed what is missing from an MP3. Turns out quite of bit of interesting and valuable information that makes music more beautiful is dropped. They claim the full data can be extracted, but I'm concerned. It will cause a lag and delay. I use tick and execution data in my work. I am guessing there are players that can get full uncompressed data. I am concerned the compression will alter the true nature of the trades, their times, and other info. I use DTNIQ who says they can decode the packets to full data, but why do I doubt this? Who will benefit from this?

Anyone else have a take on this?

Jeff Sasmor writes: 

I was actually at a conference where the group that developed MPEG audio presented. Interesting place - Mohonk Mountain House in New Paltz NY; well worth checking out, it's iconic for a number of reasons. I think it was the IEEE ICASSP conference at the mountain house where I first heard about MPEG audio.

But I digress: MPEG audio is what's called perceptual encoding. The (basic) concept is that you can remove information that people won't hear anyway and they won't be able to tell that it's not there.

It's not the same as the sort of compression used when you zip a Word document. As you suggest, there is a delay due to decoding it, but it's short and generally not relevant for a typical use case.

Compressing a stream of 'ticks' is probably very different. There's a lot of repeated data time-domain-wise so you could use something like run-length encoding (which is sort of what IB does) although I'm sure it's more complex than that. As you suggest, one would want something that has low time latency in the decode process. Happily, it's much simpler to decode than encode for most types of data compression AFAIK. But there would be some delay. But it's most likely lossless.

Finally, I have read and find convincing the idea that an uncompressed data stream of tick data would overload the data transmission channel for any name that had any serious volume. If you're co located at the exchange or pay for a private fiber connection to the exchange it could be fine but not over the retail Internet. Even forgetting about the data rate you'd need to store it as it came in without overrunning buffers and the buffering itself would cause latency.

So if you have $$$$$$$$$ and an IT department you could make uncompressed work but probably not otherwise. But as the encoding of this sort of data is probably reliably lossless it may be less of an issue than you're thinking of.

My 2 cents.

Chris Tucker writes: 

If you are a person that enjoys nature, Mohonk has a very particular and endearing charm. Summer is nice for the family–swimming and hanging out at the lake, walking trails with amazing views–try the trail up to Smiley's Tower for incredible views of the Hudson River Valley. Autumn is spectacular as the trees change, cool nights on the veranda wrapped in a sweater and parked on an old rocking chair are delightful.

There is excellent golfing. But my favorite thing about the place is that it is ensconced in the heart of the Gunks. The Shawangunk Ridge is a focal point for serious rock climbers from near and far and I've spent some very memorable times there, including an utterly bizarre encounter with an Apache helicopter that must have been practicing popping up from behind a cliff. I could see my reflection in the pilots visor. Scary close.

If you enjoy mountain biking, I can HIGHLY recommend the nearby trails (relatively easy) at Lake Minnewaska. The views from Gertrude's Nose are breathtaking. Lake Minnewaska was home to two grand Catskill hotels until the seventies, but they both burned to the ground. See some interesting history here.


Comments

Name

Email

Website

Speak your mind

Archives

Resources & Links

Search