FAQs

Currently the past 30 days of CME Globex Channels 382 and 386 are immediately available, plus access to an extended archive of a 1+ years worth of data from AWS Glacier. Markets.AI maintains the PCAP data for all Globex Channels, and offers them upon request. Other exchange data is also available and can be decoded and transformed on a normalized basis. (Trademarks are held by their respective companies, and no direct affiliation exist beyond data provision).

Each packet includes multiple nanosecond timestamps. One embedded in the Ethernet protocol for the time the Ticker Plant received the packet, another in the Packet Header for the Sending Time from the Exchanges Gateway, and one or more Transact Time stamps for when the Exchanges Order Book and Matching Engine processes the message (a single packet can have multiple transactions). The IPCAP csv files use the SndTime only, as it is the first time the data is available publicly. The other two times are really only valuable to specialized latency studies, and can always be decoded from the raw pcaps if needed.

Before you can access the IPCAP files, you must license the data directly from the exchange, or from an authorized data provider. If you have one already, we can verify that you have licensed access. If not, we can facilitate the PCAP license for you. We don't own the data, the exchanges do.

Markets.AI believes in analytic transparency, and the source code of the MAILIB, while not open source, is open to all subscribers of the IPCAP file service. The proprietary part of the Markets.AI system is in the decoding and transformation overnight processing. That said, the Exchange PCAP files can always be spot checked against the IPCAP csv file data to verify its accuracy. We are committed to the principle that the performance of capital at risk must be verifiable.

The IPCAP decoding and transformation process occurs overnight such a that all the csv files are available the following morning. It has not been created for real-time streaming of decoded messages as there are many services that do this already. It has been designed to solve an entirely different set of analytic processes that are virtually impossible (currently) to effectively do in real-time or near-real-time. The Exchange Volume of broadcast data rates exceed that of other feeds like Twitter on a consolidated channel basis. It is big data.

Commercially, the two processes can be defined as real-time algorithmically responsive decision processes, and algorithmically generative processes, similar to the application of neural network analysis. Training of neural networks can take hours or days depending on the datasets size and complexity, but the objective weighted function can be responsive to streaming input. There are only a few predictive techniques, like Hidden Markov Models (Bayesian), that can truly be updated in near real-time. The fact that the thesis for a given strategy is generated on a T+1 basis should be cause for little concern. The same cast of trading characters, both human and electronic, show up day-to-day or at their allotted periodic participation, and vary little in their regular order execution techniques from one day to the next. The insights from T+1 analytics only rarely change. What is important is that regime change on an order book involving changes in major market participants coming or going be detected so that executions can be suspended or modified, but the frequency of these types of changes are are almost exclusively from one daily session to the next, not intra-day.

Technically, the differences between real-time analytics and T+1 analytics are equivalent to the difference between On-Line Transaction Processing (OLTP) and On-Line Analytic Processing (OLAP). OLTP is focused on a very specific filtered stream of data, and OLAP analyzes a global set of data all at once. In network graph analysis, it is the difference between a depth-first-search and a breadth-first-search.

The two types of analyses serve different purposes, and like the objectives of tactical and strategic warfare, have different objective functions.

You do not need to. They are not your competition. HFT liquidity providers are market operators with a very different set of objectives than buy-side or large commercial firms, primarily targeted at the provision of passive trade liquidity and flow. Being aware of how they operate is healthy and worthy of investigation. Liquidity provision is a core focus of the Markets.AI paradigm as witnessed by the construction of the MAILIB Single-Order Graphics program.

Your real competition are the other traders that are trying to achieve similar types of price/size/fills that you are. The goal of the Markets.AI process is to do so for less slippage, in a shorter period of time, while revealing the minimum amount of information about your strategy. That is the specific goal of the Markets.AI analytics capabilities and organization of the underlying IPCAP csv data.