PCAP Files are typically recorded from the On-Exchange Network Fabric in hourly files per exchange channel. These binary files are decoded and translated by Markets.AI backend server systems that result in the IPCAP csv files available for easy download from our AWS S3 Bucket.
This large prompt crude IPCAP file took 3 seconds to download from within a Markets.AI Toolbox notebook. Local performance is similar.
This is a payload from a single ethernet packet taken off the wire and decoded by a packet sniffer. To accommodate the extremely high volume of data streamed by the exchange in real-time, the simple binary encoding and the MDP specification define a streaming format with no duplication of data, and a self-referencing beginning and end of message encapsulation that implies related parts of the message by its internal and order and reference. This streaming entanglement makes the raw decoded file relatively useless to analytic processing.
An example of this is where a trade on the book takes Implied liquidity. The TradeID for the implied match will be for thje Implies book. For analytic purposes we want both the direct book's TradeID and the Implied Book's TradeID within a single record so that we can accurately source the liquidity for a given direct trade on the book.
Some purveyors of Tick Data do not use the actual recorded PCAPs, but instead download full replay files at the end of the trading session. These replay files are a subset of the actual PCAP files and contain incomplete information.
The raw message payloads are transformed into a normalized columnar format that for a given message, MBO, Quote, Trade, or Match element, data fills all those columns that are applicable.
Accept for the Index and last 6 columns, all other fields have direct Fix definitions or are part of the Packet Header (e.g. SndTime).
The added columns at the end of the list are to link the aggressor type, implied source refence and data, back to the direct record.
If you'd like a sample of the data, send us a note on the feedback page and we'll send it to you.
Copyright ©2020 SACIENT LLC - All Rights Reserved