Difference between revisions of "Market Mechanics"
m (→Timestamps in Bookmap)
(→Standard Matching Algorithm: Price-Time priority)
|Line 163:||Line 163:|
Here is an illustration of this
Here is an illustration of this
Revision as of 01:02, 4 February 2019
Market mechanics describe what are orders, the microstructure and the dynamics of order book / order flow inside exchanges (or trading venues). It shows how matching engines use various matching algorithms to process the orders, and how it is reflected in the market data that they generate. This article is a 'crash course' on Market Mechanics, brief, but intense. It doesn't require any background knowledge in trading and it doesn't assume a specific market, making it suitable for Futures, Stocks, Digital- Cryptocurrency, and so on.
- 1 Why should I read it?
- 2 Orders
- 3 Order Book
- 4 Market Data types
- 5 Commissions
- 6 Matching Algorithms
- 7.1 Why low latency is important
- 7.2 Atomic processing of orders
- 7.3 Study case: artificial 'speed bumps'
- 7.4 Latency vs decision quality
- 7.5 Most typical mistakes in Quantitative analysis in trading
- 7.6 Timestamps in Bookmap
- 7.7 Implementations
- 8 Feedback
- 9 Useful links
Why should I read it?
There are number of reasons why it's recommended to understand market mechanics before starting to trade or developing a trading strategy. Here are some of them.
The price is determined by orders of traders
Price, traded volume, and all thousands of market data based indicators are all 100% determined by the actions of traders and nothing else. Once an action reaches the exchange and assuming that it is valid, it is processed by matching engine using predefined, public, and deterministic matching algorithm. The result of this process may or may not lead to an execution, but it must always lead to an update of the order book. Consequently, exchange generates corresponding market data, still in some deterministic way based on what happened. Therefore, any market data based indicator including the price itself (in the form of best Bid/Ask, Last Trade or anything else) is determined by the actions of traders.
Price modeling vs orders modeling
Many trading strategies and studies are based on mathematical modeling of price behavior, including usage of random walk models. But since price is a direct function of orders, modeling the behavior of orders can be more fruitful, including modeling of a random market by using random orders which represent uninformed traders. This is because the same action, e.g. buy 1000 units at market price, can lead to different results depending on the current state of the order book, i.e. previously sent actions. As a bonus, such modeling of market allows greater flexibility while still being simple.
Wars do not affect the market
..., i.e. not directly. Any events, whether scheduled or unscheduled, anywhere on Earth or outside can affect only the decisions of traders, their actions, and as a result -- the market. This is why observing via the market data the actions of those who are the first to respond, such as Market Makers and high frequency traders (HFT) in general, is in a sense the fastest global news feed .
There are no 'macro' events in the market
... i.e. formally, any macro event is a cumulative effect of micro events - the orders of traders, which are processed by exchanges typically in microseconds. Long term investors may use monthly candlesticks to analyze the major price movements over the last decade. Understandably, they don't need to analyze terabytes of raw market data during that period. But it's useful to know that if they decide to process such data, they would calculate exactly the same monthly candlesticks. It's just that someone else already did it for them.
Put your money where your mouth is
This saying, although a little rude, has a basis. Market data is always a history. Traders watch it to make a prediction about the future. Even if they watch it to develop a trading strategy, the purpose is to be able to make better prediction in the future. That includes all types of traders from long term investors to HFT. But since market data is generated by activity of traders, observing it is in a sense, an attempt to understand what other traders think about the future.
It’s reasonable and supported by evidences that when people bet on future events with real money, their biased views and wishful thinking have lower impact on their decisions . Such betting systems provide a more accurate prediction than polls even for hardly predictable political events or the results of football games. Observing these risk-taking votes allow even non-participants to understand better what do participants think about the future and to filter out potential bias or wishful thinking.
The analogy with markets is straightforward: these risk-taking voters are traders who actually place orders (unlike, for instance, educators and commentators who don't). But in regards to trading, this method of truth extraction is much more powerful because unlike football fans, traders actually play and thus affect the price. Moreover, as shown above, traders are the only ones who affect it. That also includes manipulation and deception tactics by large traders. In fact, it's the reason why such tactics are being used -- exactly because they affect the market. But they affect it indirectly, through the actions of other traders who observe them. Unlike fake large orders, real orders affect the market directly.
Know the rules of the game
Trading is a multiplayer real-time game. Like in most of such games, there are winners and losers, but that competitiveness is what makes such games interesting. In trading, the rules are determined by exchanges in the form of matching algorithms and sometimes different latency priorities. These rules may favor certain types of market participants such as registered Market Makers by offering them higher matching priorities or even by adding artificial speed bumps for other traders (scroll down for Latency). But these rules can be considered fair simply because they are public. Traders who know the rules of the game can adjust their trading strategy accordingly, or walk away to a markets with better rules, or decide not to trade at all.
Factors for success in trading
There are number of factors that make trading more successful, such as:
- Having an access to high quality data
- Ability to convert available data into information and knowledge using strong infrastructure, analytical tools, and high quality visualization
- Wisdom to develop effective tactics based on knowledge, the ability to predict future events (including one's own impact), and to manage the risks
- Speed of data processing and execution of actions, i.e. latency and infrastructure
- Brute force which can simply move the market in a desired direction
Together, these factors give even stronger competitive advantage than a sum of them because they complement each other and depend on each other. For instance, fast execution of a wrong prediction of price direction leads to a worse execution price than slow execution of the same decision.
Orders are the most basic elements in trading, but at the same time, as shown above, the only elements that affect the market. An order is a set of parameters defined by traders.
Market order is the the most basic type of orders and has just two parameters:
- Buy or Sell
All other types of orders inherit the definition of Market order and extend it with additional parameters.
Limit order, the most frequently used type of orders, adds the price parameter:
- Price (this instructs the exchange that the order must not be executed at a worse price than specified)
Iceberg order adds another parameter:
- Maximum displayed size (The tip of the iceberg, instructs the exchange not to display more than specified of the total order size)
Stop order is a conditional order. It adds a condition parameter:
- Stop price (instructs the exchange to publish/release the order only if a trade occurs at equal or worse than specified price)
There are many other types of either instant or conditional orders which use additional parameters including time-in-force (TiF) that tells the exchange when the order can be published/released, chains of orders, and so on. Exchanges may offer to traders additional types of orders with any level of customization. Intermediate parties such as brokers and order managements systems (OMS) may offer even more flexibility while managing the orders in the exchange co-located facilities which allows sub-millisecond response time.
Actions of traders
The set of actions available to traders didn't change significantly over centuries and probably millenniums since the first sort of trading venue was established. What has changed significantly is the speed in which these actions can be delivered to the exchange and the speed at which exchanges process them and provide corresponding feedback. Also, market data has changed significantly, mainly because of improved technology & bandwidth. Market data today allow much greater market transparency.
The exact list of actions that available to traders and their parameters is defined by exchanges, but typically there are just two or three of them. Assuming regular limit orders, it looks like this:
|1||Send a new order||Buy or Sell, Quantity, Limit Price||Order ID|
|2||Cancel an order||Order ID|
Actions can be rejected due to insufficient funds, invalid parameters, or if referenced order doesn't exist anymore (canceled or executed), etc. Additional responses of exchange include updates of orders' status such as partial or full execution, cancellation confirmation, and so on. Also, most exchanges but not all (e.g. not supported by GDAX exchange) allow order modification as following:
|3||Modify an order||Order ID, New quantity, New limit price|
Order modification is in a sense a redundant action because it can be replaced by cancelling the order and sending a new one. But modify action allows to reduce the latency because both of its sub-actions are performed instantly in the exchange, and allows to avoid the risk of undesired exposure due to the in-flight condition: for instance if trader sends a pair of actions cancel/send, when they reach the exchange the order may already be executed, but the new order can still be accepted. Also, if the requested modification only reduces the order size within the same price, the order keeps its position in the orders queue.
Order book is a collection of orders of traders, constructed by matching engines of exchanges. It consists of
- Visible in market data collection of limit buy and sell orders (the order book). These orders are resting in the order book because their limit price didn't permit yet their match/execution
- Invisible in market data conditional orders. These orders have a certain trigger or condition to be published/released by the exchange and to become visible.
The collection of orders at the same price level is called order queue. Orders can advance in the queue when other orders in front of them are canceled or executed/matched with new arriving orderes. The arrangement of orders in the queue and the priority of their execution is determined by matching algorithm as will be show further.
Market Data types
The order book changes only when traders conduct new actions or if conditional orders released according to their time-in-force settings. Consequently, exchanges generate market data and inform traders about what has changed. Today there are two main distinct types of market data:
Market by Price
Market-by-Price (MBP) describes a price-based data feed that provides the ability to view the total size of all orders at each price level. MBP can describe either full market depth or a limited number of nearest Bid and Ask price levels, e.g. 5, 10, or 20 price levels.
Market by Order
Market-by-Order (MBO) or Order-by-Order describes an order-based data feed that provides the ability to view individual orders and their evolution. The order's information contains its unique Order ID, limit price, size, and its location in the queue. MBO typically provides full market depth, describing orders at each price level. This is the more modern and more transparent type of market data.
Other types of market data
Candlestick charts are built by aggregation of initially limited market data of last trades only (Times & Sales) which consist around 5% of the market data (or information).
Comparison of quality of market data sources
Did you know? Bookmap allows users themselves to compare market data of 2 or more different data vendors. The time range can be either hours or milliseconds. Here is an example: 
Commissions that exchanges charge for matching and execution between orders is the main source of their income. They aim to select a matching algorithm that attracts more traders and increases the liquidity of a particular market or asset.
Market maker vs Market taker
Each trade always occurs between two orders from opposite sides: Buy and Sell. one of the orders must always be a resting limit order while another is one is the newly arrived order, sometimes called aggressive order. The aggressive order is also called Market Taker while the resting order is called Market Maker, hence the name of corresponding trading strategy called Market Making .
In order to increase liquidity and attract more traders, exchanges may define various commissions structures such as lower (up to negative) commissions for the market makers, and higher commissions for the market takers. Both matching algorithm and commissions structure affect the trading strategy of traders and thus the behavior of a particular market. For instance, the commissions for XBTUSD at BitMEX is -0.0250% (negative) for market maker and 0.0750% for market taker, which is ~5 points of its price 6735. To execute an order as market taker, the trader needs to anticipate price movement of at least 5 points. As a result, it's noticeable that price typically moves in steps of 5 points or more:
Here is such a move in a higher resolution. The intensive attempts of traders to execute their orders as market maker and receive commissions instead of paying them are also noticeable.
The exact algorithm of processing new actions of traders is defined by Matching Algorithm which is a part of instrument/asset specification. There is a variety of matching algorithms, even within the same exchange. For instance, here is an overview of CME matching algorithms, followed by their detailed description. The most popular matching algorithm is FIFO, described in details below. The next most popular matching algorithm is Pro-Rata, which matches orders according to their size and thus allows execution of orders at the end of the queue as well. In general, exchanges select the matching algorithm which satisfies best the traders of a particular market, thus increasing liquidity of that product and traded volume, and thus the income from commissions.
Standard Matching Algorithm: Price-Time priority
Price-Time priority (aka FIFO: First-in-First-out) is probably the most widely used matching algorithm, for instance, it is used by CME for E-mini S&P 500 (ES) futures.
|Processing of a new order||
|Processing of a cancellation request:||
|Processing of a modification request:||
Here is an illustration of this process
As shown on the top diagram, latency comprises of three components:
- The speed of receiving market data
- The speed of processing market data taking into account trader's own status
- Round-trip-time (RTT) is the time period from sending an order (or cancellation or modification) until receiving corresponding feedback from the exchange. It comprises of two ways latency and exchange processing time. Traders typically need to get a status update about their previous action before conducting a new one. Otherwise they may encounter the risk of higher than desired exposure.
Why low latency is important
Traders aim to get advantage of short term trading opportunities such as arbitrage between correlated and dependent on each other markets, fundamental factors, behavior of other traders, and so on. There is large number of trading styles and motivations, well covered and structured by a zero-sum article from 1993  (see tables at the end). Low latency allows to get advantage of short term trading opportunities earlier than other traders who noticed the same opportunity. If executed earlier, the other traders will move the price in the desired direction by their actions, but may have worse execution price, and vice versa .
Atomic processing of orders
Traders send their actions asynchronously and may have different latency to the exchange due to geographical location or different infrastructure. These actions must be synchronized at the exchange because different order of processing of any two actions may lead to different results as shown on this simple example. Typically actions are synchronized according to the order of their arrival (even if two actions arrive at the same nanosecond, there is still the first one). In practice, exchanges may use a smarter approach, allowing to process actions in parallel (e.g. two orders being cancelled), but only if the result is the same as if they were processed in the atomic manner. As shown in the Latency section, exchanges may use artificial speed bumps and offer a highway for preferred market participants such as registered market makers. Still, there is an atomic synchronization that takes place at some level before Matching Engine, but it is done by a different logic.
Study case: artificial 'speed bumps'
Chicago Stock Exchange (CHX) was the first to propose artificial Speed Bumps which should artificially delay incoming traders' actions by 350 microseconds except of the actions of particular market makers who are allowed to use the highway. At first it was rejected as being discriminatory and because technically it can lead to the following situation: ...concern is that a delayed message might well be delayed for 351 microseconds, 500 microseconds, thousands of microseconds, or even until next Tuesday while the participant submitting that message waits for its message's disposition . But later the proposal was approved not only for CHX, but also for NYSE  and other stock exchanges including not only incoming messages, but also outgoing messages. One of the arguments for the approval was that exchanges already give an advantage for registered market makers in various forms including, for instance, determining the position on the orders queue based on FIFO with LMM Matching Algorithm which is an enhanced FIFO algorithm that allows for LMM allocations prior to the FIFO allocations . Artificial speed bumps in effect provide very similar advantage for preferred by the exchange market participants.
This study case alone demonstrates the importance of latency and the importance of knowing the rules of the game prior to participating in it.
Latency vs decision quality
Low latency amplifies both good and bad decisions, leading to better or worse results accordingly. For instance, a fast execution of a bad prediction of price direction leads to a worse execution price than slower execution of the same decision.
Some trading opportunities are very obvious. For instance, when a sharp move occurs at one of two highly correlative markets, when this information reaches traders, it almost certainly will affect their decisions about the other market and thus affect its price. In such scenarios the speed is the crucial (and sometimes the only) component of successful arbitrage.
Most typical mistakes in Quantitative analysis in trading
There are number of counter-intuitive mistakes that can be made during development and operation of an automated trading strategy. Here are just two of them, both are related to wrong assumptions about historical data during development of a strategy. The only 'treatment' in both cases is real trading or trading in an equally competitive and interactive environment.
Emulation of latency
Here the mistake is an assumption of a constant uniform latency. In fact, the latency (RTT or especially latency of market data) has typically very fat-tailed distribution. For instance the difference between exchange timestamps and timestamps measured by a co-located server may be 1 millisecond on average while only 0.01% of data updates have latency above 100 milliseconds. It would be a mistake to ignore these outliers in simulations. But it would be almost identically bad mistake to assume that these outliers appear randomly by, for instance, using historical fat-tailed distribution of latency and generating it randomly in simulations. These outliers are highly correlated with bursts of activity on the market and because matching engine becomes overloaded. In other words, latency has a conditional distribution depending on bursts of activity in the market, which are typically hard to predict in advance. These bursts of activity indicate that many data driven trading strategies simultaneously spot the same trading opportunity (or rush to cancel their orders), which means this market situation is pretty obvious to many of them. It's therefore expected for a reasonable pattern detection AI/ML algorithm to find the same opportunities in the historical data. Moreover, given a trade-off, it will sacrifice less obvious but potentially real patterns in favor of obvious but illusionary patterns. If during simulation it assumes the average latency, e.g. 1 ms, it will obviously demonstrate great results, which is an illusion. From the other hand, assuming much higher latency, e.g. 100 ms would significantly distort the simulation because the actual average latency is 1 ms. Also, there are always even farther outliers, e.g. 500 ms latency that occur in 0.0001% of the data.
This problem is equally relevant for much lower frequency trading strategies including those that use daily data samples, e.g daily candlesticks.
Emulation of own impact
Here the mistake is an assumption of negligible impact of simulated on historical data strategy on market data and on other market participants. The market data wouldn't be the same because trading actions that don't affect the market simply don't exist. The only question is how much would it change the market. Because trading is interactive process, even relatively small additional activity would impact the data and also trigger different reactions of other market participants, creating a cumulative effect much stronger than added activity. Such cumulative effect of tiny actions is similar to behavior of chaotic system, i.e. where infinitesimal differences in the starting conditions lead to drastically different results as the system evolves as shown on this animation.
Timestamps in Bookmap
Bookmap allows to zoom-in to nanoseconds, which is the resolution of its timestamps. This obviously isn't necessary for a chart trader. But whether you use millisecond, microseconds, or nanosecond, such timestamp requires 64-bit Integer. Even seconds timestamp will require 64-bit in several years from now. So, there is no any disadvantage of using nanoseconds timestamps in the application. The advantages however is that it is useful for HFT firms who use Bookmap Quant solution to visualize data with timestamps recorded at exchange co-located servers where some firms, for instance, Nanotick may use GPS clock with 100 nanoseconds precision.
By default, Bookmap displays timestamps of receiving the data and orders updates on the client side, using its computer clock. This is useful for traders regardless of their latency to the exchange because it shows where they could potentially act and where they actually acted. It takes into account potential packet loss during transmission or short-term disconnections, which also allows trader to be alerted in such cases. Displaying exchange timestamps during such cases or within high or non-uniform latency in general, would mean changing the history from trader perspective. This would not explain to the trader high slippage during correct decisions. Also, displaying the chart with data receiving timestamps guarantees that market data is stored in chronological order when connecting to low quality data vendors who may, for instance, provide trades, BBO, and market depth asynchronously.
It's easy to imagine that ancient traders were interested in faster ships and horses to reduce the latency. Today commercial firms invest hundreds of millions USD in layering cross-Atlantic cables under the ocean, reducing the latency by number of milliseconds and being almost solely purposed for (and afforded by) HFT firms. Another direction being developed and already offered is radio communication, which allows to communicate almost at the speed of light 0.99c while cable communication speed is ~0.8c. It's possible that HFT firms in future may use neutrino based communication because neutrinos can travel at the speed of light 1c trough the Earth's core instead of traveling around the Earth like radio waves do .
Bookmap team will appreciate your anonymous feedback here:
- Iceberg Orders Tracker
- Market Makers as a news feed
- CME matching algorithms
- How to compare the data from different market data vendors
- Practical use of Zero-Sum-Game principle in trading
- Styles and Motivations of traders and relationship between them
- What Is Market Making?
- Prediction quality by betting systems