Heatmap in Trading: How to Learn What Market Depth is Hiding?
Heatmap visualizes the evolution of market depth data and is the heart of Bookmap which literally stands for Order Book and Heat Map. In this article, we will describe the value provided by this type of visualization and the development process behind such application of heatmap. This will allow traders to get the best of its powers.
Here’s how it works. Bookmap takes the market depth numerical data, also known as order book, and visualizes it graphically in a colored heatmap using a predefined colormap selected on the toolbar. This mapping is applied to both the current state of the order book (on the right side of the thin vertical timeline) and on the historical part of the chart on the left. This allows traders to analyze the actions in a market depth of other traders, including the actions of algorithmic traders, HFT, and especially larger players who cause greater impact on the market.
Here the price bounced off large concentration of sellers (upper orange line) and moved down. However, it became hesitative when the number of buyers below started to grow. Later the price moved up again
In Bookmap there is no limitation on zoom. This allows traders to analyze the market on different time frames up to nanoseconds (although in practice it’s never necessary). Here is an example of a higher resolution view:
Green and red lines represent Best Bid and Best Ask accordingly.
Trades are shown as red and green Volume Dots depending on the aggressor side
What Bookmap is not
A typical mistake is to think of it as just another indicator off-the-shelf that tells traders whether to buy or sell or in other words — how to lose money quickly. This misunderstanding even caused some to write negative reviews because Bookmap “does not tell where to buy or to sell.” We believe that there are no holy grails in trading, at least they are not for sale.
What Bookmap is
There are 3 main stages of trading:
- Acquiring relevant information;
- Interpreting the information and decision making;
Bookmap helps with the first and the last parts. First, it provides the most objective representation of market data at the current moment and in recent history in an intuitive way. Then it gives the tools to execute trading decisions in a one-click manner on the chart. This lets traders to concentrate their skills on the essential part of the art of trading — interpreting the information and decision making.
The data on heatmap is not an indicator, but the market data itself. Actually, more than 90% of it. A good analogy for trading is real-time multiplayer games: a player can choose whether to watch the moves made by other players or not to watch. The heatmap displays true and non-aggregated market depth data with a precision of up to the pixel resolution of your monitor. Bookmap merely converts the numerical values of market depth data, which is the total size of order book at each price-time coordinate, into a color according to its colormap. Traders can develop an understanding of how markets work and based on this automate or semi-automate their trading strategies.
Market data visualization in the era of HFT and AI
Is it still relevant? We know that High-Frequency Trading (HFT) infrastructure allows executing orders with sub-millisecond RTT (round-trip-time). Machine Learning algorithms and advances in AI (artificial intelligence) development combined with higher computation power make it possible for both HFT and LFT to detect patterns in vast amounts of multidimensional market data. As chart day traders, we do not want to compete against any of these powers. So, what is the area in which a chart trader can still get a competitive advantage over machines? The answer is visualization. A quick proof of it is the existence of Captchas:
The reason captchas are being used is the human ability to spot complex patterns better than computer programs.
We can also observe a similar “proof” using Bookmap itself. Take a look at the image below. We can determine that it was generated by a single trader with a single glance and we can see how it affects the price. Those who are familiar with computer science and machine learning understand how challenging it would be to detect this with a computer program in real time. The pattern itself consists of just several order replacements, but this small snapshot of the chart contains many thousands of market data events. This is a noise that such program must filter out. In addition, the program must deal with the noise generated by this trader himself in a form of different time intervals and order size during order modifications.
The typical “stairs” pattern
Large traders are usually more influential on the market than small traders, even if both “groups” are defined in such a way that they produce similar volume. On one side, they can use their weight to move the market in the desirable direction. On the other hand, they have the challenge to execute or manage large orders due to liquidity which is relatively low for them. They try to keep their activity undetected or, in other words, not to show that this activity belongs to a single trader. One of the techniques to obscure their activity is to split large orders into smaller orders and place them one by one. To make it even more difficult for other robots, they add “noise” in a form of non-equal sizes of smaller orders and non-equal time periods between them. This is an effective method against other robots, but the human eye can spot it with a glance as shown below.
Notice the different time scale of the two charts
Here is another example which is even more complicated for computer vision but still easy for human sight:
This is unlikely to be a spontaneously created pattern by multiple traders
Why visualize market depth data?
We receive market data as a stream of updates that inform us directly or indirectly about events that occur in the market. This stream consists of just 2 types of updates:
- Market depth data such as order was placed or canceled or modified.
- Trades — when an “aggressive” order was executed (i.e. matched) against one or more resting orders in market depth.
A side note: in some modern data feeds, both types in some cases can be reported in a single update. For instance, “order executed” message reports a trade and also requires to update market depth affected by it. But for the sake of simplicity, we can treat them as two data updates.
The main conceptual difference between the two types is that trades events inform us about the past while market depth events inform about the presence and future. As long as market depth data isn’t updated, it spans into the future. In a sense, it’s the declaration of the traders’ intentions even if some of them intend to create an illusion of intentions to trick other traders.
From the quantitative perspective market data consists 90-97% of market depth data and the rest are trades. Moreover, on average, large and, therefore, influential traders execute more than half of their volume as resting limit orders rather than aggressive orders. There are several reasons for that:
- Lower up to negative cost of exchange commissions for orders executed as “market making” rather than “market taking.”
- Better quality of price prediction, connection to other markets and fast news feeds, combined with the ability to cancel or move orders when needed in a sub-millisecond time frame.
This makes market depth data even more important. Here is one of the simplest but more practical ways of getting a competitive advantage by observing it: Market Makers are the fastest source of news and volatility alerts. For free.
In a sense, market depth is the 3rd most important dimension of market data in addition to price and time dimensions. An obvious analogy of it is topography. There are 3 basic ways to visualize topography: Contour plot, Camera viewpoint (shades), and Heatmap as well as their combinations:
We find heatmap to be more suitable for market depth visualization because price and time are distinct dimensions, unlike X-Y in topography. However, in future, it can be combined with camera viewpoint approach, allowing to rotate the chart and see even more details like in VR.
How is heatmap generated from market depth data?
At each Price/Time coordinate market depth is a numerical value, representing the total number of shares or contracts in the Bid or Ask Order book. Our goal is to convert these numbers into colors using a predefined colormap. We will discuss the selection of colormap later. Here are examples of different colormaps:
We chose the left side to represent the lowest number, and the right side — the highest number. The lowest number in market depth data is zero. But here is a challenge: what is the highest size of order book? We know the data range in the currently visible chart and scale it into colors according to this range. However, we don’t know the size of new incoming data. What is more, the highest value may have been determined by disappearing part of the chart on the left side. The solution lies in adjusting the scaling frequently so that the actual range of order book values in current chart uses the full scale of the colormap.
Also, to address the “heavy-tailed” distribution of sizes in market depth we need the upper cutoff.
Imagine a situation where the range of market depth data is [0, 2500], plus a very short 1-pixel wide moment of size 5000 that could be caused by a flash order. Without upper cutoff, we would display size 5000 as the rightmost color and therefore will use only half of the colormap for the rest of the data. This would make it difficult to observe the activity in market depth. Using the topography analogy, it’s difficult to grasp hills and valleys in a desert when the Mount Everest is also on the map and defines its scaling. It becomes even more difficult if this mount frequently appears and disappears in the visible chart range. For instance, when zooming in/out by price or time, or when shifting the chart. The solution is to set the upper cutoff, which means that all the sizes above upper cutoff will be displayed with the same rightmost color.
This makes the scaling and, thus, heatmap colors more stable, not affected by very short-term extraordinary sizes. However, the drawback of this solution is that we do want to spot extraordinary sizes. This is the tradeoff that traders need to understand. Bookmap gives full flexibility to configure it. If you prefer the scaling to be done by the highest size in the order book on the account of potential instability of the scaling, set the slider to 100%.
Upper cutoff: Manual or Automatic?
If both upper and lower cutoffs set manually, the scaling will never change. If only upper cutoff set manually, the scaling may change, but not dramatically. Manual setting of upper cutoff is more suitable for traders who:
- Focus on trading particular instrument, for instance, ES;
- Regularly trade during the same hours;
- Know what order book sizes are considered small, large, and extraordinary large.
For them, the manual upper cutoff may allow developing a color memory — the ability to estimate order book size in particular place by mere looking at the colors on the heatmap.
Nevertheless, this is difficult to develop when observing charts of different instruments. Also, it would be troublesome to observe market depth during different hours and various levels of market activity. Using the topography analogy, if the upper cutoff was set using a desert, the topography of Everest would be unnoticeable, and vice versa.
Automatic upper cutoff assists with setting a percentile of the heatmap that is below/above the upper cutoff using a slider. For instance, if the percentile is set to 97%, then 3% of the observable chart will be displayed as the rightmost color on the colormap that corresponds to the highest sizes in the order book. This makes the market depth activity in the observed chart more noticeable, but it also implies that the numerical value of upper cutoff is not constant.
The natural lower cutoff for market depth is zero. However, traders usually consider low order book sizes less important. When lower cutoff is set above zero, the sizes below or equal it will be displayed with the same, leftmost color of the colormap. This helps to reduce the “noise” in low liquid areas, and also keeps the whole scale of the colormap for the rest of data range, which makes it more distinct.
Bookmap applies manual and automatic settings of lower cutoff in a similar manner as the upper cutoff.
Choosing the colormap
Choosing the right colormap for heatmap is the key for successful visualization. It highly depends on the nature of data on which the colormap is applied, and on the goal that we aim to achieve by the visualization. There is a number of challenges. The ability to display other color-hungry features on top of colored heatmap is just one of them.
For the optimal colormap in our research, we have developed and tested various approaches. We also examined techniques applied in other areas, including visualization in astronomy of non-visual electromagnetic spectrum.
This chapter describes the challenges and the solutions that we offer.
Why NOT use the different colormaps for Bid and Ask?
Traders used to see Bid and Ask (buyers and sellers) in different colors. By inertia, many of alternative chart tools attempt to apply different colormaps for Bid and Ask as shown below:
Examples of other heatmap charts that use different colormap for Bid and Ask
Although this approach is intuitive, it’s not useful for heatmap because it reduces the ability to compare visually between sizes of Bid and Ask order book, which is even more important than the ability to compare between different sizes of, for instance, Bid.
The Big Tradeoff
There are two main but mutually exclusive requirements for the colormap of market depth heatmap. For the purpose of objectivity, we dismiss the aesthetics factor.
1. Wide dynamic range
This means the ability to distinguish the colors, representing slightly different underlying data (the order book sizes). A typical example of colormap with wide dynamic range is a rainbow also known as JET:
2. Perceptual sequentiality
When looking at two different colors, it should be intuitively easy to determine which of the two represents a higher value of the underlying data. A typical example of such colormap is the old gray:
Other single color colormaps are also perfectly perceptually sequential. For example, a green color is known to be the best for this purpose due to evolutionary factors such as recognition of food sources and predators. But in Bookmap green is already taken for visualization of Bid.
The overall perceptual quality of the heatmap created with each colormap depends on the underlying data on the observed chart and, therefore, on the zoom level of the same data. The two next animations demonstrate why the requirements of that tradeoff are mutually exclusive.
Here the rainbow colormap is both intuitively sequential and allows to see more details…
… while here it’s not intuitive which color represents higher size in market depth
Because the rainbow colormap is not intuitively sequential, we would need to look constantly at the colormap bar as a reference to understand the values on the heatmap.
Interesting fact: Matlab, a scientific computing framework, has been using the rainbow as a default colormap for more than a decade, and only recently switched to a new colormap called “parula”:
According to Matlab, the main reason for this change is that “unless you have a colorbar or know the order of colors, it is difficult to tell which color represents a higher or lower value than another color.”
In Bookmap we made a research and took a different approach for a number of reasons. One of them is to allow rendering other chart components of different colors such as volume dots and make them distinguishable on heatmap background.
Color wheel arranges colors based on research of the perception of colors by human vision. According to it, for any chosen color the other color that provides maximum contrast to it is located on the opposite side of the wheel. The location of white color in the center is aligned with its physical property discovered in 1666 as being the combination of all colors:
Building the colormap
Our approach is to create a colormap by drawing a curve on the color wheel. It’s clear that the wheel already contains all kinds of single-color colormaps from any point on the edge to the center. But to pursue the requirement for a wider dynamic range we want to include more colors.
For the purpose of Bookmap, it was natural to start with dark/black color as a representation of low (zero) values of data and then to step into the color wheel. Due to the fact that green and red colors are already being used for Bid and Ask, we had a choice between stepping at the blue or yellow edge. Because yellow feels brighter, we have chosen blue. From that point, the transition was determined by the color wheel to the yellow via white.
Taking into the account the application of our heatmap, we then extended the colormap into orange. This last step allows distinguishing better between large and very large sizes in market depth. Here is the result of the entire process:
An approximate colormap curve on the color wheel
In addition, we created another colormap that is also available in Bookmap:
This was done simply by extending the curve of the previous colormap into red. It provides even greater ability to distinguish between sizes in market depth data, especially between large and very large sizes. Note, that with this colormap, in some cases you may notice the problem we described earlier in relation to the rainbow colormap. Switch into the blue-orange colormap if you feel that colors become too “noisy”.
Technological challenges of market depth visualization
Market data typically consists of 90-97% of market depth updates (order book) and the rest is trades data. This is why observing market depth is so important. But that also implies technological challenges. Note that most trading charts don’t need to handle market depth data and even the trades data is being aggregated into candlesticks. Even if those charts display DoM, they need to handle only the current snapshot of market depth. Bookmap 7 also provides recent historical market data until the moment of subscription to the real-time data, which requires integration of the historical part of the data with real-time data. Therefore, as far as we know, no other trading platform offers this.
Bookmap chart may contain millions of market data events. The technological challenges are to allow smooth real-time chart display like HD video experience and also to allow quick and seamless chart manipulation. This required fast access to the market data collected so far. Handling all this data in RAM is not an option due to the large amounts of the data which can reach many gigabytes before compression.
To solve this we have developed a proprietary database and a data structure that effectively use both RAM and disk drive. The Bookmap\Cache folder contains the latter. It can easily grow to several gigabytes and it’s normal. The proprietary data structure is optimized for fastest access with both SSD and HDD, but SSD is obviously faster. Therefore if there is enough space on your SSD drive, it’s recommended to put the User folder (by default C:\Bookmap\) on it because this is where the market data is maintained by Bookmap.
The advances in technology help not only HFT but also with visualization: more powerful GPU, faster multicore CPU, more memory, faster internet — all these components are necessary for the type of visualization that gives traders a competitive advantage. We use all of it in Bookmap while also planning our steps years ahead for the technology that will be available in the future.
Will visualization be relevant in future?
Computer vision, Artificial intelligence, and Machine Learning algorithms keep advancing. You may have noticed that captchas become more and more difficult to solve. It happens even faster than advancements in computers’ computation power and hardware which is essential for high-quality visualization. In Bookmap, we will always rely on the latest technologies in both hardware and software. In our view, application of visualization is not going away anytime soon. Moreover, it is likely that the next era of visualization will take place in Virtual reality and Augmented reality, giving traders even better perception of market activity while at the same time reducing the cost of infrastructure such as workspace and monitors.