Difference between revisions of "Bookmap Performance FAQ"
(→Bookmap takes long time to connect (or does not connect at all))
|Line 3:||Line 3:|
==Bookmap takes long time to connect (or does not connect at all)==
==Bookmap takes long time to connect (or does not connect at all)==
Sometimes there can be certain routing issues or certain datasources can experience difficulties. While solution is to wait for the datasource to fix the problem (or for some ISP to fix routing issues), you can often work around that by picking
Sometimes there can be certain routing issues or certain datasources can experience difficulties. While solution is to wait for the datasource to fix the problem (or for some ISP to fix routing issues), you can often work around that by picking connection point, if data source provides such option. E.g. for Rithmic, you can try selecting Europe or US even if it's on the other side of the globe, as that will force traffic to take another route and will avoid some routing problems or issues if those are not affecting all datacenters.
==Bookmap becomes slow after some time==
==Bookmap becomes slow after some time==
Revision as of 15:52, 15 August 2019
- 1 Bookmap takes a long time to connect (or does not connect at all)
- 2 Bookmap becomes slow after some time
- 3 Market data is lagging
- 4 Data lagging with NinjaTrader connection
- 5 Data is lagging not because of priority settings
- 6 Bookmap Cache folder and its size
- 7 Installing Bookmap on SSD drive
- 8 Graphic Video cards and GPU
- 9 Memory or RAM consumption of Bookmap
- 10 Expected CPU load
Bookmap takes a long time to connect (or does not connect at all)
Sometimes there can be certain routing issues or certain datasources can experience difficulties. While the solution is to wait for the datasource to fix the problem (or for some ISP to fix routing issues), you can often work around that by picking another connection point, if data source provides such option. E.g. for Rithmic, you can try selecting Europe or US even if it's on the other side of the globe, as that will force traffic to take another route and will avoid some routing problems or issues if those are not affecting all datacenters.
Bookmap becomes slow after some time
Start with removing all external indicators and restart bookmap. Some might be storing data within java heap and after some time garbage collection has to kick in too often. For indicator developers: it is a bad practice to store massive amounts of data in RAM. If you want to save a copy of all incoming data for further processing - make sure you are saving it on disk and not in RAM.
Market data is lagging
Take a look at how exactly data is lagging:
- if time line is moving consistently (matches your system clock) - it probably is an issue with receiving the data. Make sure you are connected to the nearest connection point (e.g. Rithmic provides you with a choice) and make sure your internet connection is good. If you are connected via wifi - try using a cable. As a last resort you can try unsubscribing from some instruments, but that's just a way to avoid the problem, not a way to solve it.
- if time line lags behind (i.e. time within bookmap is behind your system clock) and moves inconsistently (getting faster and slower) - problem lies within the bookmap process. First thing to try is removing all external indicators (and restarting bookmap afterwards). If issue continues to occur - contact bookmap support.
Data lagging with NinjaTrader connection
The problem I'm having is that ninjatrader lags pretty bad when there is significant movement in the markets. By lag I meant the NT dom lags (can't move or place orders, price doesn't update for 2-3 seconds) as well as NT charts not updating promptly. Bookmap seems to be running fine with no lag while this is happening. If I remove the bookmap indicators from the three charts in ninjatrader then the lag no longer happens.
Make sure that both Bookmap and NT are set with equal priority as shown below. It’s highly recommended to set both with Normal priority. Based on your description it’s likely that Bookmap has higher priority than NT. Bookmap itself never sets or changes those settings. The default priority is Normal, set by Windows OS. However it happened in the past that Bookmap users change this settings manually and then forgot about it. If this is not the case and you still have this problem, please contact us and tech support will help.
Data is lagging not because of priority settings
Bookmap is set to "Normal" priority. Ninjatrader is also set to Normal, I never changed these values. I am running Windows 10 btw. I still experience the lag. Again, this lag is usually apparent under heavier market activity. At this time I'm only running CL and ES on Bookmap.
Starting from Bookmap 7 and above it appears in the task manager as bookmap.exe instead of java.exe.
The next candidate for the problem source is your Antivirus (specifically host intrusion prevention system, sandbox) that may cause the lag. Try to disable your antivirus.
If this doesn't help, update NT7 to NT8. NT8 seems to run indicators in a separate thread, not in the UI (user interface) thread. So, even if communication between NT and Bookmap isn't fast enough to handle spikes in data activity, the UI of NT8 will not freeze. This at least would be a workaround.
In general, note that NT doesn't have a standard API exposed to other processes. It's intended only to indicators (NT scripts), running inside NT process. But because Bookmap is itself a standalone program, it cannot be rewritten as NT indicator. Instead, when you add a Bookmap indicator to NT, this indicator communicates with Bookmap through Windows interprocess communication protocol. While it's pretty fast and we've done a lot to optimize it, it's not as fast as direct connections as with the rest of platforms and API's that Bookmap supports. Also, this interprocess communication can be blocked or slowed down by Antiviruses as false positive detection. If the above didn't help, write to Bookmap support and request remote desktop support referencing this page.
Bookmap Cache folder and its size
I did notice that the C:\Bookmap\Cache\ folder grows to about 2.5 GB by end of the day. Not sure this is normal.
Yes, this is normal. Market data typically consists of 90-97% of market depth updates (order book) and the rest is trades data. Besides, this is one of the reasons why observing market depth is so important. But that also implies technological challenges. Most trading charts don’t need to handle market depth data and even the trades data is aggregated into candlesticks. Even if those charts display DOM, they need to handle only the current snapshot of market depth data.
Bookmap chart may contain millions of market data events. The technological challenges are to allow smooth real-time chart display like HD video experience and also to allow quick and seamless chart manipulation. This required fast access to the market data collected so far. Handling all this data in RAM is not an option due to the large amounts of the data which can reach many gigabytes before compression. To solve this Bookmap has developed a proprietary database and a data structure that effectively use both RAM and disk drive. The Bookmap\Cache folder contains the latter. It can easily grow to several gigabytes and it’s normal. The proprietary data structure is optimized for fastest access with both SSD and HDD, but SSD is obviously faster.
If you're willing to store your cache somewhere else (on another disc for example) you need to reinstall your Bookmap and select another place for installation. If you're willing to save all your settings you need to replace all contents of Bookmap folder (C:\Bookmap by default). Do not forget to save your workspace and open it in a new instance.
Installing Bookmap on SSD drive
There are two main folders which Bookmap creates during installation. Users can change the default location.
- Installation folder. By default: C:\Program Files\Bookmap\
- User folder. By default: C:\Bookmap
From the performance point of view it doesn't really matter on which drive Installation folder is. But if there is enough space on the SSD drive, it’s recommended to install the User folder on it because this is where market data is maintained by Bookmap and SSD is faster than HDD.
Graphic Video cards and GPU
GPU’s are Graphics Processing Unit - the core of graphic/video cards. Unlike CPU that are optimized for computations, GPU’s are optimized for quick and smooth rendering on display, thus reducing the load on CPU’s and RAM. Bookmap utilizes modern GPU technologies and uses OpenGL 3.0 interface. OpenGL 3.0 is available since 2008, so most computers are supporting it. Users with older graphic cards can disable it via Settings->GPU acceleration. Another feature is Settings->Enable smoothing. This allows smooth transition of chart display on sub-pixel level. It’s mostly noticeable on low resolution monitors.
Today many computers have both integrated and dedicated (typically more powerful) graphic cards. Users can let Bookmap use the more powerful graphic card as following:
- Right click on your desktop, and open NVIDIA Control Panel.
- Select “Manage 3D settings”, find Bookmap in the list of programs, and define which graphic card it should use.
Memory or RAM consumption of Bookmap
Bookmap maintains large amounts of market data. While most of the data is located in Bookmap/Cache folder, some part of it is managed in RAM for even quicker access. A typical memory consumption that appears in Windows’ Task Manager is 1.5 GB. That doesn’t mean that all this memory is being used by Bookmap, but this is how much is allocated for Bookmap. Technically Bookmap uses less memory, but reducing the allocation limit would make Garbage Collector (Bookmap is a Java program) to optimize memory size more frequently, thus slightly interrupting smooth operation of the program. Bookmap 7 (released on 16-Jan-2018) can automatically set maximum RAM allocation based on computer hardware. Users can adjust the settings manually via Settings->Memory allocation:
Expected CPU load
Typically Bookmap consumes 2-10% of CPU depending on computer hardware.
It’s highly recommended to keep hardware drivers updated or to make sure they are configured for automatic updates. This is especially important for newer hardware. Keep Windows OS up to date is also important. For instance recently Desktop Window Manager (dwm.exe) was causing a significant memory leak, but later it was fixed with one of Windows updates.
CPU consumption is expected to be higher during
- Chart manipulations such as shifting, zooming, adding indicators or changing their settings, etc.
- Starting from Bookmap 7 -- loading historical market depth data after subscription to an instrument.
- Loading data in Replay mode (see progress bar in Replay controls panel).
If you experience higher CPU load by Bookmap in other situations, please contact our support for assistance.
Additional factors that can increase CPU load:
- Not using GPU acceleration or old and low memory graphic card.
- Very high resolution displays.
- Number of Bookmap charts detached for a separate view.
- Number of subscribed instruments / tabs.
- High speed Replay mode.
- Increased chart refresh rate. Set larger refresh period (means lower FPS - frames per second) via Settings->Chart refresh rate.
- Zooming out. The more price levels are displayed on the chart and the longer period - the more computations are required.
- Number of indicators displayed on the chart.