Hi, a work-mate and I use a common indicator that we have developed and occasionally we get different results from the indicator on my installation of bookmap vs. his installation of bookmap. We have tried on multiple versions of bookmap but still have the same behavior. In tracing the problem further, I found that the issue is always the same. Occasionally his dataset of TRADE messages received has 1 or more extra TRADE messages with a size of 0 (used to indicate the end of a particular trade). We find this happening when we are looking at orders that are split between being filled as aggressor and passive. When the extra TRADE records with 0 sizes come in, it causes the indicator to register different fill sizes. For example, a 500 contract order on my system may show 120+380=500 contracts, but on his system it will show 120+75+305=500 contracts. The same overall counts, but broken up differently.
He is located in Florida, and I am in Connecticut. We are assuming that we are connecting to 2 different Rithmic data servers, but it is odd that it always seems to be his system that gets the extra TRADE-0 records. I am attaching 2 files which demonstrate this behavior, 1.txt is from my machine, and 2.txt is from his machine. The files are just a dump of all the TRADE records received by the indicator. If you look for the order ID 648316288977, you will see 2 extra TRADE-0 records in his dump file (2.txt) that are not present in 1.txt, on line #92, and line #431. Otherwise the records received are the same across both systems.
Can you explain this behavior? Can it be addressed on your side? In looking at the extra TRADE-0 records, they don't look like they belong.
Rithmic Data different over two different machines
-
rickbarlow
- Posts: 43
- Joined: Mon Feb 08, 2021 5:37 pm
- Been thanked: 1 time
-
Svyatoslav
- Site Admin
- Posts: 278
- Joined: Mon Jun 11, 2018 11:44 am
- Has thanked: 2 times
- Been thanked: 31 times
Re: Rithmic Data different over two different machines
Hi,
Time is the key here. Bookmap assumes that the reconstructed trade ended and sets corresponding flag whenever it either sees a new trade with different aggressor or when 100ms pass since last event. Normally that works mostly fine, as unless there is some problem with the data you will rarely see more than 100ms delay between multiple updates that were part of the same atomic event on the exchange. That's a simple but pretty universal approach that works on most data sources without relying too heavily on particular details in how data is transmitted (you could rely on those and achieve better results, and you still can if you want - just ignore the flags and compute those yourself)
Theoretically there is some room for interpretation if aggressor was a limit order that fully consumed a level, then got moved back and then again moved to the same now-refilled level - that could probably fool our algorithm, if there were no other trades in between. It's very rare, but I thought I must mention it here for completeness.
But what we see here is much simpler: you receive data relatively fast, and entire batch of trades is received between 44.046280600 and 44.177005200, so within about 131ms. It's still not ideal as exchange likely sent it out nearly simultaneously, but I can't judge which part of the chain introduced this delay (could be anything, including your network connection, Rithmic infrastructure, R Trader in plugin mode, etc).
Your colleague is receiving the same data between 11.128681301 and 11.965412101, which is much longer interval - 837ms. And there are couple pauses that make bookmap think trade has ended. (btw, one of you has computer clock pretty far off, but I guess you noticed it). This is a sign of bad connection somewhere or slow hardware.
He can try other Rithmic gateway, might give better results. Also might want to check all the network equipment. Also, no wifi if possible - that's a known source of random delays.
As for ways to work around that - depending on your algorithm, you might want to prioritize reliable reconstruction over fast reconstruction. In that case - just reconstruct yourself and ignore markers. With Rithmic that's really easy, just watch the aggressor field. You can also watch MBP or MBO data updates to catch that theoretical case that I mentioned above, but unless you need really high reliability of detection (and unless you actually want that case to be handled differently) - you don't have to.
Time is the key here. Bookmap assumes that the reconstructed trade ended and sets corresponding flag whenever it either sees a new trade with different aggressor or when 100ms pass since last event. Normally that works mostly fine, as unless there is some problem with the data you will rarely see more than 100ms delay between multiple updates that were part of the same atomic event on the exchange. That's a simple but pretty universal approach that works on most data sources without relying too heavily on particular details in how data is transmitted (you could rely on those and achieve better results, and you still can if you want - just ignore the flags and compute those yourself)
Theoretically there is some room for interpretation if aggressor was a limit order that fully consumed a level, then got moved back and then again moved to the same now-refilled level - that could probably fool our algorithm, if there were no other trades in between. It's very rare, but I thought I must mention it here for completeness.
But what we see here is much simpler: you receive data relatively fast, and entire batch of trades is received between 44.046280600 and 44.177005200, so within about 131ms. It's still not ideal as exchange likely sent it out nearly simultaneously, but I can't judge which part of the chain introduced this delay (could be anything, including your network connection, Rithmic infrastructure, R Trader in plugin mode, etc).
Your colleague is receiving the same data between 11.128681301 and 11.965412101, which is much longer interval - 837ms. And there are couple pauses that make bookmap think trade has ended. (btw, one of you has computer clock pretty far off, but I guess you noticed it). This is a sign of bad connection somewhere or slow hardware.
He can try other Rithmic gateway, might give better results. Also might want to check all the network equipment. Also, no wifi if possible - that's a known source of random delays.
As for ways to work around that - depending on your algorithm, you might want to prioritize reliable reconstruction over fast reconstruction. In that case - just reconstruct yourself and ignore markers. With Rithmic that's really easy, just watch the aggressor field. You can also watch MBP or MBO data updates to catch that theoretical case that I mentioned above, but unless you need really high reliability of detection (and unless you actually want that case to be handled differently) - you don't have to.
-
rickbarlow
- Posts: 43
- Joined: Mon Feb 08, 2021 5:37 pm
- Been thanked: 1 time
Re: Rithmic Data different over two different machines
Thanks for the explanation. Makes sense. Is there a way to check which Rithmic gateway each of us is using so we can compare and set both to the same one? Maybe we can root cause the slowness via process of elimination of the gateway first.
-
Svyatoslav
- Site Admin
- Posts: 278
- Joined: Mon Jun 11, 2018 11:44 am
- Has thanked: 2 times
- Been thanked: 31 times
Re: Rithmic Data different over two different machines
In Rithmic connection settings you can select the gateway. Default is Chicago.