Rithmic latencies - HW,windows or rithmic? possible workaround

Market data, Trading, Technical support, Features requests, etc
dap
Posts: 14
Joined: Mon Mar 09, 2020 6:38 pm
Has thanked: 26 times
Been thanked: 2 times

Rithmic latencies - HW,windows or rithmic? possible workaround

Post by dap » Mon May 18, 2020 8:54 am

Hello I would like to share some latency issues I got the last week, I got lantencies up to 10/20/30 seconds in different times a day.
I think latency got worse when I installed R Trader Pro and bookmap connected via api. (I used bookmap conneted directly with rithmic, without the api, only for 1,2 weeks, so the tests cases are not a lot)

I think the problems are the poor windows cpu/threads management and r trader pro which uses 100% cpu sometimes.

1) Windows
I opened the windows task manager->performance and select the cpu/logical processors charts. 
From what I'm seeing, if you open programs, browser, everything etc windows put everything in cpu0 (only if the program can manage multithreading windows will use other cpuXXX)
I have a ryzen 1700x (8 core/16 logical processors/threads) and windows rarely move opened softwares beyond the firsts 8 logical processors (so I have the latest 8 threads always almost empty, doing nothing)
If your cpu0 (first logical processor) goes to 100% your PC will slow down (e.g. your mouse will slow down, network will slow down etc)

I found a workaround which limited the problem, you could set the affinity of the processes in task manager->details->right click on process and set the affinity. Here you can select for each opened software which logical cpus you want to use.
Moving bookmap and r trader in the latest cpus (from cpu4 to cpu15) is helping a bit, because windows seems put every new open programs into the firsts cpus (cpu0,1,2,3)

2) R trader pro
The other problem is R Trader pro which seems to do spikes of 100% cpu, in those times the latency will higher up.
This is the real problem for the latencies, every time the cpus (where I put rithmic via the affinity) go to 100% I get the latency (from some milliseconds up to 10,20,30 seconds [and bookmap saying "slow computer"]) 
Please note that R Trader uses only 2 logical processors (or maybe maximum 4, I don't remember), I've seen that from the cpu spikes on task manager. Attached a small example of task manager with 100% cpu and a small latency. 

Maybe a possible workaround is to use bookmap conneted to rithmic in the old way (without the API), so having R trader turned off avoiding these cpu spikes.
I didn't tested this method with task manager open


If I'm saying somenthing wrong please correct me, I'm just trying to avoid those latencies above 10 seconds


---
Just as reference I put my HW configuration, but I think it shouldn't be such a limit 
ryzen 1700x
32gb ram 3200 MHz
radeon 580x 4 GB
ssd
Internet connection 200Mbit, but connected via wifi
Attachments
taskmanager.JPG
taskmanager.JPG (80.37 KiB) Viewed 4244 times

Svyatoslav
Site Admin
Posts: 70
Joined: Mon Jun 11, 2018 11:44 am
Has thanked: 2 times
Been thanked: 7 times

Re: Rithmic latencies - HW,windows or rithmic? possible workaround

Post by Svyatoslav » Mon May 18, 2020 10:15 am

Thanks for the input. The part about CPU is very valuable to us.

I'd like to ask you to:
1) connect directly to make sure wifi has nothing to do with it (it most likely is unrelated, but that would be standard thing to try).
2) If you could do similar test with direct connection - that would also be very useful. I assume that it might happen that Bookmap will spike to 100% CPU on some cores if you do that, but regardless - it will be very helpful to know.

In the meantime we'll try to figure out what's going on....

Thanks,
Svyatoslav

dap
Posts: 14
Joined: Mon Mar 09, 2020 6:38 pm
Has thanked: 26 times
Been thanked: 2 times

Re: Rithmic latencies - HW,windows or rithmic? possible workaround

Post by dap » Mon May 18, 2020 11:13 am

Hello

1) sadly I cannot do it, I have a tower pc far from the router, cannot use a very long ethernet cable or move the PC near to the router. 
In the next days I will receive a wifi repeater, so (I hope) the wifi signal wouldn't be a problem [fritz repeater 310]

But this cpu "analisys" could be easily replicated by any PC, just opening the task manager and R Trader pro + Bookmap

2)Just switched to direct connection (to Frankfurt), I'll keep the task manager open to see if some cpu spike happens.
I set the Bookmap task manager->affinity to Cpu8 to Cpu15, so the spikes will probably be from bookmap alone

dap
Posts: 14
Joined: Mon Mar 09, 2020 6:38 pm
Has thanked: 26 times
Been thanked: 2 times

Re: Rithmic latencies - HW,windows or rithmic? possible workaround

Post by dap » Mon May 18, 2020 1:46 pm

Checked at 9:30 and 9:37 EST with direct rithmic connection. The cpus didn't go over 60%, one time I saw red text saying "data latency", but it disappeared in some milliseconds. (not like with r trader pro lasting some seconds)
To me seems related to R Trader Pro which uses a very high cpu

I also saw during zoom & vertical resize, some cpus at (or almost) to 100% but I think this is the normal bookmap behaviour. 
This is not a problem since it lasts some milliseconds

However If I see other strange 100% cpu peaks (not related to zoom/recenter etc) I will come back
 
Attachments
1537.JPG
1537.JPG (84.71 KiB) Viewed 4228 times
1530.JPG
1530.JPG (84.42 KiB) Viewed 4228 times

dap
Posts: 14
Joined: Mon Mar 09, 2020 6:38 pm
Has thanked: 26 times
Been thanked: 2 times

Re: Rithmic latencies - HW,windows or rithmic? possible workaround

Post by dap » Mon May 18, 2020 3:02 pm

The tests I attached previously were with only 1 bookmap window.
In this screenshot instead (16:42 CEST) I opened 4 separated bookmap windows. This is the only thing changed from previous screenshots, Don't know if it matters. I also tried to open 8 different bookmap windows and cpu didn't increase (more than the 4)

With 4 or 8 windows the cpu12 is stable at 80/85% and cpu13 is 60/65%, but no data latency so far. 

PS: It seems bookmap mainly uses 2 threads/local processors
Attachments
1642.JPG
1642.JPG (80.29 KiB) Viewed 4222 times

Svyatoslav
Site Admin
Posts: 70
Joined: Mon Jun 11, 2018 11:44 am
Has thanked: 2 times
Been thanked: 7 times

Re: Rithmic latencies - HW,windows or rithmic? possible workaround

Post by Svyatoslav » Tue May 19, 2020 10:01 am

Hi. I'm a bit concerned by https://www.bookmap.com/forum/download/ ... 01ab502794 screenshot. You say that R Trader is bound to cpu4 to cpu15, but for some reason it is executed on cores 14 and 15. But cores 14 and 15 belong to same physical core, unless I'm mistaken. Is it Windows scheduler doing that, or was affinity set somehow differently at that point? Please confirm that it was not a mistake and you actually had it assigned to 12 cores and not to 2.

Also, few things I'd like to clarify:
1. Which version of Windows are you running? Are all updates installed? (I recall that old versions of win 10 actually did have some scheduler issues)
2. I'm not sure if windows applies cpu microcode on boot (I know linux can do that, but not sure about Windows). Unless you are certain that it does - how old is the BIOS on your motherboard (I believe that's where another copy of cpu microcode is, so unless Windows applies it on boot - that's what your CPU will be running with)?

dap
Posts: 14
Joined: Mon Mar 09, 2020 6:38 pm
Has thanked: 26 times
Been thanked: 2 times

Re: Rithmic latencies - HW,windows or rithmic? possible workaround

Post by dap » Tue May 19, 2020 2:46 pm

Hi,
sorry I didn't explained very well, I meant I assigned "r trader" from cpu4 to cpu15, then windows choose the core/logical processors he likes, in this case only 14&15 because (I think) r trader uses only 2 logical processors.
Attached an "affinity example" on bookmap from cpu4 to 15

1. Latest version: windows 10 pro 1903 build 18362.778
2: Seems windows updates microcodes: https://techcommunity.microsoft.com/t5/ ... -p/1000845
However in case not, my bios is pretty old from May 2018 (AGESA code 1.0.0.2a) motherboard MSI B350 Tomahawk
Attachments
affinity.JPG
affinity.JPG (37.96 KiB) Viewed 4198 times

Svyatoslav
Site Admin
Posts: 70
Joined: Mon Jun 11, 2018 11:44 am
Has thanked: 2 times
Been thanked: 7 times

Re: Rithmic latencies - HW,windows or rithmic? possible workaround

Post by Svyatoslav » Tue May 19, 2020 3:01 pm

The part that is weird to me is why Windows decided to put 2 threads that consume 100% of cpu time onto 2 SMT threads (14&15) belonging to the same physical core (7) - I would expect scheduler to spread tasks around trying to keep those on different physical cores. But maybe I'm missing something...
However, could you please try to explicitly set R Trader to only even core numbers? So like let it run on threads 8,10,12 and 14 (but not on anything else), make sure Bookmap is not allowed to run on cores 8-15 (so it only runes on 0-7 and does not fight for resources with R Trader), and let's see if the problem continues. I want to make sure it isn't getting limited by single physical core.

About microcode - right, it makes sense for Windows to update it. Let's not dig into this option for now.

Another (pretty far fetched) guess is wifi driver. I know you said your PC is far from router, but if you maybe have a long cable and could try plugging it in just for a test - that would help. I don't really think that's the issue, but I did see wifi drivers doing weird things in the past (not 100% positive, but I think I've seen one causing system to almost freeze while data is transferred).

P.S. Our QA still trying to reproduce. No luck just yet, but we aren't out of options (will try more instruments).

dap
Posts: 14
Joined: Mon Mar 09, 2020 6:38 pm
Has thanked: 26 times
Been thanked: 2 times

Re: Rithmic latencies - HW,windows or rithmic? possible workaround

Post by dap » Tue May 19, 2020 5:20 pm

I have connected r trader pro with "even affinity" 8,10,12,14. Bookmap instead from 0 to 7
I'll keep it monitored and see if cpu goes again to 100%. Normally it will occour on high volatility periods like 9:30 ET, but also during the day

PS: I saw a Bruce webinar "Advanced 05-12-20" (at minute 9:07) he had a rithmic delay up to 13 seconds, this is very similar to what I experience when I have 2 logical cores at 100%

--

Instead, without r trader pro, connecting bookmap directly to rithmic without api, there aren't long delay. I see some core with 80% load but it's rare they reach 100% so they don't cause the longer delays like with r trader pro

--

I don't have a long enough cable (like 15+ meters), neither 2 long cable + network switch. 
Wireless driver up to date for windows, in tplink website are older from 2014 (TP-Link TL-WDN4800)

 
Svyatoslav wrote:
Tue May 19, 2020 3:01 pm

.. (not 100% positive, but I think I've seen one causing system to almost freeze while data is transferred).

Don't know if we are talking about the same thing. I found that the freeze during network data transfering are related to cpu, let me explain.
Sometimes when I download a sw, game, video streaming etc I had big slowdown, like mouse very slow, cracking music/audio, pc slowed down/almost freezed. Then I saw that the first cpu (cpu0) was always at 100%.
If I disable the "windows core parking" (an energy saver activated by default) and use the "affinity" to move the processes (and keep cpu0 lower than 100%), I get the PC at normal speed during data transfers, without slowdowns.
This is tested on 2 PCs (but both on wifi)
Don't know if wifi drivers could be the culprit, but to me seems more a bad process management by windows, which by default put everything on cpu0 and reaching 100% became very easy during network transfers. This is an hipothesys, please correct me if I'm wrong


----
----
----

3 test cases, all different behaviour


1) delay lasted about 10 seconds, the delay written in bookmap was about 1 to 2 seconds, but this red label took around 10 second to disappear from bookmap
Here rithmic used mainly 1 logical processor (10) then moved to another (12) (overlapped for a bit with 12)
rith_pari_10sec_1cpu.JPG

2) the delay written in bookmap in the red label was max 3,8 seconds( delay lasted about 4/5 seconds).
Uses only 1 logical cpu (10) 
rith_pari_4sec_1cpu.JPG

3) short delay lasted about 3 seconds. Uses 2 logical processors in different core (8 and 10)
rith_pari_3sec_2cpu.JPG

Tomorrow we'll see if I can get one of the biggest delay    
    
Attachments
rith_pari_10sec_1cpu.JPG
rith_pari_10sec_1cpu.JPG (81.85 KiB) Viewed 4176 times
rith_pari_4sec_1cpu.JPG
rith_pari_4sec_1cpu.JPG (80.21 KiB) Viewed 4176 times
rith_pari_3sec_2cpu.JPG
rith_pari_3sec_2cpu.JPG (78.09 KiB) Viewed 4176 times

mpxtreme
Posts: 30
Joined: Thu Oct 25, 2018 12:20 am
Been thanked: 6 times

Re: Rithmic latencies - HW,windows or rithmic? possible workaround

Post by mpxtreme » Wed May 20, 2020 4:30 am

Same issue for me.

Understandably with the recent extreme volatility in March/April,but even now as the market has somewhat returned to some semblance of normal liquidity there are frequent 1-5 second latency warnings.The R Pro has made it worse but it is to be noted that R Pro itself also has a "slow computer" alert that will occasionally trigger at the ES open and 4pm close, which can be expected.

But this has gotten worse since using Bookmap plugged in to R Pro.

I have a nearly identical setup as the original poster and was considering an upgrade.
Ryzen 1700x ( upgrade to 3700x?)
16g Ram
256 SSD
Radeon RX580  3x29" LG Ultrawide monitors
400mbit wired connection.

Most traders are using at least 4 core CPUs on the low end, if using an older system and 8+ cores in more recent computer builds.

I have mentioned and asked several times for multi-core/threaded capability, but no one listens.



 
Attachments
bmb.PNG
bmb.PNG (174.71 KiB) Viewed 4162 times
bma.PNG
bma.PNG (48.67 KiB) Viewed 4162 times

Post Reply