Having multiple memory issues daily. Need help,

[dmavro]

Hey guys, 

For the past few weeks I’ve been having issues with Bookmap and as of last night I thought things might have been starting to work again. Whether it was due to one of the hundred things ive tried or if it just magically started on its own I really don’t know. But today I now have a completely different problem that confusing the hell out of me. I’m at the point where I just want to build a new rig for trading because im sick of the issues ive been having.

FYI- In beginning of year I raised my memory allocation for BM to just over 8000MB and closed 2 of my 5 subscriptions(now have only had ES,CL and YM for most of year) to try a cope with the issues i was experiencing. I originally thought it had something to do with the addons I developed and assumed I had coded them poorly and they needed to be recoded to run more efficiently but during my troubleshooting I decided to use my laptop one day instead and I was amazed at how well it handled running BM and tradestation with 2 external 4k monitors attached to it. It was like night and day difference between the desktop and the laptop.  

The 2 systems are as follows,,,
Desktop- i7-7820x, 32Gb of DDR4 2666mhz Ram, Nvme drive and 2x Visiontek Radeon HD7750 GPU’s with 6x AOC 27” 4k monitors and 1x ASUS 24” monitor all at recommended resolutions. Ive also been OC’ing CPU to 4.5ghz(all cores, all the time) since before I started using BM.

Laptop- i7-10750H, 32Gb of DDR4 2933mhz Ram, Nvme drive and GeForce GTX 1660 Ti 6 GB GDDR6 with 2 of the 4k monitors mentioned above.

Now to the issue… A few weeks ago I started waking up to memory error popup in BM and a few times BM would basically be unresponsive to the point where I would have to quit process in task manager. Each time I had to do that BM’s CPU use would be about 80% and memory use would be 1-2 gigs above my allocation setting in BM.  To try and work around the issue I increased my ram allocation in BM to 9216 and same thing would happen. I then went to 11,264 and same thing would happen. So I started rebooting in AM before open and I would watch BM’s memory use go higher throughout day. Sometimes faster then others which I assume has to do with how busy the day happens to be but I would always get above my allocated setting. After the memory would go above the allocated setting my CPU use would start climb next and BM would start to get sluggish(slow rendering after zooms, delay when switching tabs.) Eventually CPU use would get 70% or higher and id get memory error popup or just become unresponsive again.

I started playing with garbage allocation algo and switched it from G1GC to Serial and system was worse. Then I tried parallel and it seemed like my memory use would take longer to reach its peak but I would still have issue. I tried shutting off all my addons and same thing happened. Then I tried shutting off all BM addons(S&I’s, Absorption, Sweeps, LT pro) only and same thing happened. Even gave multithreading a try in settings which slowed the process down but still had issue. I tried with and without GPU acceleration and I tried everything with and without my Oveclock on.
I then checked for software update and upgraded to current Beta version and same things but that time G1GC seemed to be horrible and Serial setting worked better so I left it set to that. I tried 7.3 version and had same issues plus some new ones. Now system told me i couldnt use GPU acceleration(which i always had on) due to some OpenGL issue. So i reinstalled 7.2 and still couldnt use it. 

I got sick of everything right around here and did a clean install of windows to try and see if it was a driver issue and reinstalled 7.2 again and this time serial setting was giving me short random latency issues and whenever I zoomed I would get 6-7 second latencies which never occurred before. So I went back to G1GC and things worked better on that now after the clean install. However Ive still noticed some of my alerts stuttering(repeating prior parts of sentence over and over again) later in the day. I was able to turn GPU acceleration on again after the new install.

Today i woke up to Ram use at 6300mb and cpu at 5-10% but by 10:30 RAM was at 9400mb and CPU was 70%ish and i had to restart. Then around 12:30 i decided to roll ES to March contract but when i did i also moved it to its own window again and instantly had my other subscriptions in main BM window start to render everything wrong like this and new window looked like this. Even more confusing is I had to use 'Disable Direct 3D use for UI' for 1st time ever to fix it. 

But by end of day i was still running and I didnt get error till after 4pm close. My RAM use was 8592mb (about 400mb above the setting that’s allocated) and my CPU wasnt going thru the roof (it was between 20-30%). Depending on current market conditions I can get the error in 30 minutes, an hour or 3 hours. Again this is just with 3 subscriptions.
 
Could this possibly be some part or some driver in my PC just not playing nice with BM? Considering what ive tried and how well my laptop handled running everything during the day I cant think of anything else. What kills me is I run Tradestation all day also and have no issues with that at all.

This weekend I ran memtest86 on my RAM and Prime95 and Blender to look for instabilities with CPU and GPU and had no issues.
 
 The fact that my laptop ran so much smoother without any errors is making me think I should build a new PC and give up on trying to get to the bottom of this.

Any input would be appreciated!

Thanks,

Dean
 

[Svyatoslav]

Hi,
I'm sorry to hear that bookmap is behaving like that. Now, let's try to figure it out...

First of all - whenever it shows that memory popup - please chose "create a heap dump" option and send the dump to support. Zip it and upload to google drive or similar, but only share the link with those who have @bookmap.com emails upon request, since dump might contain sensitive data.

I'll also appreciate if you can provide log files. Just send to support, mention that it's about this forum post and ask support to ping Svyatoslav.

About the stuttering with SerialGC - that's actually expected, it is the most primitive GC that just stops the app. G1 is generally expected to perform the best as long as it has enough free heap space to work with, AFAIK.
About bookmap going ~1-2GiB above settings is also OK, even though a bit high. That's because those settings are for the size of java heap, and there is some native memory usage too.

I see two possibilities as the most likely, and logs+dump might help figuring it out:
- somehow java heap gets filled with something. Possibly due to memory leak, or maybe due to something else. From there everything makes sense - GC uses bunch of CPU trying to reclaim memory, popup appears warning you that bookmap ran out of heap. In this case heap dumps should help a lot. You can artificially give it more ram (like 16GiB) to make the dump easier to read - by the time the leak eats up 16GiB it should be more obvious what exactly the leak is, as it should ensure the dump has more of those "unnecessary" objects.
- bookmap somehow does not keep up. It could be due to various reasons, and logs should help with understanding that better. There are exotic ones, like CPU deciding to throttle down due to overheated VRMs or something of that sort on certain hardware, or more simple ones like an addon (though you said you unloaded all) or possibly a bug (but why does it only appear on one machine?). Anyway, we need to dig into this, and logs are a good place to start.

Best,
Svyatoslav

[dmavro]

Svyatoslav,

I actually wasnt able to create heap dumps 90% of the time i woke up to problem because the system was so overwhelmed by that time. I was able to save 4 or 5 throughout different steps of my troubleshooting process which i have zipped and are ready to go. I reinstalled 7.3 last night and rebooted PC at1:30AM and just got the low memory warning again so i just created a new heap dump now too. Which logs, just common?

Edit- Now that im attempting to use 7.3 again should i enable GC Logging in memory allocation settings? Also, what about Coredump on crash?

Thanks,
Dean

[dmavro]

I also just noticed that GPU acceleration was disabled in 7.3 after i installed. When i tried to enable it i get the OpenGL error again like last time but in 7.2 i was able to use the setting like i mentioned in original post. Does 7.3 possibly use a newer version of OpenGL that my video cards might not support?

Dean

[dmavro]

Svyatoslav,

I contacted support as requested and sent heap dumps. Forgot to send logs from earlier today when I originally got memory popup. Will send again in another email.

I also raised allocation to max setting like you mentioned and noticed something that i havent seen happen in past. After i staretd BM again with new settings i checked task manager maybe around 10-20 minutes later and saw BM Ram use just over 11,000 but then i looked again about 30 minutes later and it actually went lower to about 8k'ish. I then checked again after about the same amount of time and it went lower to just over 6k. Ive never seen this computer ever release Ram for BM like that before. It was short lived though, because i looked again about 20-30 minutes later and mem use was at 16,042GB and its still there over an hour later with no error. CPU is hanging round 11% and hasnt bumped up at all yet. I wasnt even playing around with BM during that time so nothing like zooming or anything else could be a factor like ive seen in past.
 
Dean

[Svyatoslav]

Hi,
Thank you very much for the dumps and logs. From those it looks like possibly SI-on-chart addon has a memory leak. We are investigating... In either case - this isn't a problem with your computer, it's a bug - we are working on it.

Until then you can try to unload that indicator or just keep memory allocation high and restart bookmap periodically.

In case you are wondering - the drop in memory usage described in your previous message is in fact normal - if Java has allocated significantly more RAM than it needs - it will give some back eventually. It's pretty "greedy" in that regard so it will often overallocate within specified limit and only release memory if it didn't need it for a long time. So by allocating 16GiB you've let it actually face a situation where JVM realized that it allocated too much. As mentioned above, you might want to keep it at 16GiB just so you can use bookmap until the bug is fixed - memory leak is still going to be there, it's just not going to manifest as quickly.

Svyatoslav
 

[Svyatoslav]

Ah, and also... You've mentioned that you had the same issue without any addons. If you have a memory dump from that - please send it too, I'll take a look. Cause the above does not explain this case.

[dmavro]

Thank you for the update. I will leave it at 16gb like you suggested. My system is using 86% total Ram as of the 9Am this morning so might not return the 64GB kit i bought for testing purposes and just throw it back in system. It has 3200mhz speed too which is a step up from my 2666mhz kit. 

Still confused about certain things that i had to do over past few days to get BM to run smoothly. Why did i all of a sudden have to Disable Direct3D use for UI for 1st time ever and why does GPU acceleration have to be disabled all of a sudden. Or r those RAM sensitive settings that i could maybe try and use again now that i raised memory allocation to 16GB?

Also, since i last had issue yesterday morning around 10:30Am which is when i decided to try 7.3 alpha again and when i increased allocation to 16GB i have only gotten 1 low memory warning (around 3:15PM.) I created heap dump and sent it last night so not sure if u got that one yet but i chose to continue instead of restarting only because BM was very responsive still and seemed to be fine. So im still running without a restart since yesterday when i restarted in AM. 

One other thing, does it make sense to you that i had to go down to 3 subscriptions since i started with BM all those months ago? I doubt there has been a memory leak in addon for that long. Or could memory leak have been there all along for me and only me due to some my system not playing nice with BM? Just wondering because i havent seen any other posts on forum from other BM users who are having issues. That keeps making me think there has to be something else going on here also besides the leak. 

Im going to try and add 1 or 2 more subs to BM and see what happens today.

Thanks again,

Dean

[Svyatoslav]

> My system is using 86% total Ram
You might be ok with 32GiB, but having more won't hurt. I've copied part of my reply from elsewhere relevant to this:

Bookmap stores data on disk (that big Cache folder that you see), but it does not use "normal" files for that. It uses MMF's, which is a way to tell OS "map this file to address space" (sort of to memory), and then work with it like with RAM. Now, OS will automatically dump it to disk and load it from disk when necessary. Nice part is that it also will use free RAM to store those files, but will automatically shift data to disk if other apps need more RAM (usually you'll see up to 90-95% RAM usage in "Performance" tab of task manager when that happens). This speeds recentering up, while not claiming the memory exclusively, as if we'd just store it in RAM.

So RAM usage in task manager on the "performance" tab should not be taken literally.

> Why did i all of a sudden have to Disable Direct3D use for UI for 1st time ever

The only reason to do that would be a problem with some video drivers. It normally does not affect anything, just tells Java to use/not use Direct3D for UI (alternatively it uses OpenGL, if available)

> why does GPU acceleration have to be disabled all of a sudden.

You should be ok enabling it back. I'm not sure why the check fails, might be a false-negative. Basically we perform a test if GPU acceleration can be used after every unespected termination.

> does it make sense to you that i had to go down to 3 subscriptions since i started with BM all those months ago?

Generally main reason to do so would not be memory but ensuring bookmap is able to keep up with the data. Depending on addons used, computer specs, and specific instruments you are watching, you might or might not be OK with more instruments. Also, current level of market volatility plays a big role - in volatile days having less instruments might help a lot, but when market is calm you might be ok with many instruments.
I'd say that if you are having memory issues with ~10 instruments while having 8+GiB allocated and are not running something extreme (e.g. setting BTCUSD pips to 0.01 is basically a guaranteed way to run into issues sooner or later, while something around 10 or 50 should be pretty safe) - it's worth further investigation. That being said, 10 instruments with MBO data could cause performance issues.

[dmavro]

One last thing, RAM is now 17,500mb and CPU use is starting to go up again. 20-35% now with occasional peaks up to 40%. I do see an occasional dip down to 12-17% but its basically running above 20% constantly now. If i zoom its goes to 30-45% for about a minute then it comes back down 20-30%. I did not add any new subscriptions yet like i mentioned in last post. Should i consider those normal operating thresholds and expect CPU to continue to raise to a certain point?