[bisq-network/bisq] High CPU Consumed (#4370)

Sun Dec 13 17:35:47 CET 2020

I spent a lot of time in Feb 2020 profiling Bisq's RAM usage on Linux (Ubuntu), followed by some conclusions detailed [here](https://github.com/bisq-network/bisq/issues/3918#issuecomment-594236960).

Long story short, I found three types of problems, but no perfect solutions yet.

1.   Uncontrollable off heap memory use by Java / JavaFX based app.
Any Java/JFX based UI is a greedy beast, and if you don't configure the jvm's maximum RAM limit -- how much the jvm 'thinks' it has --  it may bring your machine to a halt as it tries to use more ram than your machine actually has.  This is not a generalization about all Java UIs, but does seem to be generally true for JavaFX on Linux, regardless of whether you use mesa or gpu drivers.  On the other hand, reducing your JVMs max ram limit too much risks an earlier Java app OOME than would normally happen.  This leaves me with having to choose between two options: which do I want to see run out of memory first, my Bisq instance or my computer?  
To control Bisq's off heap memory use, we decided to start Bisq on Linux and OSX with this jvm option:  `-XX:MaxRAM=4g"`.  That means Bisq's JVM normally uses about 25% of the maximum RAM limit, or about 1 Gb.  The upside is Bisq user's computers are not liable to crash because of Bisq, but it increases the risk Bisq may run out of memory, and throw an `OutOfMemoryException`.
See the **Setting JVM's MaxRAM** section of this comment for a few details.  

2.  A long chain sync at startup locks up the UI thread until it's done, gobbles off heap memory, and doesn't give it back to the OS.
This is unavoidable (so far).  I work around it like so: wait a few minutes (don't touch the UI) while my Bisq installation finishes the chain download (check log), then shut down and restart Bisq. The UI will become responsive more quickly after the restart.  

3.  There are numerous and serious memory leak problems in JFX 11, and Bisq is stuck with it until we can upgrade it with the migration to JDK 15 (in progress).   See the section below about **JavaFX Memory Leaks** .  

**Setting JVM's MaxRAM**

My OpenJDK 11 JVM is configured with a default MaxRAM setting of 128 Gb, while my machine has only 32 Gb.

To check yours...

```
java -XX:+PrintFlagsFinal -version | grep MaxRAM
 uint64_t MaxRAM                                   = 137438953472                           {pd product} {default}
    uintx MaxRAMFraction                           = 4                                         {product} {default}
   double MaxRAMPercentage                         = 25.000000                                 {product} {default}

```   

I show how to lowering the jvm's `MaxRAM` can affect memory usage [here](https://github.com/bisq-network/bisq/issues/3918#issuecomment-594236960). 

Today, the bisq-desktop startup script sets this value to 4Gb by including the setting on the java command:  `DEFAULT_JVM_OPTS="-XX:MaxRAM=4g"`

**JavaFX Memory Leaks**

One of the major sources of these problems is the JFX 11 dependency.  The upgrade to JDK 15 will allow us to also upgrade Bisq's JFX dependency, which I do hope will not be as leaky.  The newer JFX will have many leak related bug fixes -- and hopefully not too many new leak related bugs.

Pasted from an [earlier comment](https://github.com/bisq-network/bisq/pull/4048#issuecomment-625445260) on the subject.

> > Btw, this means Bisq (or any JavaFX app) could behave differently on different systems, depending on that system's video settings (graphic card hardware? graphics drivers? amount of shared RAM between video and system? etc -- don't know what could play a role, but smth to keep in mind when looking at performance).
> 
> JavaFX does behave differently with different GPUs. It does use native GPU drivers, and there have been many leak problems in JFX related to GPU and mesa drivers.
> 
> Here is a list of some [jfx leak related issues](https://bugs.openjdk.java.net/browse/JDK-8115293?jql=project%20%3D%20JDK%20AND%20component%20%3D%20javafx%20AND%20text%20~%20%22leak%22) -- old and new. You have to hunt around for those related to GPUs, and for issues about leak related bugs found and fixed after JFX 11 was released.
> 
> Issue [8188094](https://bugs.openjdk.java.net/browse/JDK-8188094) shows how to make JFX leak from a simple app running on Linux. The unresolved issue reports the problem in JDK/JFX 8/9, and I reproduced it running JFX 11 on Ubuntu + NVIDIA GPU in Feb. 2020, when I started working on this problem.
> 
> JFX 14 has some leak bugfixes, but Bisq is holding back on upgrading to OpenJDK 14 until the new [java packaging tool](https://openjdk.java.net/jeps/343) is released from incubation to GA.
> 
> I did a lot of profiling to find the leaks (maybe you saw some of my comments in issue [3918](https://github.com/bisq-network/bisq/issues/3918)) using JMC/JFR, JProfiler, async-profiler, bcc and bpftrace tools, under the assumption the problem was/is in Bisq and JFX. I'm sure there are leaks to find and fix, but the only firm conclusion my analysis supported was that Bisq's JVM was reserving far more resident and virtual memory at startup than the Linux JVM needs. (I could not find problems with Bisq's on-heap management.) The problem was more serious when it starts with an empty data dir, syncs the entire chain, and the JVM never gives back any memory (to the OS).

**Well, that sucks.**

Yes, it does.  If anyone can advise on how solve these difficult problems, do tell!  I would get right to work, but moving off Java is not an option in the near or medium term.  I'm not going to spend time here defending or bashing Java because I'm more worried about my nice, fast GPU's  native GPU driver that crashes my machine at random times, sometimes right after clicking a browser tab.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/bisq-network/bisq/issues/4370#issuecomment-744033597
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bisq.network/pipermail/bisq-github/attachments/20201213/55e0b017/attachment.htm>