[bisq-network/bisq] Improve DAO state management (Issue #5783)

[chimp1984]

Here an update of my work on that:

I tried out a few different key/value DB solutions ([MapDB](https://github.com/jankotek/mapdb), [LevelDB](https://github.com/pcmind/leveldb), [LMDB](https://github.com/lmdbjava/lmdbjava), [Xodus](https://github.com/JetBrains/xodus), [Chronicle-Map](https://github.com/OpenHFT/Chronicle-Map)...). I did not get Xodus and Chronicle-Map fully working for my example case due APIs which did not fit well to my use cases. I also tried to use the standard logging framework by adding a custom logger but it would have required to write a custom rolling file adapter for our needs and parsing all byte array data to hex was not convenient as well (and would have been probably slow), so did not follow further on that approach.

LevelDB was my favorite candidate for a DB solution but after reading more about its concepts I realized that it does not fit really our use case and requirements as it would not provide determinisitic storage files. It also would add risky dependencies (from the not much used java lib) and would be an overkill for what we need. I tried it also in the PersistenceManager for all sotrage files, but that would lead to pretty bad performance as there are too many compactation cycles triggered by the kind of persistence we do (replacing data and not mainly appening data).

The goal to get off-heap by using memory mapped files (e..g by using MappedByteBuffer,...) would not have led to the desired result as we use Protobuf and that does not has zero-copy support (Capt'n Proto and FlatBuffers does). Also that would have been a large refactoring and not really sure if that would work out well. But its something to keep in mind for Misq/Bisq 2 as it could reduce a lot of memory footprint and performance by avoiding those copy operations and parsing as with Capt'n Proto. 

So after all that, I came to the conclusing its best to do only the minimal changes where it hurts most (BSQ Blocks) and use the simple file based model we use already in the PersistenceManager but split up the blocks in buckets.

Here are out requirements:
- Read only once at startup
- Write infrequent (each 20 blocks for snapshotting)
- Persist in a deterministic manner so we can ship files as resourced with a deterministic set of blocks
- Avoid manual management of the resource files as we require for the version based historical data stores
- Fast write and read, low memory costs
- If possible avoid adding new dependencies, specially of not from well known sources
- If possible avoid JNI based libraries as that might reduce capabilty with some hardware (Raspberry Pi, Apple M1,...)

Implemented solution:
- Split BSQ blocks in buckets of 1000 blocks (about 1 MB, about each week a new bucket) and store those inside a BsqBlocks directory. Buckets are named with the block range inside that bucket (BsqBlocks_584001-585000)
- Auto-migrating existing DaoStateStore to the new model by adding all blocks to the new data structure and then not storing blocks anymore inside the DaoState which is persisted in DaoStateStore. We keep the rest of the DaoStateStore untouched (blockHashes, DaoState). So only the blocks got extracted to its own persistence structure.
- At snapshots we keep only the blocks from the last persisted snapshot up to the most recent (usually 20 blocks as that is the snapshot interval). This reduces memory requirement for the snapshot from 135 MB to a few kb. The rest of the DaoState is about 18MB. The Hashes could be treated also according to the new model but as they are tiny its not worth it.

I made some performance comparision from the branch with the optimizations for the hashChain and that new implementation.

I tested with a fresh start syncing from resources and full-mode for daoStateMonitoring (otherwise we do not do snapshots during parsing).

Here are the results:

**Old version:**
READ:
Reading DaoStateStore about 3-4 sec

WRITE:
Writing the serialized DaoStateStore completed in 2-3 sec

SNAPSHOTS:
Parsing 5808 blocks took 2010.5 seconds (33.51 min.) / 346.16 ms in average / block 
GC reduced memory by 520 MB. Total memory before/after: 2.441 GB/1.934 GB. Took 709 ms. 
Total GC invocations: 242 / Total GC time 125.883 sec 

**New version:**
READ:
Reading and deserializing 135814 blocks took 1342 ms 
PersistenceManager: Reading DaoStateStore completed in 303 ms 
-> Total time for datState+blocks: about 1.5-2 sec

WRITE:
Persist (serialize+write) 806 blocks took 15 ms 
Writing the serialized DaoStateStore completed in 265 msec 
-> Total time for datState+blocks: about 200-300 ms

SNAPSHOTS:
Parsing 5808 blocks took 1575.13 seconds (26.25 min.) / 271.2 ms in average / block 
GC reduced memory by 667 MB. Total memory before/after: 2.348 GB/1.696 GB. Took 516 ms. 
Total GC invocations: 130 / Total GC time 60.478 sec 

So read performance is about half of the old verison, write performance about 10 times faster and total parsing is about 25% faster (hashing is the slow operation here) and there is about 50% work for GC (I am still in the process to optimize that further). Peak memory was also a bit lower in the new version.








-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/bisq-network/bisq/issues/5783#issuecomment-955764177
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bisq.network/pipermail/bisq-github/attachments/20211031/a4c16248/attachment.htm>