[bisq-network/bisq] Reduce initial request size (#4233)

[Florian Reimair]

<!-- 
- make yourself familiar with the CONTRIBUTING.md if you have not already (https://github.com/bisq-network/bisq/blob/master/CONTRIBUTING.md)
- make sure you follow our [coding style guidelines][https://github.com/bisq-network/style/issues)
- pick a descriptive title
- provide some meaningful PR description below
- create the PR
- in case you receive a "Change request" and/or a NACK, please react within 30 days. If not, we will close your PR and it can not be up for compensation.
- After addressing the change request, __please re-request a review!__ Otherwise we might miss your PR as we tend to only look at pull requests tagged with a "review required".
-->

Part of https://github.com/bisq-network/projects/issues/25

This PR reduces the initial request size to a O(1). By numbers, the number of historical keys to be sent to the seednodes is 1 now. With 1.3.4, that is 1 vs. 100k+ object keys and 2x1kB vs. 2x2500kB at startup.

## Details
This is a big step towards a more pleasant user experience and due to changes to parts of the data store implementation it is also high risk.
I see the split data store implementations as a temporary solution until I can proceed to https://github.com/bisq-network/projects/issues/29. Hence, even as there could be more tests, there aren't.

## Testing efforts
I created a number of test classes handling the basics of the migration and update scenarios, everyday use cases plus a few special cases. Additionally, there is a shell script running a kind of manual integration test which helped in finding more issues. The "manual" test takes care of setting up a testing testnet and through the steps upgrades pieces of software as well. Take a look at it if you so please. The results of the test have to manually observed although there is a `result.log` once finished which lists all the request and response sizes. (please not that some logs are put there twice because that is simply how it is).

Please give this a thorough test. If there are any questions, please feel free to contact me on keybase.

You can view, comment on, or merge this pull request online at:

  https://github.com/bisq-network/bisq/pull/4233

-- Commit Summary --

  * Initial tests
  * Version helper uses real Bisq version
  * Tests use static fixtures
  * Migration test succeeds
  * GetMap test succeeds
  * GetMapSinceFilter works
  * updateScenario works
  * freshInstall scenario works
  * put scenario works
  * Introduce SplitStore and Service to handle BL
  * Moved file handling logic to SplitStoreService
  * Activate other split data stores
  * Refactored tests
  * Initial tests for requests
  * Requests are smaller
  * Prevent adding duplicate data
  * Enforce version format and filter faulty ones
  * Test incoming special keys from the future
  * Javadoc cosmetics
  * Init integration test script
  * Persist live database after split
  * Integration test script tests stock
  * Use version history array
  * Use full sync between seednodes
  * Integration test script tests new system
  * Cleanup integration test script

-- File Changes --

    A PreferencesPayload (0)
    M common/src/main/java/bisq/common/app/Version.java (13)
    M core/src/main/java/bisq/core/account/sign/SignedWitnessStorageService.java (16)
    M core/src/main/java/bisq/core/account/sign/SignedWitnessStore.java (12)
    M core/src/main/java/bisq/core/account/witness/AccountAgeWitnessStorageService.java (22)
    M core/src/main/java/bisq/core/account/witness/AccountAgeWitnessStore.java (12)
    M core/src/main/java/bisq/core/trade/statistics/TradeStatistics2StorageService.java (13)
    M core/src/main/java/bisq/core/trade/statistics/TradeStatistics2Store.java (12)
    A core/src/test/java/bisq/core/network/p2p/FileDatabaseTest.java (318)
    A core/src/test/java/bisq/core/network/p2p/FileDatabaseTestUtils.java (145)
    A core/src/test/java/bisq/core/network/p2p/RequestDataTest.java (235)
    A core/src/test/resources/o1 (0)
    A core/src/test/resources/o1o2 (0)
    A core/src/test/resources/o2 (0)
    A core/src/test/resources/o2o3 (0)
    A core/src/test/resources/o3 (0)
    M p2p/src/main/java/bisq/network/p2p/storage/P2PDataStorage.java (50)
    M p2p/src/main/java/bisq/network/p2p/storage/persistence/AppendOnlyDataStoreService.java (12)
    M p2p/src/main/java/bisq/network/p2p/storage/persistence/MapStoreService.java (15)
    A p2p/src/main/java/bisq/network/p2p/storage/persistence/SplitStore.java (19)
    A p2p/src/main/java/bisq/network/p2p/storage/persistence/SplitStoreService.java (190)
    A reduce_initial_request_size_test.sh (165)

-- Patch Links --

https://github.com/bisq-network/bisq/pull/4233.patch
https://github.com/bisq-network/bisq/pull/4233.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/bisq-network/bisq/pull/4233
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bisq.network/pipermail/bisq-github/attachments/20200504/00d9e79c/attachment.html>