<p>Part of <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="576403913" data-permission-text="Title is private" data-url="https://github.com/bisq-network/projects/issues/25" data-hovercard-type="issue" data-hovercard-url="/bisq-network/projects/issues/25/hovercard" href="https://github.com/bisq-network/projects/issues/25">bisq-network/projects#25</a></p>
<p>This PR reduces the initial request size to a O(1). By numbers, the number of historical keys to be sent to the seednodes is 1 now. With 1.3.4, that is 1 vs. 100k+ object keys and 2x1kB vs. 2x2500kB at startup.</p>
<h2>Details</h2>
<p>This is a big step towards a more pleasant user experience and due to changes to parts of the data store implementation it is also high risk.<br>
I see the split data store implementations as a temporary solution until I can proceed to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="595721729" data-permission-text="Title is private" data-url="https://github.com/bisq-network/projects/issues/29" data-hovercard-type="issue" data-hovercard-url="/bisq-network/projects/issues/29/hovercard" href="https://github.com/bisq-network/projects/issues/29">bisq-network/projects#29</a>. Hence, even as there could be more tests, there aren't.</p>
<h2>Testing efforts</h2>
<p>I created a number of test classes handling the basics of the migration and update scenarios, everyday use cases plus a few special cases. Additionally, there is a shell script running a kind of manual integration test which helped in finding more issues. The "manual" test takes care of setting up a testing testnet and through the steps upgrades pieces of software as well. Take a look at it if you so please. The results of the test have to manually observed although there is a <code>result.log</code> once finished which lists all the request and response sizes. (please not that some logs are put there twice because that is simply how it is).</p>
<p>Please give this a thorough test. If there are any questions, please feel free to contact me on keybase.</p>
<hr>
<h4>You can view, comment on, or merge this pull request online at:</h4>
<p> <a href='https://github.com/bisq-network/bisq/pull/4233'>https://github.com/bisq-network/bisq/pull/4233</a></p>
<h4>Commit Summary</h4>
<ul>
<li>Initial tests</li>
<li>Version helper uses real Bisq version</li>
<li>Tests use static fixtures</li>
<li>Migration test succeeds</li>
<li>GetMap test succeeds</li>
<li>GetMapSinceFilter works</li>
<li>updateScenario works</li>
<li>freshInstall scenario works</li>
<li>put scenario works</li>
<li>Introduce SplitStore and Service to handle BL</li>
<li>Moved file handling logic to SplitStoreService</li>
<li>Activate other split data stores</li>
<li>Refactored tests</li>
<li>Initial tests for requests</li>
<li>Requests are smaller</li>
<li>Prevent adding duplicate data</li>
<li>Enforce version format and filter faulty ones</li>
<li>Test incoming special keys from the future</li>
<li>Javadoc cosmetics</li>
<li>Init integration test script</li>
<li>Persist live database after split</li>
<li>Integration test script tests stock</li>
<li>Use version history array</li>
<li>Use full sync between seednodes</li>
<li>Integration test script tests new system</li>
<li>Cleanup integration test script</li>
</ul>
<h4>File Changes</h4>
<ul>
<li>
<strong>A</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-199edc7e572c6a53e9e7ead643e9dee0">PreferencesPayload</a>
(0)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-b50fd8d07cd41eecbb51bc6aacdaf6aa">common/src/main/java/bisq/common/app/Version.java</a>
(13)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-f54fd83e87ddb67b0e63f048b898f9e8">core/src/main/java/bisq/core/account/sign/SignedWitnessStorageService.java</a>
(16)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-d41c2b0482b3a816bdc6937013840693">core/src/main/java/bisq/core/account/sign/SignedWitnessStore.java</a>
(12)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-fee209e3a15872104ac8dce473f74835">core/src/main/java/bisq/core/account/witness/AccountAgeWitnessStorageService.java</a>
(22)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-c69601cb63d42ddee3aec5bea1d388fc">core/src/main/java/bisq/core/account/witness/AccountAgeWitnessStore.java</a>
(12)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-12ec2e36c5d1f9da8903a6e846b8eed9">core/src/main/java/bisq/core/trade/statistics/TradeStatistics2StorageService.java</a>
(13)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-f12e49a26a007a7123c25619a2830837">core/src/main/java/bisq/core/trade/statistics/TradeStatistics2Store.java</a>
(12)
</li>
<li>
<strong>A</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-f3381bb5252308c1d715549555065039">core/src/test/java/bisq/core/network/p2p/FileDatabaseTest.java</a>
(318)
</li>
<li>
<strong>A</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-74f59918f2d950c82935fbdde56f5062">core/src/test/java/bisq/core/network/p2p/FileDatabaseTestUtils.java</a>
(145)
</li>
<li>
<strong>A</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-b0c4050182d09d3ed00078d701a0f24b">core/src/test/java/bisq/core/network/p2p/RequestDataTest.java</a>
(235)
</li>
<li>
<strong>A</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-1fc82571de1d2e530c8934ecff697ae2">core/src/test/resources/o1</a>
(0)
</li>
<li>
<strong>A</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-ce4262b031db561fde374e827aca658f">core/src/test/resources/o1o2</a>
(0)
</li>
<li>
<strong>A</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-3ca4b464dbe2f248e876234db90cf611">core/src/test/resources/o2</a>
(0)
</li>
<li>
<strong>A</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-2bce3333d3247594e51f35e14fb3f045">core/src/test/resources/o2o3</a>
(0)
</li>
<li>
<strong>A</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-66594a760b3229a592efde4f86cb584b">core/src/test/resources/o3</a>
(0)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-926901286366061dc8575034045cb922">p2p/src/main/java/bisq/network/p2p/storage/P2PDataStorage.java</a>
(50)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-7e8e9d59ce010f6263fb6b7ed6682447">p2p/src/main/java/bisq/network/p2p/storage/persistence/AppendOnlyDataStoreService.java</a>
(12)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-27d3eec44fcbd8d9a7b864bbf7efaa5c">p2p/src/main/java/bisq/network/p2p/storage/persistence/MapStoreService.java</a>
(15)
</li>
<li>
<strong>A</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-b206bb1bff49bb5664b03ca230e0a8f0">p2p/src/main/java/bisq/network/p2p/storage/persistence/SplitStore.java</a>
(19)
</li>
<li>
<strong>A</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-38d5de477d38e425da3f6fe698d5ad34">p2p/src/main/java/bisq/network/p2p/storage/persistence/SplitStoreService.java</a>
(190)
</li>
<li>
<strong>A</strong>
<a href="https://github.com/bisq-network/bisq/pull/4233/files#diff-be9828a52db619fddde90b83784f94fc">reduce_initial_request_size_test.sh</a>
(165)
</li>
</ul>
<h4>Patch Links:</h4>
<ul>
<li><a href='https://github.com/bisq-network/bisq/pull/4233.patch'>https://github.com/bisq-network/bisq/pull/4233.patch</a></li>
<li><a href='https://github.com/bisq-network/bisq/pull/4233.diff'>https://github.com/bisq-network/bisq/pull/4233.diff</a></li>
</ul>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />You are receiving this because you are subscribed to this thread.<br />Reply to this email directly, <a href="https://github.com/bisq-network/bisq/pull/4233">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/AJFFTNVNFEEI453SQE2UBV3RP3QLRANCNFSM4MY33J4A">unsubscribe</a>.<img src="https://github.com/notifications/beacon/AJFFTNWSANCPWQIJNDM27TTRP3QLRA5CNFSM4MY33J4KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4JD2DWCA.gif" height="1" width="1" alt="" /></p>
<script type="application/ld+json">[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/bisq-network/bisq/pull/4233",
"url": "https://github.com/bisq-network/bisq/pull/4233",
"name": "View Pull Request"
},
"description": "View this Pull Request on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]</script>