<p>Starts <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/bisq-network/bisq/commit/62f6bac958b24fc0910728efa45b48f814f47697/hovercard" href="https://github.com/bisq-network/bisq/commit/62f6bac958b24fc0910728efa45b48f814f47697"><tt>62f6bac</tt></a></p>
<p>Submitting as a draft since it touches the <code>GetData</code> logic and it may be a good idea to let this bake on master if the code freeze is close. The only bug that was fixed was around truncating entries and it isn't needed in a hotfix.</p>
<p>But, the patch stack is only getting larger and earlier eyes on the changes will limit code churn later.</p>
<h2>Motivation</h2>
<p>The GetData synchronization sequence was currently untested and the code was tangled between the connection request handling logic and the <code>P2PDataStorage</code> update logic. In order to implement a new feature (<strong><em>replay removes during synchronization</em></strong>) I needed a way to verify current behavior and write tests for the changes I am making.</p>
<h2>Changes</h2>
<p>This PR moves the <code>GetDataRequest</code>-><code>GetDataResponse</code>-><code>ProcessDataResponse</code> logic into the<br>
P2PDataStorage where it can operate locally on the various data structures that need to be accessed<br>
and updated. This has a few benefits:</p>
<ul>
<li>Easier testing: tests don't need to mock <code>Future</code> and <code>Connection</code> to validate behavior</li>
<li>Public APIs can be removed that allow operations on internal data structures in favor of letting<br>
the P2PDataStorage handle the adds/removes/updates.</li>
<li>Optimizations can be handled internally to the <code>P2PDataStorage</code> limiting external knowledge of implementation details</li>
<li>Conceptually, the synchronization logic is all about synchronizing two separate <code>P2PDataStorage</code> instances so letting the actual objects build and apply the synchronization messages makes more intuitive sense.</li>
</ul>
<h2>Tests</h2>
<p>Full unit test for each individual stage now exists as well as an integration test that creates two separate <code>P2PDataStorage</code> instances, runs the synchronization logic, and verifies they are synchronized. This type of "multiple node" testing will be used to validate the new feature development and will limit errors in development.</p>
<h2>Maintainability fixes</h2>
<p>Since this patch tests the <code>GetData</code> messages sent back and forth, there was an opportunity to clean up the guarantees around null vs. empty w.r.t. sets and Capabilities. I think this is a good pattern moving forward to limit accidental NullPointerExceptions when we can do something less brittle.</p>
<hr>
<h4>You can view, comment on, or merge this pull request online at:</h4>
<p> <a href='https://github.com/bisq-network/bisq/pull/3667'>https://github.com/bisq-network/bisq/pull/3667</a></p>
<h4>Commit Summary</h4>
<ul>
<li>[PR COMMENTS] Make maxSequenceNumberBeforePurge final</li>
<li>[TESTS] Clean up 'Analyze Code' warnings</li>
<li>[REFACTOR] HashMapListener::onAdded/onRemoved</li>
<li>[REFACTOR] removeFromMapAndDataStore can operate on Collections</li>
<li>Change removeFromMapAndDataStore to signal listeners at the end in a batch</li>
<li>Update removeExpiredEntries to remove all items in a batch</li>
<li>ProposalService::onProtectedDataRemoved signals listeners once on batch removes</li>
<li>Remove HashmapChangedListener::onBatch operations</li>
<li>[TESTS] Regression test for #3629</li>
<li>[BUGFIX] Reconstruct HashMap using 32-byte key</li>
<li>[BUGFIX] Use 32-byte key in requestData path</li>
<li>[DEAD CODE] Remove getProtectedDataStoreMap</li>
<li>[TESTS] Allow tests to validate SequenceNumberMap write separately</li>
<li>Implement remove-before-add message sequence behavior</li>
<li>[TESTS] Allow remove() verification to be more flexible</li>
<li>Broadcast remove-before-add messages to P2P network</li>
<li>[TESTS] Clean up remove verification helpers</li>
<li>[TESTS] Introduce MapStoreServiceFake</li>
<li>Persist changes to ProtectedStorageEntrys</li>
<li>[DEADCODE] Remove protectedDataStoreListener</li>
<li>[DEADCODE] Remove unused methods in ProtectedDataStoreService</li>
<li>[BUGFIX] Fix duplicate sequence number use case (startup)</li>
<li>[TESTS] Add tests of requestData</li>
<li>[REFACTOR] Introduce buildGetDataRequest variants</li>
<li>[TESTS] Add tests of new RequestData APIs</li>
<li>[TESTS] Add tests of GetDataRequestHandler</li>
<li>[REFACTOR] Introduce buildGetDataResponse</li>
<li>[REFACTOR] Extract connectionInfo String</li>
<li>[REFACTOR] Extract getDataResponse logging</li>
<li>[REFACTOR] Extract truncation logging</li>
<li>[REFACTOR] Pass peerCapabilities into buildGetDataResponse</li>
<li>[TESTS] Unit tests of buildGetDataResponse</li>
<li>Remove redundant HashSet lookups in filter functions</li>
<li>[REFACTOR] Move required capabilities log</li>
<li>[REFACTOR] Inline capability check for ProtectedStorageEntries</li>
<li>[REFACTOR] Inline filtering functions</li>
<li>[REFACTOR] Remove duplication in filtering functions</li>
<li>[BUGFIX] Fix off-by-one in truncation logic</li>
<li>[TESTS] Add test of RequestDataHandler::onMessage</li>
<li>[REFACTOR] Introduce processGetDataResponse</li>
<li>[TESTS] Make verify() functions more flexible</li>
<li>[TESTS] Add unit tests for processGetDataResponse</li>
<li>Remove static from initialRequestApplied</li>
<li>[TESTS] Write synchronization integration tests</li>
<li>[REFACTOR] Clean up processGetDataResponse</li>
<li>[RENAME] LazyProcessedPayload to ProcessOncePersistableNetworkPayload</li>
<li>Remove @Nullable around persistableNetworkPayloadSet</li>
<li>Remove @Nullable around supportedCapabilities in GetDataResponse</li>
<li>Remove @Nullable around supportedCapabilities in PreliminaryGetDataRequest</li>
<li>[DEADCODE] Remove old request handler tests</li>
</ul>
<h4>File Changes</h4>
<ul>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-0">core/src/main/java/bisq/core/account/sign/SignedWitness.java</a>
(4)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-1">core/src/main/java/bisq/core/account/witness/AccountAgeWitness.java</a>
(4)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-2">core/src/main/java/bisq/core/alert/AlertManager.java</a>
(32)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-3">core/src/main/java/bisq/core/dao/governance/proposal/ProposalListPresentation.java</a>
(49)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-4">core/src/main/java/bisq/core/dao/governance/proposal/ProposalService.java</a>
(65)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-5">core/src/main/java/bisq/core/dao/governance/proposal/storage/temp/TempProposalPayload.java</a>
(4)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-6">core/src/main/java/bisq/core/dao/governance/proposal/storage/temp/TempProposalStore.java</a>
(2)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-7">core/src/main/java/bisq/core/filter/FilterManager.java</a>
(31)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-8">core/src/main/java/bisq/core/offer/OfferBookService.java</a>
(37)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-9">core/src/main/java/bisq/core/support/dispute/agent/DisputeAgentManager.java</a>
(23)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-10">core/src/main/java/bisq/core/trade/statistics/TradeStatistics.java</a>
(4)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-11">core/src/main/java/bisq/core/trade/statistics/TradeStatistics2.java</a>
(4)
</li>
<li>
<strong>A</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-12">core/src/test/java/bisq/core/dao/governance/proposal/ProposalServiceP2PDataStorageListenerTest.java</a>
(127)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-13">monitor/src/main/java/bisq/monitor/metric/P2PMarketStats.java</a>
(19)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-14">monitor/src/main/java/bisq/monitor/metric/P2PSeedNodeSnapshot.java</a>
(19)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-15">p2p/src/main/java/bisq/network/p2p/P2PModule.java</a>
(1)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-16">p2p/src/main/java/bisq/network/p2p/P2PService.java</a>
(13)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-17">p2p/src/main/java/bisq/network/p2p/peers/getdata/GetDataRequestHandler.java</a>
(122)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-18">p2p/src/main/java/bisq/network/p2p/peers/getdata/RequestDataHandler.java</a>
(92)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-19">p2p/src/main/java/bisq/network/p2p/peers/getdata/messages/GetDataResponse.java</a>
(35)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-20">p2p/src/main/java/bisq/network/p2p/peers/getdata/messages/PreliminaryGetDataRequest.java</a>
(21)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-21">p2p/src/main/java/bisq/network/p2p/storage/HashMapChangedListener.java</a>
(14)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-22">p2p/src/main/java/bisq/network/p2p/storage/P2PDataStorage.java</a>
(336)
</li>
<li>
<strong>R</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-23">p2p/src/main/java/bisq/network/p2p/storage/payload/ProcessOncePersistableNetworkPayload.java</a>
(7)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-24">p2p/src/main/java/bisq/network/p2p/storage/persistence/MapStoreService.java</a>
(5)
</li>
<li>
<strong>D</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-25">p2p/src/main/java/bisq/network/p2p/storage/persistence/ProtectedDataStoreListener.java</a>
(26)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-26">p2p/src/main/java/bisq/network/p2p/storage/persistence/ProtectedDataStoreService.java</a>
(16)
</li>
<li>
<strong>A</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-27">p2p/src/test/java/bisq/network/p2p/storage/P2PDataStorageBuildGetDataResponseTest.java</a>
(481)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-28">p2p/src/test/java/bisq/network/p2p/storage/P2PDataStorageClientAPITest.java</a>
(14)
</li>
<li>
<strong>A</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-29">p2p/src/test/java/bisq/network/p2p/storage/P2PDataStorageGetDataIntegrationTest.java</a>
(193)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-30">p2p/src/test/java/bisq/network/p2p/storage/P2PDataStoragePersistableNetworkPayloadTest.java</a>
(5)
</li>
<li>
<strong>A</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-31">p2p/src/test/java/bisq/network/p2p/storage/P2PDataStorageProcessGetDataResponse.java</a>
(246)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-32">p2p/src/test/java/bisq/network/p2p/storage/P2PDataStorageProtectedStorageEntryTest.java</a>
(128)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-33">p2p/src/test/java/bisq/network/p2p/storage/P2PDataStorageRemoveExpiredTest.java</a>
(32)
</li>
<li>
<strong>A</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-34">p2p/src/test/java/bisq/network/p2p/storage/P2PDataStorageRequestDataTest.java</a>
(175)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-35">p2p/src/test/java/bisq/network/p2p/storage/P2PDataStoreDisconnectTest.java</a>
(7)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-36">p2p/src/test/java/bisq/network/p2p/storage/TestState.java</a>
(259)
</li>
<li>
<strong>R</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-37">p2p/src/test/java/bisq/network/p2p/storage/mocks/MapStoreServiceFake.java</a>
(43)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-38">p2p/src/test/java/bisq/network/p2p/storage/mocks/PersistableNetworkPayloadStub.java</a>
(12)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3667/files#diff-39">p2p/src/test/java/bisq/network/p2p/storage/mocks/ProtectedStoragePayloadStub.java</a>
(2)
</li>
</ul>
<h4>Patch Links:</h4>
<ul>
<li><a href='https://github.com/bisq-network/bisq/pull/3667.patch'>https://github.com/bisq-network/bisq/pull/3667.patch</a></li>
<li><a href='https://github.com/bisq-network/bisq/pull/3667.diff'>https://github.com/bisq-network/bisq/pull/3667.diff</a></li>
</ul>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />You are receiving this because you are subscribed to this thread.<br />Reply to this email directly, <a href="https://github.com/bisq-network/bisq/pull/3667?email_source=notifications&email_token=AJFFTNVGPS2QS3EI4BLI7NDQVFUDXA5CNFSM4JQ3FUZKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H3SKX7Q">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/AJFFTNUS35OXJREDIDXM3KLQVFUDXANCNFSM4JQ3FUZA">unsubscribe</a>.<img src="https://github.com/notifications/beacon/AJFFTNWECNJ6VRABW5CJYOTQVFUDXA5CNFSM4JQ3FUZKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H3SKX7Q.gif" height="1" width="1" alt="" /></p>
<script type="application/ld+json">[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/bisq-network/bisq/pull/3667?email_source=notifications\u0026email_token=AJFFTNVGPS2QS3EI4BLI7NDQVFUDXA5CNFSM4JQ3FUZKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H3SKX7Q",
"url": "https://github.com/bisq-network/bisq/pull/3667?email_source=notifications\u0026email_token=AJFFTNVGPS2QS3EI4BLI7NDQVFUDXA5CNFSM4JQ3FUZKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H3SKX7Q",
"name": "View Pull Request"
},
"description": "View this Pull Request on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]</script>