<h3>Motivation</h3>
<p>There was suspicion of a corruption bug inside FileManager. In order to unblock a review, I went through and audited the code and cleaned it up to make the concurrency more obvious.</p>
<p>At the end of the analysis, there was no corruption bug regarding the TODO in FileManager.java:75, but the interactions between the <code>persistable</code> field and the <code>savePending</code> were hard to understand. This has been updated with comments and an <code>AtomicReference</code> usage to make it more clear.</p>
<p>In some interleavings, two <code>writeToFile</code> calls for the same exact data could have occurred and this remains the case since references are passed in whose underlying data can change while <code>saveToFile</code> is running on another thread. That is why many implementations use a <code>Concurrent</code> data structure. Although, it is worth point out that not all do and it <em>could</em> be the source of some strange bugs.</p>
<h3>Analysis</h3>
<p>The comment was concerned with a situation where an in-progress <code>saveFileTask</code> would allow a UserThread <code>saveLater</code> call to schedule another write. There are two reasons this is OK here:</p>
<ol>
<li>Reference writes are atomic</li>
<li><code>saveToFile</code> is <code>synchronized</code></li>
</ol>
<p>In the event that an in-progress <code>saveLater</code> call spawned another <code>saveFileTask</code>, only one would be allowed to write the file at a time.</p>
<p>It is possible that the second call could write the file first, but since all callers share the same persistable reference, <strong><em>both</em></strong> <code>saveToFile</code> calls will write the latest data.</p>
<h3>Bug Fix</h3>
<p>There was one strange behavior that was fixed in this PR. If the first <code>saveLater</code> call had a large delay it would override any future <code>saveLater</code> delays until the original one was finished. This was because the first <code>saveLater</code> set <code>savePending</code> so all future <code>saveLater</code> calls returned early without scheduling a thread.</p>
<p>The update causes all requests to spawn a task so if the second <code>saveLater</code> call has a shorter delay, it will run and batch with the first call. The second task will finally get scheduled and it will immediately exit since there is no work to do.</p>
<h3>Future Work</h3>
<p>I think if the end goal is to have all writes completely thread-safe. You would need to do something like have <code>PersistableEnvelope</code> define a <code>clone()</code> function that all subclasses implement. The FileManager could then just clone the object prior to passing it off to the writing thread and have some guarantees.</p>
<p>That is way outside the scope of this work, but it may be a good piece for someone else to pick up.</p>
<hr>
<h4>You can view, comment on, or merge this pull request online at:</h4>
<p> <a href='https://github.com/bisq-network/bisq/pull/3690'>https://github.com/bisq-network/bisq/pull/3690</a></p>
<h4>Commit Summary</h4>
<ul>
<li>[PR COMMENTS] Make maxSequenceNumberBeforePurge final</li>
<li>[TESTS] Clean up 'Analyze Code' warnings</li>
<li>[REFACTOR] HashMapListener::onAdded/onRemoved</li>
<li>[REFACTOR] removeFromMapAndDataStore can operate on Collections</li>
<li>Change removeFromMapAndDataStore to signal listeners at the end in a batch</li>
<li>Update removeExpiredEntries to remove all items in a batch</li>
<li>ProposalService::onProtectedDataRemoved signals listeners once on batch removes</li>
<li>Remove HashmapChangedListener::onBatch operations</li>
<li>[TESTS] Regression test for #3629</li>
<li>[BUGFIX] Reconstruct HashMap using 32-byte key</li>
<li>[BUGFIX] Use 32-byte key in requestData path</li>
<li>[DEAD CODE] Remove getProtectedDataStoreMap</li>
<li>[TESTS] Allow tests to validate SequenceNumberMap write separately</li>
<li>Implement remove-before-add message sequence behavior</li>
<li>[TESTS] Allow remove() verification to be more flexible</li>
<li>Broadcast remove-before-add messages to P2P network</li>
<li>[TESTS] Clean up remove verification helpers</li>
<li>[BUGFIX] Fix duplicate sequence number use case (startup)</li>
<li>Clean up AtomicBoolean usage in FileManager</li>
<li>[DEADCODE] Clean up FileManager.java</li>
<li>[BUGFIX] Shorter delay values not taking precedence</li>
<li>[REFACTOR] Inline saveNowInternal</li>
</ul>
<h4>File Changes</h4>
<ul>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3690/files#diff-0">common/src/main/java/bisq/common/storage/FileManager.java</a>
(59)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3690/files#diff-1">core/src/main/java/bisq/core/alert/AlertManager.java</a>
(32)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3690/files#diff-2">core/src/main/java/bisq/core/dao/governance/proposal/ProposalListPresentation.java</a>
(49)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3690/files#diff-3">core/src/main/java/bisq/core/dao/governance/proposal/ProposalService.java</a>
(65)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3690/files#diff-4">core/src/main/java/bisq/core/dao/governance/proposal/storage/temp/TempProposalStore.java</a>
(2)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3690/files#diff-5">core/src/main/java/bisq/core/filter/FilterManager.java</a>
(31)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3690/files#diff-6">core/src/main/java/bisq/core/offer/OfferBookService.java</a>
(37)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3690/files#diff-7">core/src/main/java/bisq/core/support/dispute/agent/DisputeAgentManager.java</a>
(23)
</li>
<li>
<strong>A</strong>
<a href="https://github.com/bisq-network/bisq/pull/3690/files#diff-8">core/src/test/java/bisq/core/dao/governance/proposal/ProposalServiceP2PDataStorageListenerTest.java</a>
(127)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3690/files#diff-9">p2p/src/main/java/bisq/network/p2p/P2PModule.java</a>
(1)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3690/files#diff-10">p2p/src/main/java/bisq/network/p2p/P2PService.java</a>
(13)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3690/files#diff-11">p2p/src/main/java/bisq/network/p2p/peers/getdata/RequestDataHandler.java</a>
(2)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3690/files#diff-12">p2p/src/main/java/bisq/network/p2p/storage/HashMapChangedListener.java</a>
(14)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3690/files#diff-13">p2p/src/main/java/bisq/network/p2p/storage/P2PDataStorage.java</a>
(133)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3690/files#diff-14">p2p/src/test/java/bisq/network/p2p/storage/P2PDataStorageClientAPITest.java</a>
(8)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3690/files#diff-15">p2p/src/test/java/bisq/network/p2p/storage/P2PDataStoragePersistableNetworkPayloadTest.java</a>
(1)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3690/files#diff-16">p2p/src/test/java/bisq/network/p2p/storage/P2PDataStorageProtectedStorageEntryTest.java</a>
(120)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3690/files#diff-17">p2p/src/test/java/bisq/network/p2p/storage/P2PDataStorageRemoveExpiredTest.java</a>
(32)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3690/files#diff-18">p2p/src/test/java/bisq/network/p2p/storage/P2PDataStoreDisconnectTest.java</a>
(7)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3690/files#diff-19">p2p/src/test/java/bisq/network/p2p/storage/TestState.java</a>
(191)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/bisq-network/bisq/pull/3690/files#diff-20">p2p/src/test/java/bisq/network/p2p/storage/mocks/ProtectedStoragePayloadStub.java</a>
(2)
</li>
</ul>
<h4>Patch Links:</h4>
<ul>
<li><a href='https://github.com/bisq-network/bisq/pull/3690.patch'>https://github.com/bisq-network/bisq/pull/3690.patch</a></li>
<li><a href='https://github.com/bisq-network/bisq/pull/3690.diff'>https://github.com/bisq-network/bisq/pull/3690.diff</a></li>
</ul>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />You are receiving this because you are subscribed to this thread.<br />Reply to this email directly, <a href="https://github.com/bisq-network/bisq/pull/3690?email_source=notifications&email_token=AJFFTNRFT6KJTV6PCKJLJV3QVRPB5A5CNFSM4JRQIML2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H36YOJA">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/AJFFTNUU3N2PAHHQF43PU73QVRPB5ANCNFSM4JRQIMLQ">unsubscribe</a>.<img src="https://github.com/notifications/beacon/AJFFTNXJXHPDYDHL4PEBT3TQVRPB5A5CNFSM4JRQIML2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H36YOJA.gif" height="1" width="1" alt="" /></p>
<script type="application/ld+json">[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/bisq-network/bisq/pull/3690?email_source=notifications\u0026email_token=AJFFTNRFT6KJTV6PCKJLJV3QVRPB5A5CNFSM4JRQIML2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H36YOJA",
"url": "https://github.com/bisq-network/bisq/pull/3690?email_source=notifications\u0026email_token=AJFFTNRFT6KJTV6PCKJLJV3QVRPB5A5CNFSM4JRQIML2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H36YOJA",
"name": "View Pull Request"
},
"description": "View this Pull Request on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]</script>