<p></p>
<p><b>@freimair</b> commented on this pull request.</p>

<hr>

<p>In <a href="https://github.com/bisq-network/bisq/pull/4233#discussion_r427842198">p2p/src/main/java/bisq/network/p2p/storage/persistence/SplitStoreService.java</a>:</p>
<pre style='color:#555'>> +                    e.printStackTrace();
+                }
+            } else {
+                log.debug(file + " file exists already.");
+            }
+        });
+
+        // split
+        // - get all
+        history = new HashMap<>();
+        store = readStore(getFileName());
+        resourceFiles.forEach(file -> {
+            SplitStore tmp = readStore(file.replace(postFix, ""));
+            history.put(file.replace(postFix, "").replace(getFileName(), "").replace("_", ""), tmp);
+            // - subtract all that is in resource files
+            store.getMap().keySet().removeAll(tmp.getMap().keySet());
</pre>
<blockquote>
<p>With this change I think we're making the resource files the final arbiter of what is the distributed data. Is that what we want? Even for full nodes that are always on.</p>
</blockquote>
<p>This seems to be the only way of introducing "checkpoints" so that we do not have to query all the data.</p>
<blockquote>
<p>It looks like it will just throw away its own local data on new versions though, that seems incorrect.</p>
</blockquote>
<p>It does not throw away any data. It only de-duplicates the data. If we have a data store "incoming" from resources, we remove the data that is contained in the resource store from our live data store.</p>
<p>An example:<br>
Given <em>ResourceStore={1,3}</em> and <em>LiveStore(t)={1,2,3,4,5}</em> and we enter the copy and split business logic, the outcome would be having the <em>ResourceStore</em> being copied to the working directory and the new <em>LiveStore(t+1)=LiveStore(t) \ ResourceStore = {2,4,5}</em>. Together, they hold {1,2,3,4,5}.</p>
<p>Bringing it all together:<br>
In terms of initial data request size, we can now ask for all data except <em>ResourceStore</em> simply by telling other that we have the "ResourceStore_1.3.4" -> sending {"1.3.4"} instead of {1,3}. And because in real live, the data stores count >100k objects, we save a whole lot of data. Additionally, we actually ask for all data "since" (instead of "except") and thus keep the request object count at O(1) in the future.</p>

<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />You are receiving this because you are subscribed to this thread.<br />Reply to this email directly, <a href="https://github.com/bisq-network/bisq/pull/4233#discussion_r427842198">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/AJFFTNX2YJSR4KUYRHZCKT3RSOKDHANCNFSM4MY33J4A">unsubscribe</a>.<img src="https://github.com/notifications/beacon/AJFFTNS2LKV45ERX2IUO7G3RSOKDHA5CNFSM4MY33J4KYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGODC7DMJA.gif" height="1" width="1" alt="" /></p>
<script type="application/ld+json">[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/bisq-network/bisq/pull/4233#discussion_r427842198",
"url": "https://github.com/bisq-network/bisq/pull/4233#discussion_r427842198",
"name": "View Pull Request"
},
"description": "View this Pull Request on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]</script>