[bisq-network/bisq] Reduce initial request size (#4233)

[Florian Reimair]

@freimair commented on this pull request.



> +                    e.printStackTrace();
+                }
+            } else {
+                log.debug(file + " file exists already.");
+            }
+        });
+
+        // split
+        // - get all
+        history = new HashMap<>();
+        store = readStore(getFileName());
+        resourceFiles.forEach(file -> {
+            SplitStore tmp = readStore(file.replace(postFix, ""));
+            history.put(file.replace(postFix, "").replace(getFileName(), "").replace("_", ""), tmp);
+            // - subtract all that is in resource files
+            store.getMap().keySet().removeAll(tmp.getMap().keySet());

> With this change I think we're making the resource files the final arbiter of what is the distributed data. Is that what we want? Even for full nodes that are always on.

This seems to be the only way of introducing "checkpoints" so that we do not have to query all the data.

> It looks like it will just throw away its own local data on new versions though, that seems incorrect.

It does not throw away any data. It only de-duplicates the data. If we have a data store "incoming" from resources, we remove the data that is contained in the resource store from our live data store.

An example:
Given _ResourceStore={1,3}_ and _LiveStore(t)={1,2,3,4,5}_ and we enter the copy and split business logic, the outcome would be having the _ResourceStore_ being copied to the working directory and the new _LiveStore(t+1)=LiveStore(t) \ ResourceStore = {2,4,5}_. Together, they hold {1,2,3,4,5}.

Bringing it all together:
In terms of initial data request size, we can now ask for all data except _ResourceStore_ simply by telling other that we have the "ResourceStore_1.3.4" -> sending {"1.3.4"} instead of {1,3}. And because in real live, the data stores count >100k objects, we save a whole lot of data. Additionally, we actually ask for all data "since" (instead of "except") and thus keep the request object count at O(1) in the future.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/bisq-network/bisq/pull/4233#discussion_r427842198
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bisq.network/pipermail/bisq-github/attachments/20200520/a994abb8/attachment.html>