[bisq-network/bisq] [WIP] Use keyset delta algorithm to speed up GetData requests (#3896)

[chimp1984]

I think we have to separate different types of messages.

Trade statistics and account age witness objects are append only and ever growing. They are in sum about 80 000 objects as far I remember.

Offers and mailbox messages are not growing linear with number of user or trade volume IMO as conversion time for an open offer to a trade will be faster with more traders. Offers and mailbox messages are about the same number most of the time as far I remember. Atm 383 offers and 455 mailbox messages [1]. 

Trade statistics and account age witness use a 20 bytes hash (other data 32 bytes). 
Total size for 80k objcest would be 1,6 MB, so thats the big chunk of the request. If we would treat those objcets different (as proposed above using some split of historical data) we would decrease the issue with the large request quite a bit (and decrease load for seed nodes by processing less keys).

I think the optimisation of the keys (bloom filter like) is still great but might get lower priority of the trade statistics and account age witness are treated differently.

So I would suggest to treat both problems separate and develop a roadmap how to implement and deploy.

Regarding deployment: 
I think we cannot use the capability feature here as at the first request from a node to a seed node the requester node does not know the seed nodes capabilities (not 100% sure about that).
But seed nodes are under our control so we can ensure all are updated with the feature before rolling it out to users. Only remaining problem would be that in case no seed node is available a normal node would be choosen and then if that would be a not updated node a disconnect would be triggered if it receives a unknown message. But I think we can ignore that theorectical problem as this would only become an issue if all seed nodes go offline (happend only once in 3,5 years). 

I also suggest to use a new API (e.g. new message types) instead of overloading the existing one.
 
It still would require lots of care and testing but might not be as difficult to deploy like new features which are used between normal nodes.

[1]
```
BlindVotePayload=181
RefundAgent=1
Filter=2
TempProposalPayload=87
MailboxStoragePayload=455
ProposalPayload=258
Alert=1
Mediator=2
OfferPayload=383
SignedWitness=1768
BSQ Blocks=613414
``` 
(From http://104.248.88.175/)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/bisq-network/bisq/pull/3896#issuecomment-575902164
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bisq.network/pipermail/bisq-github/attachments/20200118/420660fe/attachment-0001.html>