[bisq-network/projects] Reduce bandwidth-requirement on Bisq app startup (#25)

[Florian Reimair]

During startup, Bisq is required to send >4MB of data to seednodes in order to get its initial data. This is an issue because
- these requests tend to timeout on slow internet connections and leave the users stranded.
- the protocol as is does not scale and thus, startup will eventually become a much bigger issue

The primary goal of this project is to reduce the amount of data to be sent on startup.

## Why/why now?
- fixes https://github.com/bisq-network/bisq/issues/2547
- fixes https://github.com/bisq-network/bisq/issues/2549
- fixes https://github.com/bisq-network/bisq/issues/3763
- might fix https://github.com/bisq-network/bisq/issues/3797
- the solution as is does not scale, issue will grow worse

<details>

# Problem statement
On startup, a Bisq application first requests up-to-date network data from two seednodes. Once data comes in, the Bisq app jumps from the loading screen to the trading UI. However, if no data arrives, Bisq stays at the loading screen forever.

There are two main reasons why this happens:
- internet uplink is too slow and hits a seednode's connection timeout during request
- the initial data request is huge. It by the time of writing exceeds 4MB and is bound to grow further

As of 2020-03-05 17:00 CET the numbers are:
- 103726 keys worth 2282.901kB in total
- only < 100 keys are not "AppendOnly" data keys
- are sent twice (seed node redundancy, first response wins)
- in parallel (because we can)

# Proposed Solution
By adding the info "I am Bisq v1.2.1" to the list of known objects, we know what objects the client has - namely, objects shipped with the data stores of v1.2.1.

- bin up the data
- create a "special" key for addressing bins
- tie those bins to the application version
- create a new bin on every release holding only new data

## Advantages
- reduce the number of historical keys to be send to O(1) (right now, it is O(n), @stejbac's solution is probably O(ln(n))(?))
- no new fields needed in the protocol
- robust? if a requestee does not know the "special" key, it just sends all the data
- much simpler and therefore, easier to maintain compared to @stejbac's [approach](https://github.com/bisq-network/bisq/pull/3896)

## Disadvantages
- introduce a dependency on "time", ie. the bisq version
</details>

## Risks
- as always with p2p network stuff, it might break everything (not if we do our jobs properly)
- if we fail to address the issue, the network might suffer (because the current solution does not scale)

## Alternative approaches
- https://github.com/bisq-network/bisq/pull/3896

# Tasks
- [ ] feasibility study (ie. can we create bins easily, can we use bins easily)
- [ ] proof-of-concept implementation with benchmarks
- [ ] make implementation production ready
- [ ] thorough testing
- [ ] upgrade seed nodes before releasing the feature
- [ ] release the feature

# Criteria for Delivery
- [ ] benchmark data
- [ ] test reports for application startup on all major OSs
- [ ] upgraded seednodes
- [ ] release

# Estimates
| Task | Amount [USD] |
| --- | ---:|
| feasibility study | 600,00 |
| proof-of-concept impl | 2400,00 |
| production-ready | 1100,00 |
| testing | 700,00 |
| other | 500,00 |
| **total** | **5300,00** |



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/bisq-network/projects/issues/25
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bisq.network/pipermail/bisq-github/attachments/20200305/cf312886/attachment.html>