<p>During startup, Bisq is required to send >4MB of data to seednodes in order to get its initial data. This is an issue because</p>
<ul>
<li>these requests tend to timeout on slow internet connections and leave the users stranded.</li>
<li>the protocol as is does not scale and thus, startup will eventually become a much bigger issue</li>
</ul>
<p>The primary goal of this project is to reduce the amount of data to be sent on startup.</p>
<h2>Why/why now?</h2>
<ul>
<li>fixes <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="421868445" data-permission-text="Title is private" data-url="https://github.com/bisq-network/bisq/issues/2547" data-hovercard-type="issue" data-hovercard-url="/bisq-network/bisq/issues/2547/hovercard" href="https://github.com/bisq-network/bisq/issues/2547">bisq-network/bisq#2547</a></li>
<li>fixes <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="421879645" data-permission-text="Title is private" data-url="https://github.com/bisq-network/bisq/issues/2549" data-hovercard-type="issue" data-hovercard-url="/bisq-network/bisq/issues/2549/hovercard" href="https://github.com/bisq-network/bisq/issues/2549">bisq-network/bisq#2549</a></li>
<li>fixes <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="534572122" data-permission-text="Title is private" data-url="https://github.com/bisq-network/bisq/issues/3763" data-hovercard-type="issue" data-hovercard-url="/bisq-network/bisq/issues/3763/hovercard" href="https://github.com/bisq-network/bisq/issues/3763">bisq-network/bisq#3763</a></li>
<li>might fix <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="538567473" data-permission-text="Title is private" data-url="https://github.com/bisq-network/bisq/issues/3797" data-hovercard-type="issue" data-hovercard-url="/bisq-network/bisq/issues/3797/hovercard" href="https://github.com/bisq-network/bisq/issues/3797">bisq-network/bisq#3797</a></li>
<li>the solution as is does not scale, issue will grow worse</li>
</ul>
<details>
<h1>Problem statement</h1>
<p>On startup, a Bisq application first requests up-to-date network data from two seednodes. Once data comes in, the Bisq app jumps from the loading screen to the trading UI. However, if no data arrives, Bisq stays at the loading screen forever.</p>
<p>There are two main reasons why this happens:</p>
<ul>
<li>internet uplink is too slow and hits a seednode's connection timeout during request</li>
<li>the initial data request is huge. It by the time of writing exceeds 4MB and is bound to grow further</li>
</ul>
<p>As of 2020-03-05 17:00 CET the numbers are:</p>
<ul>
<li>103726 keys worth 2282.901kB in total</li>
<li>only < 100 keys are not "AppendOnly" data keys</li>
<li>are sent twice (seed node redundancy, first response wins)</li>
<li>in parallel (because we can)</li>
</ul>
<h1>Proposed Solution</h1>
<p>By adding the info "I am Bisq v1.2.1" to the list of known objects, we know what objects the client has - namely, objects shipped with the data stores of v1.2.1.</p>
<ul>
<li>bin up the data</li>
<li>create a "special" key for addressing bins</li>
<li>tie those bins to the application version</li>
<li>create a new bin on every release holding only new data</li>
</ul>
<h2>Advantages</h2>
<ul>
<li>reduce the number of historical keys to be send to O(1) (right now, it is O(n), <a class="user-mention" data-hovercard-type="user" data-hovercard-url="/users/stejbac/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/stejbac">@stejbac</a>'s solution is probably O(ln(n))(?))</li>
<li>no new fields needed in the protocol</li>
<li>robust? if a requestee does not know the "special" key, it just sends all the data</li>
<li>much simpler and therefore, easier to maintain compared to <a class="user-mention" data-hovercard-type="user" data-hovercard-url="/users/stejbac/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/stejbac">@stejbac</a>'s <a href="https://github.com/bisq-network/bisq/pull/3896" data-hovercard-type="pull_request" data-hovercard-url="/bisq-network/bisq/pull/3896/hovercard">approach</a></li>
</ul>
<h2>Disadvantages</h2>
<ul>
<li>introduce a dependency on "time", ie. the bisq version</li>
</ul>
</details>
<h2>Risks</h2>
<ul>
<li>as always with p2p network stuff, it might break everything (not if we do our jobs properly)</li>
<li>if we fail to address the issue, the network might suffer (because the current solution does not scale)</li>
</ul>
<h2>Alternative approaches</h2>
<ul>
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="548748119" data-permission-text="Title is private" data-url="https://github.com/bisq-network/bisq/issues/3896" data-hovercard-type="pull_request" data-hovercard-url="/bisq-network/bisq/pull/3896/hovercard" href="https://github.com/bisq-network/bisq/pull/3896">bisq-network/bisq#3896</a></li>
</ul>
<h1>Tasks</h1>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> feasibility study (ie. can we create bins easily, can we use bins easily)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> proof-of-concept implementation with benchmarks</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> make implementation production ready</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> thorough testing</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> upgrade seed nodes before releasing the feature</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> release the feature</li>
</ul>
<h1>Criteria for Delivery</h1>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> benchmark data</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> test reports for application startup on all major OSs</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> upgraded seednodes</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> release</li>
</ul>
<h1>Estimates</h1>
<table>
<thead>
<tr>
<th>Task</th>
<th align="right">Amount [USD]</th>
</tr>
</thead>
<tbody>
<tr>
<td>feasibility study</td>
<td align="right">600,00</td>
</tr>
<tr>
<td>proof-of-concept impl</td>
<td align="right">2400,00</td>
</tr>
<tr>
<td>production-ready</td>
<td align="right">1100,00</td>
</tr>
<tr>
<td>testing</td>
<td align="right">700,00</td>
</tr>
<tr>
<td>other</td>
<td align="right">500,00</td>
</tr>
<tr>
<td><strong>total</strong></td>
<td align="right"><strong>5300,00</strong></td>
</tr>
</tbody>
</table>

<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />You are receiving this because you are subscribed to this thread.<br />Reply to this email directly, <a href="https://github.com/bisq-network/projects/issues/25?email_source=notifications&email_token=AJFFTNTUF5A43LMO7LI7OKDRF7L4FA5CNFSM4LCOORUKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IS3HHEQ">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/AJFFTNVWSVJQQUCTPG3CLDDRF7L4FANCNFSM4LCOORUA">unsubscribe</a>.<img src="https://github.com/notifications/beacon/AJFFTNT5A2A2PDOEKNVOLZ3RF7L4FA5CNFSM4LCOORUKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IS3HHEQ.gif" height="1" width="1" alt="" /></p>
<script type="application/ld+json">[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/bisq-network/projects/issues/25?email_source=notifications\u0026email_token=AJFFTNTUF5A43LMO7LI7OKDRF7L4FA5CNFSM4LCOORUKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IS3HHEQ",
"url": "https://github.com/bisq-network/projects/issues/25?email_source=notifications\u0026email_token=AJFFTNTUF5A43LMO7LI7OKDRF7L4FA5CNFSM4LCOORUKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IS3HHEQ",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]</script>