<h2>Cycle 3 report</h2>
<h3>Summary</h3>
<p>Running 2 seed nodes.</p>
<ul>
<li>fl3mmribyxgrv63c.onion</li>
<li>3f3cu2yw7u457ztq.onion</li>
</ul>
<h3>Issues Encountered</h3>
<p><strong>Issue 1:</strong><br>
On July 4th, fl3mmribyxgrv63c was delivering outdated blocks as head blocks. See the <a href="https://monitor.bisq.network/d/iVpBgXXmk/dao-sync-status?refresh=5m&orgId=1&panelId=16&fullscreen&from=1562259600000&to=1562263200000" rel="nofollow">monitor</a> for more details. The issue appeared to be caused by failing to receive notifications from bitcoind (see log snippet below). No idea why at this point. I ended up restarting the seed node and it resoved the issue.</p>
<pre><code>bisq.core.dao.node.full.RpcException: NotificationHandlerException(super=com.neemre.btcdcli4j.daemon.NotificationHandlerException: Error #1004004: The operation failed due to an unknown IO exception., error=Errors(code=1004004, message=The operation failed due to an unknown IO exception.), code=1004004)
at bisq.core.dao.node.full.RpcService.lambda$setup$0(RpcService.java:138)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: NotificationHandlerException(super=com.neemre.btcdcli4j.daemon.NotificationHandlerException: Error #1004004: The operation failed due to an unknown IO exception., error=Errors(code=1004004, message=The operation failed due to an unknown IO exception.), code=1004004)
at com.neemre.btcdcli4j.daemon.notification.worker.NotificationWorker.call(NotificationWorker.java:64)
at com.neemre.btcdcli4j.daemon.notification.worker.NotificationWorker.call(NotificationWorker.java:22)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
... 3 more
Caused by: java.net.SocketException: Connection reset
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
at java.base/java.io.BufferedReader.fill(BufferedReader.java:161)
at java.base/java.io.BufferedReader.readLine(BufferedReader.java:326)
at java.base/java.io.BufferedReader.readLine(BufferedReader.java:392)
at com.neemre.btcdcli4j.daemon.notification.worker.NotificationWorker.call(NotificationWorker.java:46)
... 7 more
</code></pre>
<p><strong>Issue 2:</strong><br>
<a class="user-mention" data-hovercard-type="user" data-hovercard-url="/hovercards?user_id=24721710" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/alexej996">@alexej996</a> was encountering issues with his seed node. While looking at the logs, it seemed to be hitting the memory limit and restarting:</p>
<pre><code>%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We are over our memory limit (1200) and trigger a restart. usedMemory: 1275 MB. freeMemory: 295 MB
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
</code></pre>
<p>And frequently going over 80%:</p>
<pre><code>%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We are over 80% of our memory limit (960) and call the GC. usedMemory: 1156 MB. freeMemory: 414 MB
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
</code></pre>
<p>There is a PR that was merged recently that may or may not help with memory usage: <a class="issue-link js-issue-link" data-error-text="Failed to load issue title" data-id="417676938" data-permission-text="Issue title is private" data-url="https://github.com/bisq-network/bisq/issues/2501" data-hovercard-type="pull_request" data-hovercard-url="/bisq-network/bisq/pull/2501/hovercard" href="https://github.com/bisq-network/bisq/pull/2501">bisq-network/bisq#2501</a></p>
<p>If it doesn't help, we may require further improvements or potentially increase the memory limit for now.</p>
<h3>Maintenance Performed</h3>
<p>No maintenance performed this month.</p>
<h3>Expenses Incurred</h3>
<p>Expenses incurred for the month (USD):</p>
<ul>
<li>2 x $20 for server (2 CPU, 4 GB RAM)</li>
<li>2 x $30 for storage (300 GB)</li>
</ul>
<p><strong>Total:</strong> $100</p>
<p>/cc <a href="https://github.com/bisq-network/compensation/issues/309" data-hovercard-type="issue" data-hovercard-url="/bisq-network/compensation/issues/309/hovercard">bisq-network/compensation#309</a></p>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />You are receiving this because you are subscribed to this thread.<br />Reply to this email directly, <a href="https://github.com/bisq-network/roles/issues/15?email_source=notifications&email_token=AJFFTNX7GWA2MMWBY2QWFW3P6OKM3A5CNFSM4DZPNCH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZOEV3Q#issuecomment-509364974">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/AJFFTNRRMWMM6BB2AOLMEZDP6OKM3ANCNFSM4DZPNCHQ">mute the thread</a>.<img src="https://github.com/notifications/beacon/AJFFTNUBJZOKVCXSAQRE76DP6OKM3A5CNFSM4DZPNCH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZOEV3Q.gif" height="1" width="1" alt="" /></p>
<script type="application/ld+json">[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/bisq-network/roles/issues/15?email_source=notifications\u0026email_token=AJFFTNX7GWA2MMWBY2QWFW3P6OKM3A5CNFSM4DZPNCH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZOEV3Q#issuecomment-509364974",
"url": "https://github.com/bisq-network/roles/issues/15?email_source=notifications\u0026email_token=AJFFTNX7GWA2MMWBY2QWFW3P6OKM3A5CNFSM4DZPNCH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZOEV3Q#issuecomment-509364974",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]</script>