<ul>
<li>I created a windows box on aws to test stuff on it.</li>
<li>The workaround of building bitcoinj with jdk8 and bisq with jdk10 works fine at build time. At runtime, everything is run with java 10, so there is still a problem.</li>
<li>Opening and and closing the Bisq app on windows works smoothly.
<ul>
<li>My guess: During "bitcoinj shutdown", NoSuchMethodError is thrown when sun.misc.Cleaner code is going to be executed by WindowsMMapHack but since WalletConfig.shutdown() has a “catch Throwable” that does not do a log, nothing is printed on the log. Since the app closes, maybe locks are cleaned up and the problem is not exposed to the user.</li>
</ul>
</li>
<li>Bisq restore from seed on windows fails.
<ul>
<li>My guess: NoSuchMethodError is thrown and not logged (same thing that happens when Bisq is closed). Then when a “new bitcoinj” is created, we get an exception.</li>
<li>The UI informs the user an error has ocurred. The log prints:</li>
</ul>
</li>
</ul>
<pre><code>Dec-26 19:26:50.393 [JavaFX Application Thread] INFO o.b.c.MnemonicCode: PBKDF2 took 14.38 ms
Dec-26 19:26:50.632 [ STOPPING] INFO o.b.c.PeerGroup: Awaiting PeerGroup shutdown ...
Dec-26 19:26:50.633 [PeerGroup Thread] INFO o.b.c.PeerGroup: Stopping ...
Dec-26 19:26:51.063 [PeerGroup Thread] INFO o.b.c.PeerGroup: Stopped.
Dec-26 19:26:51.615 [ STOPPING] INFO o.b.s.SPVBlockStore: Windows mmap hack: Forcing buffer cleaning
Dec-26 19:26:51.766 [RestoreBTCWallet-%d] INFO b.c.b.s.WalletsSetup: Socks5Proxy for bitcoinj: socks5Proxy=127.0.0.1:50495 Version 5
Dec-26 19:26:51.775 [RestoreBTCWallet-%d] INFO o.b.c.Context: Creating bitcoinj 0.14.4.11 context.
Dec-26 19:26:51.777 [RestoreBTCWallet-%d] INFO b.c.b.n.BtcNetworkConfig: You connect with peerAddresses: [[r3dsojfhwcm7x7p6.onion]:8333, [vlf5i3grro3wux24.onion]:8333, [fz6nsij6jiyuwlsc.onion]:8333, [c6ac4jdfyeiakex2.onion]:8333, [sjyzmwwu6diiit3r.onion]:8333, [3xucqntxp5ddoaz5.onion]:8333, [lgkvbvro67jomosw.onion]:8333, [4jyh6llqj264oggs.onion]:8333, [mxdtrjhe2yfsx3pg.onion]:8333, [3r44ddzjitznyahw.onion]:8333, [i3a5xtzfm4xwtybd.onion]:8333, [z33nukt7ngik3cpe.onion]:8333, [sslnjjhnmwllysv4.onion]:8333]
Dec-26 19:26:51.785 [ STARTING] INFO b.c.b.s.WalletConfig: Wallet directory: C:\Users\Administrator\AppData\Roaming\Bisq\btc_mainnet\wallet
Dec-26 19:26:51.788 [ STARTING] INFO b.c.b.s.WalletConfig: Renaming old wallet file C:\Users\Administrator\AppData\Roaming\Bisq\btc_mainnet\wallet\bisq_BTC.wallet to C:\Users\Administrator\AppData\Roaming\Bisq\btc_mainnet\wallet\Backup 1 for bisq_BTC.wallet
Dec-26 19:26:51.827 [ STARTING] INFO o.b.w.DeterministicKeyChain: 133 keys needed for M/44H/0H/0H/0 = 0 issued + 100 lookahead size + 33 lookahead threshold - 0 num children
Dec-26 19:26:51.878 [ STARTING] INFO o.b.w.DeterministicKeyChain: Took 46.38 ms
Dec-26 19:26:51.878 [ STARTING] INFO o.b.w.DeterministicKeyChain: 133 keys needed for M/44H/0H/0H/1 = 0 issued + 100 lookahead size + 33 lookahead threshold - 0 num children
Dec-26 19:26:51.918 [ STARTING] INFO o.b.w.DeterministicKeyChain: Took 39.75 ms
Dec-26 19:26:52.012 [ STARTING] INFO b.c.b.s.WalletConfig: Renaming old wallet file C:\Users\Administrator\AppData\Roaming\Bisq\btc_mainnet\wallet\bisq_BSQ.wallet to C:\Users\Administrator\AppData\Roaming\Bisq\btc_mainnet\wallet\Backup 1 for bisq_BSQ.wallet
Dec-26 19:26:52.013 [ STARTING] INFO o.b.w.DeterministicKeyChain: 133 keys needed for M/44H/142H/0H/0 = 0 issued + 100 lookahead size + 33 lookahead threshold - 0 num children
Dec-26 19:26:52.034 [ STARTING] INFO o.b.w.DeterministicKeyChain: Took 19.19 ms
Dec-26 19:26:52.034 [ STARTING] INFO o.b.w.DeterministicKeyChain: 133 keys needed for M/44H/142H/0H/1 = 0 issued + 100 lookahead size + 33 lookahead threshold - 0 num children
Dec-26 19:26:52.063 [ STARTING] INFO o.b.w.DeterministicKeyChain: Took 28.23 ms
Dec-26 19:26:52.090 [JavaFX Application Thread] ERROR b.c.b.s.WalletsSetup: Service failure from state: STARTING; failure={} java.io.IOException: org.bitcoinj.store.BlockStoreException: java.nio.channels.OverlappingFileLockException
at bisq.core.btc.setup.WalletConfig.startUp(WalletConfig.java:494)
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62)
at com.google.common.util.concurrent.Callables$4.run(Callables.java:122)
at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: org.bitcoinj.store.BlockStoreException: java.nio.channels.OverlappingFileLockException
at org.bitcoinj.store.SPVBlockStore.<init>(SPVBlockStore.java:133)
at bisq.core.btc.setup.WalletConfig.provideBlockStore(WalletConfig.java:331)
at bisq.core.btc.setup.WalletConfig.startUp(WalletConfig.java:406)
... 3 common frames omitted
Caused by: java.nio.channels.OverlappingFileLockException: null
at java.base/sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
at java.base/sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
at java.base/sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1191)
at java.base/java.nio.channels.FileChannel.tryLock(FileChannel.java:1160)
at org.bitcoinj.store.SPVBlockStore.<init>(SPVBlockStore.java:106)
... 5 common frames omitted
Dec-26 19:26:52.097 [JavaFX Application Thread] ERROR b.d.u.GUIUtil: java.io.IOException: org.bitcoinj.store.BlockStoreException: java.nio.channels.OverlappingFileLockException
</code></pre>
<ul>
<li>I isolated and reproduced that behavior:</li>
</ul>
<pre><code>import sun.misc.Cleaner;
import sun.nio.ch.DirectBuffer;
import java.io.File;
import java.io.RandomAccessFile;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.channels.FileLock;
public class WindowsTest2 {
public static void main(String[] argv) throws Exception {
File myfile = new File("myfile.txt");
boolean exists = myfile.exists();
RandomAccessFile randomAccessFile = new RandomAccessFile(myfile, "rw");
long fileSize = getFileSize();
if (!exists) {
//log.info("Creating new SPV block chain file " + file);
randomAccessFile.setLength(fileSize);
} else if (randomAccessFile.length() != fileSize) {
throw new RuntimeException("File size on disk does not match expected size: " +
randomAccessFile.length() + " vs " + fileSize);
}
FileChannel channel = randomAccessFile.getChannel();
FileLock fileLock = channel.tryLock();
MappedByteBuffer buffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, fileSize);
buffer.put(new byte[] {65,66,67});
System.out.println("qqqq");
buffer.force();
System.out.println("wwww");
if (System.getProperty("os.name").toLowerCase().contains("win")) {
System.out.println("eeee");
Cleaner cleaner = ((DirectBuffer) buffer).cleaner();
System.out.println("rrr");
if (cleaner != null) {
System.out.println("ttt");
cleaner.clean();
System.out.println("yyy");
}
System.out.println("uuu");
}
System.out.println("iiii");
buffer = null; // Allow it to be GCd and the underlying file mapping to go away.
System.out.println("oooo");
randomAccessFile.close();
System.out.println("pppp");
}
public final static int getFileSize() {
return 3;
}
}
</code></pre>
<ul>
<li>Running WindowsTest2 using jdk10 on windows, the log prints:</li>
</ul>
<pre><code>qqqq
wwww
eeee
Exception in thread "main" java.lang.NoSuchMethodError: sun.nio.ch.DirectBuffer.cleaner()Lsun/misc/Cleaner;
at WindowsTest2.main(WindowsTest2.java:34)
</code></pre>
<ul>
<li>I ran on windows the unit tests (<a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/bitcoinj/bitcoinj/commit/dbc4c35/hovercard" href="https://github.com/bitcoinj/bitcoinj/commit/dbc4c35">bitcoinj/bitcoinj@<tt>dbc4c35</tt></a>) Andreas wrote on bitcoinj master upstream to make sure removing the windows hack works fine and they passed (that does mean it will pass on every windows environment)</li>
</ul>
<p>So...</p>
<ul>
<li>Moving out of jdk8 for bitcoinj seems to be the way to go to solve this problem.</li>
<li>If we keep developing, testing and building bitcoinj using jdk8 and run it with java 10 we might find other problems like this one.</li>
<li>Then we have 2 possible solutions:
<ul>
<li>Add a copy of the sun.* classes to bitcoinj</li>
<li>Remove the windows hack and so the need to use sun.* (ie cherry pick what was done on bitcoinj master upstream).</li>
</ul>
</li>
<li>I will think a bit more about the 2 solutions and will share my conclusions here.</li>
</ul>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />You are receiving this because you are subscribed to this thread.<br />Reply to this email directly, <a href="https://github.com/bisq-network/bitcoinj/issues/17#issuecomment-450174590">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/AkpZtnMtmpXr0f1BHEsSZzJTSJ3oCC5Pks5u9On7gaJpZM4ZYg2L">mute the thread</a>.<img src="https://github.com/notifications/beacon/AkpZtlIl_Qnby96ovhusp8F98xpevR1Bks5u9On7gaJpZM4ZYg2L.gif" height="1" width="1" alt="" /></p>
<script type="application/json" data-scope="inboxmarkup">{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/bisq-network/bitcoinj","title":"bisq-network/bitcoinj","subtitle":"GitHub repository","main_image_url":"https://github.githubassets.com/images/email/message_cards/header.png","avatar_image_url":"https://github.githubassets.com/images/email/message_cards/avatar.png","action":{"name":"Open in GitHub","url":"https://github.com/bisq-network/bitcoinj"}},"updates":{"snippets":[{"icon":"PERSON","message":"@oscarguindzberg in #17: - I created a windows box on aws to test stuff on it.\r\n- The workaround of building bitcoinj with jdk8 and bisq with jdk10 works fine at build time. At runtime, everything is run with java 10, so there is still a problem.\r\n- Opening and and closing the Bisq app on windows works smoothly.\r\n - My guess: During \"bitcoinj shutdown\", NoSuchMethodError is thrown when sun.misc.Cleaner code is going to be executed by WindowsMMapHack but since WalletConfig.shutdown() has a “catch Throwable” that does not do a log, nothing is printed on the log. Since the app closes, maybe locks are cleaned up and the problem is not exposed to the user.\r\n- Bisq restore from seed on windows fails. \r\n - My guess: NoSuchMethodError is thrown and not logged (same thing that happens when Bisq is closed). Then when a “new bitcoinj” is created, we get an exception. \r\n - The UI informs the user an error has ocurred. The log prints: \r\n```\r\nDec-26 19:26:50.393 [JavaFX Application Thread] INFO o.b.c.MnemonicCode: PBKDF2 took 14.38 ms\r\nDec-26 19:26:50.632 [ STOPPING] INFO o.b.c.PeerGroup: Awaiting PeerGroup shutdown ...\r\nDec-26 19:26:50.633 [PeerGroup Thread] INFO o.b.c.PeerGroup: Stopping ...\r\nDec-26 19:26:51.063 [PeerGroup Thread] INFO o.b.c.PeerGroup: Stopped.\r\nDec-26 19:26:51.615 [ STOPPING] INFO o.b.s.SPVBlockStore: Windows mmap hack: Forcing buffer cleaning\r\nDec-26 19:26:51.766 [RestoreBTCWallet-%d] INFO b.c.b.s.WalletsSetup: Socks5Proxy for bitcoinj: socks5Proxy=127.0.0.1:50495 Version 5\r\nDec-26 19:26:51.775 [RestoreBTCWallet-%d] INFO o.b.c.Context: Creating bitcoinj 0.14.4.11 context.\r\nDec-26 19:26:51.777 [RestoreBTCWallet-%d] INFO b.c.b.n.BtcNetworkConfig: You connect with peerAddresses: [[r3dsojfhwcm7x7p6.onion]:8333, [vlf5i3grro3wux24.onion]:8333, [fz6nsij6jiyuwlsc.onion]:8333, [c6ac4jdfyeiakex2.onion]:8333, [sjyzmwwu6diiit3r.onion]:8333, [3xucqntxp5ddoaz5.onion]:8333, [lgkvbvro67jomosw.onion]:8333, [4jyh6llqj264oggs.onion]:8333, [mxdtrjhe2yfsx3pg.onion]:8333, [3r44ddzjitznyahw.onion]:8333, [i3a5xtzfm4xwtybd.onion]:8333, [z33nukt7ngik3cpe.onion]:8333, [sslnjjhnmwllysv4.onion]:8333]\r\nDec-26 19:26:51.785 [ STARTING] INFO b.c.b.s.WalletConfig: Wallet directory: C:\\Users\\Administrator\\AppData\\Roaming\\Bisq\\btc_mainnet\\wallet\r\nDec-26 19:26:51.788 [ STARTING] INFO b.c.b.s.WalletConfig: Renaming old wallet file C:\\Users\\Administrator\\AppData\\Roaming\\Bisq\\btc_mainnet\\wallet\\bisq_BTC.wallet to C:\\Users\\Administrator\\AppData\\Roaming\\Bisq\\btc_mainnet\\wallet\\Backup 1 for bisq_BTC.wallet\r\nDec-26 19:26:51.827 [ STARTING] INFO o.b.w.DeterministicKeyChain: 133 keys needed for M/44H/0H/0H/0 = 0 issued + 100 lookahead size + 33 lookahead threshold - 0 num children\r\nDec-26 19:26:51.878 [ STARTING] INFO o.b.w.DeterministicKeyChain: Took 46.38 ms\r\nDec-26 19:26:51.878 [ STARTING] INFO o.b.w.DeterministicKeyChain: 133 keys needed for M/44H/0H/0H/1 = 0 issued + 100 lookahead size + 33 lookahead threshold - 0 num children\r\nDec-26 19:26:51.918 [ STARTING] INFO o.b.w.DeterministicKeyChain: Took 39.75 ms\r\nDec-26 19:26:52.012 [ STARTING] INFO b.c.b.s.WalletConfig: Renaming old wallet file C:\\Users\\Administrator\\AppData\\Roaming\\Bisq\\btc_mainnet\\wallet\\bisq_BSQ.wallet to C:\\Users\\Administrator\\AppData\\Roaming\\Bisq\\btc_mainnet\\wallet\\Backup 1 for bisq_BSQ.wallet\r\nDec-26 19:26:52.013 [ STARTING] INFO o.b.w.DeterministicKeyChain: 133 keys needed for M/44H/142H/0H/0 = 0 issued + 100 lookahead size + 33 lookahead threshold - 0 num children\r\nDec-26 19:26:52.034 [ STARTING] INFO o.b.w.DeterministicKeyChain: Took 19.19 ms\r\nDec-26 19:26:52.034 [ STARTING] INFO o.b.w.DeterministicKeyChain: 133 keys needed for M/44H/142H/0H/1 = 0 issued + 100 lookahead size + 33 lookahead threshold - 0 num children\r\nDec-26 19:26:52.063 [ STARTING] INFO o.b.w.DeterministicKeyChain: Took 28.23 ms\r\nDec-26 19:26:52.090 [JavaFX Application Thread] ERROR b.c.b.s.WalletsSetup: Service failure from state: STARTING; failure={} java.io.IOException: org.bitcoinj.store.BlockStoreException: java.nio.channels.OverlappingFileLockException\r\n at bisq.core.btc.setup.WalletConfig.startUp(WalletConfig.java:494)\r\n at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62)\r\n at com.google.common.util.concurrent.Callables$4.run(Callables.java:122)\r\n at java.base/java.lang.Thread.run(Thread.java:844)\r\nCaused by: org.bitcoinj.store.BlockStoreException: java.nio.channels.OverlappingFileLockException\r\n at org.bitcoinj.store.SPVBlockStore.\u003cinit\u003e(SPVBlockStore.java:133)\r\n at bisq.core.btc.setup.WalletConfig.provideBlockStore(WalletConfig.java:331)\r\n at bisq.core.btc.setup.WalletConfig.startUp(WalletConfig.java:406)\r\n ... 3 common frames omitted\r\nCaused by: java.nio.channels.OverlappingFileLockException: null\r\n at java.base/sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)\r\n at java.base/sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)\r\n at java.base/sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1191)\r\n at java.base/java.nio.channels.FileChannel.tryLock(FileChannel.java:1160)\r\n at org.bitcoinj.store.SPVBlockStore.\u003cinit\u003e(SPVBlockStore.java:106)\r\n ... 5 common frames omitted\r\nDec-26 19:26:52.097 [JavaFX Application Thread] ERROR b.d.u.GUIUtil: java.io.IOException: org.bitcoinj.store.BlockStoreException: java.nio.channels.OverlappingFileLockException\r\n```\r\n- I isolated and reproduced that behavior:\r\n``` \r\nimport sun.misc.Cleaner;\r\nimport sun.nio.ch.DirectBuffer;\r\n\r\nimport java.io.File;\r\nimport java.io.RandomAccessFile;\r\nimport java.nio.MappedByteBuffer;\r\nimport java.nio.channels.FileChannel;\r\nimport java.nio.channels.FileLock;\r\n\r\npublic class WindowsTest2 {\r\n public static void main(String[] argv) throws Exception {\r\n File myfile = new File(\"myfile.txt\");\r\n boolean exists = myfile.exists();\r\n RandomAccessFile randomAccessFile = new RandomAccessFile(myfile, \"rw\");\r\n long fileSize = getFileSize();\r\n if (!exists) {\r\n //log.info(\"Creating new SPV block chain file \" + file);\r\n randomAccessFile.setLength(fileSize);\r\n } else if (randomAccessFile.length() != fileSize) {\r\n throw new RuntimeException(\"File size on disk does not match expected size: \" +\r\n randomAccessFile.length() + \" vs \" + fileSize);\r\n }\r\n\r\n FileChannel channel = randomAccessFile.getChannel();\r\n FileLock fileLock = channel.tryLock();\r\n\r\n MappedByteBuffer buffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, fileSize);\r\n buffer.put(new byte[] {65,66,67});\r\n System.out.println(\"qqqq\");\r\n buffer.force();\r\n System.out.println(\"wwww\");\r\n if (System.getProperty(\"os.name\").toLowerCase().contains(\"win\")) {\r\n System.out.println(\"eeee\");\r\n Cleaner cleaner = ((DirectBuffer) buffer).cleaner();\r\n System.out.println(\"rrr\");\r\n if (cleaner != null) {\r\n System.out.println(\"ttt\");\r\n cleaner.clean();\r\n System.out.println(\"yyy\");\r\n }\r\n System.out.println(\"uuu\");\r\n }\r\n System.out.println(\"iiii\");\r\n buffer = null; // Allow it to be GCd and the underlying file mapping to go away.\r\n System.out.println(\"oooo\");\r\n randomAccessFile.close();\r\n System.out.println(\"pppp\");\r\n }\r\n\r\n public final static int getFileSize() {\r\n return 3;\r\n }\r\n}\r\n```\r\n - Running WindowsTest2 using jdk10 on windows, the log prints:\r\n```\r\nqqqq\r\nwwww\r\neeee\r\nException in thread \"main\" java.lang.NoSuchMethodError: sun.nio.ch.DirectBuffer.cleaner()Lsun/misc/Cleaner;\r\nat WindowsTest2.main(WindowsTest2.java:34)\r\n```\r\n- I ran on windows the unit tests (bitcoinj@dbc4c35) Andreas wrote on bitcoinj master upstream to make sure removing the windows hack works fine and they passed (that does mean it will pass on every windows environment)\r\n\r\nSo...\r\n\r\n- Moving out of jdk8 for bitcoinj seems to be the way to go to solve this problem.\r\n- If we keep developing, testing and building bitcoinj using jdk8 and run it with java 10 we might find other problems like this one. \r\n- Then we have 2 possible solutions:\r\n - Add a copy of the sun.* classes to bitcoinj\r\n - Remove the windows hack and so the need to use sun.* (ie cherry pick what was done on bitcoinj master upstream).\r\n- I will think a bit more about the 2 solutions and will share my conclusions here.\r\n\r\n"}],"action":{"name":"View Issue","url":"https://github.com/bisq-network/bitcoinj/issues/17#issuecomment-450174590"}}}</script>
<script type="application/ld+json">[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/bisq-network/bitcoinj/issues/17#issuecomment-450174590",
"url": "https://github.com/bisq-network/bitcoinj/issues/17#issuecomment-450174590",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]</script>