[bisq-network/proposals] Mitigate issue with not confirmed deposit transactions (#132)

Fri Nov 1 17:18:10 UTC 2019

> _This is a Bisq Network proposal. Please familiarize yourself with the [submission and review process](https://docs.bisq.network/proposals.html)._

## Problem description
If the required miner fee increases very fast our fee estimation service (earn.com) is not reflecting that properly and the Bisq trader uses a too low fee which might cause that the published transaction will never get confirmed and will be removed from mempool after a while. In case it was a trade tx it can affect the other trader as well and therefor spread out like a virus.

Bisq allows spending unconfirmed outputs as otherwise usability would be terrible. There is no security risk involved with that as if one input for the deposit tx was not valid the deposit tx is also invalid and the buyer needs to wait until it is confirmed before starting the fiat/altcoin transfer. But it causes inconvenience as the application is not detecting and acting automatically to that situation and users need to do a SPV chain resync which can be very time and CPU consuming if the wallet is old and contains many transactions.

There are further potential problems but those have been never observed so far, so I will keep those out from a more detailed discussion (fee for payout tx is defined at take offer time -> can be too low as we don't know when the payout tx will be published). 

BitcoinJ does only know about transactions related to one's wallet but not about the trade peers trade fee tx. Technically it would be possible but it would require a SPV resync which is not feasible to do from a usability point of view. 

This problem is a conceptual problem and can never be prevented to 100% (even if we would have a full node instead of BitcoinJ) but we can try to reduce the reasons which triggers it and adding tools to reduce its negative effects. 

Beside low miner fee there might be some reasons causing similar effects: Bugs in the trade process (probably related to the wallet management with AddressEntryList) which could cause invalid transactions. We need to elimintate those bugs (edge cases and so far we never could reproduce it) and this reason will not be discussed in that proposal. Those are rather rare as well.

## Proposed solutions

1. **Get a better fee estimation service**
There is an old [bounty task](https://github.com/bisq-network/bisq/issues/1077) for that and we should prioritize to work on that. Our current service (earn.com) does not work well in times of high volatility and we are overpaying all the time to be on the safe side, so there is potential to make miner fees also cheaper.

We should add quality control for the fee estimation using historical data to compare how well our estimated fees have matched the fees used at real blocks. There are a few services out there and we should try to build on what is already developed and studied. It is a non-trivial topic which lacks of perfect solutions, so we should not under-estimate the effort to get that really well done. To get in touch with those who are working in that field is highly recommend (e.g. [Stefan Weiss](https://whatthefee.io/), [Laurnet](https://twitter.com/LaurentMT), https://bitcoiner.live,... ). 

We could run that service on our provide BTC nodes and Bisq users are connecting via Tor hidden service so privay is protected or we could proxy it over our exisisting price node which provides that data already now. Combining the provided BTC nodes with the fee estimation service only does not increase privacy risks but if we use the same service for lookup of trade peers transactions it potentially worsen the privacy exposure to those provided nodes as with the bloom filter the nodes learns also about with whom one is trading. So we should be careful by mixing too many of those services. 

2. **Add a check if a transaction is in the mempool**
In case the taker only took a partial part of the trade amount the seller (maker) will get a change output from the deposit tx. If either the taker fee tx of the deposit tx never gets confirmed because of too low miner fee, that change output on the makers side will also become invalid. If it was used for other offers or trades we have polluted those offers or trades as well and if another trader takes such an offer he might get polluted as well. 
To mitigate that we can add a check to see if a tx is in the mempool before we use it's outputs for funding another tx. If not we can isolate that utxo until it is confirmed from being spent.
This will only cover the already older transactions which got wiped out of the mempool but not cover the more recent transaction with too low fee still sitting in the mempool in the hope to get confirmed. So it is not clear if this will help us much. Avoiding the causes need to be the primary goal.   

3. **Don't allow spending unconfirmed outputs if they are not from the own transaction** 
We could limit the allowance to spend unconfirmed transaction to those utxos which are funded exclusively from own utxos. The deposit tx would therefor not fall into that and a potential change output would be unspendable until the deposit tx is confirmed. I think that is rather easy to do in BitcoinJ and would only affect usability in a few cases. This might be a preferred way to deal with the problem compared to the solution discussed above. But it will require more analysis if that still might cause severe usability drawbacks and if it is true that it is rather trivial from the BitcoinJ side to implement that.

4. **Add a metric or the estimated number of blocks it will take until a transaction will get confirmed**
We could use the fee estimation service this. If we see that a transaction will probably not get incuded in a certain time frame we can act on it and alert the user to double spend the tx with a higher fee and therefore canceling out the pending transaction. This approach might be more difficult and complex as it initially sounds. Our BitcoinJ version does not support RBF (replace by fee) as far i am aware ;-( (I think it is also not supported in v0.15). We still can do a forced double spend (and there is a hidden feature in Bisq to do that) but it requires a SPV resycn and we need to communicate that to the other trader and force him to a SPV resync as well. But that will only work after the double spend tx is confirmed. Also there might be race conditions that both traders do that at the same time and then their 2 different double spend txs are in a race which one will get into the block and the loser has essentially 2 txs with a double spend. So this might at the end cause more new problems as it is trying to solve or at least causes quite some complexity and UX challenges. We have to keep in mind that some users hardly understand the technology behind Bitcoin and terms like mempool or doublespend are either unknown to them or might stress them. Relying on traders to make a good decision here is a bad strategy. Forcing them from some automatiation into a stressful user experience (SPV resync can take very long) is problematic as well.

## Discussion

We have to keep in mind that this problem exists since day one and rather rarely it had caused big problems. If we would have a "perfect" fee estimation service (considering overpaying fee as a valid option to be on the safe side) this problem should nearly never happen and if it appears users can fix it by getting help from mediators or support.

Adding the 2 last solution suggestions seems to be rather complex and problematic and I would suggest to keep them as a secondary step if that problem is not getting solved by a high quality fee estimation service and rather put the effort into that work.

Lets do one step at a time and be aware that the challenges we are trying to solve are not trivial.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/bisq-network/proposals/issues/132
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bisq.network/pipermail/bisq-github/attachments/20191101/fa2331b2/attachment-0001.html>