[bisq-network/support] Add knowledge base for support (#566)

Thu Nov 26 23:13:55 CET 2020

This is a collection of background information and instructions for support agents. Please ping me if anything is missing...

Would be great if the support lead can add it to a wiki page which can serve as knowledge base for support agents as well as for interested users who want to checkout themself in case of problems.

To help getting bugs faster fixed, please report any bug to developers in the keybase support channel. Ping devs with their handle so they get notified. Repeat requests if the report is not reviewed by a dev. Escalate early if there are repeated bugs specially after a new release so we can hunt down the bug early. Be a pain for devs so they fix the bugs ;-)!

Use the report tool for monthy reports and use the category radio buttons correctly and a good summary msg which helps to get good statistics and a quick overview about the problems.

It would be good if the support lead can do a monthly summary what are the % of diff. reasons for issues, total numbers per cycle and overall trend (do issues increase/decrease). We have to get a trend of decrease of issues each cycle and if not support lead need to escalate to devs to find ways to fix the problems.

To reduce roundtrip communication in the process please try to get the below list completed if possible for bug reports (not needed for already known/resolved issues).

### Required data for requesting support from developers:
- Create a directory with the trade ID as name
- Add a short description in a plain text file of the problem/bug. Use BTC buyer/BTC seller if you refer to any trader. Ask user when the issue happened so its easier to find the issue in the logs. 
  Ask them if they have done anything non-standard or had past issues.
- Add all available trade tx IDs of both traders as plain text to the report.
- Get a screenshot of the trade details window (click on trade ID). Tell users to only blur out the payment account details but not the tx IDs or other data. There is no relevant privacy issue with exposing trade ID, tx IDs or onion address. 
- Add a screenshot from the mediators dispute details window. The contract as json data is not required initially and only needed if the dev requests it. The payment account data in the json file should be removed  before handing over to a dev to protect traders privacy.
- Get all log files from both users (at `Account/Backup` there is a button for opening the data dir) - its up to 11 files. 
  Put all logs from one user into a directory named `buyer` and `seller`
- Add a screenshot of both traders first chat message (system message containing data like version number/OS,...)
- If there have been any error/warning popups request a screenshot of that as well.
- Zip all and send it in keybase as PM to a dev who has commited to work on it. Avoid adding it to the support channel as then nobody will feel responsible to work on it. 

### Trade transaction structure
If there are any issues with transactions or doubt if the deposit or delayed payout tx is correct you can check it in a block explorer by following the transactions outputs.

There are 5 diff. txs:
- Maker fee tx
- Taker fee tx
- Deposit tx
- Delayed payout tx (only get broadcast when arbitration gets opened)
- Payout tx 

The 2 trade fee txs have as first output the trade fee. That goes either to one of the BTC receiver addresses ([1] and [2]) or to a BSQ change address of the user in case BSQ was used for the fee (the diff from BSQ input and change output is used for miner fee and is the burned BSQ).
The **second output of both trade fee txs must lead to the (same) deposit tx**. The deposit tx has an OpReturn and the first output is to a multiSig address (start with `3` if non-segwit or with `bc1` if segwit). There can be an optional change output if trade amount was smaller than max. offer amount, but there are never more then 2 outputs + opReturn. 
If the delayed payout tx or the normal payout tx was published you see that the first output of the deposit tx (the MS output) is spent and lead to one of those txs as the single input. The normal payout tx has exactly 2 outputs, 1 address for each trader and that addresses are dispayed in the trade/ticket details.
The delayed payout tx need to have exactly 1 input and 1 output to one of the addresses listed in [1] and [3]. **If arbitration is opened this tx MUST exist otherwise a refund MUST NOT be made**. The input must be the multisig output of the deposit tx (we have a code to check if its the deposit tx of the trade and not any past one - if a popup in mediation or arbitration tells you that there is an issue, take it serious and contact devs for advice).

To check the txs (and addresses) use a block explorer which shows clearly the inputs and outputs, not all explorers have a very intuitive UI to follow the flows of txs. If you select the addresses like in the below screenshot you can see the flow pretty clearly.

<img width="1129" alt="tradefee2" src="https://user-images.githubusercontent.com/54558767/100388917-f644d380-2ff9-11eb-93f0-5cae757b06fb.png">
<img width="966" alt="tradefee1" src="https://user-images.githubusercontent.com/54558767/100388921-f80e9700-2ff9-11eb-9893-7f806e9b6ee3.png">
<img width="1118" alt="payout" src="https://user-images.githubusercontent.com/54558767/100388930-fcd34b00-2ff9-11eb-9e6c-45f2ae5939ff.png">

Checking if the trade fee tx leads to the deposit tx helps to quickly see if the trade funds are locked up and all is ok with the trade. If that is not the case the traders might have lost trade fee (if their trade fee tx is published - if one side is missing that user has not lost trade fee so no fee reimbursement is justified, such users need to do a SPV resync as the tx is likely committed in their wallet but not in the mempool due to too low fees or a bug at broadcast).

### One trade fee tx not found in mempool
In case the miner fee was too low or due broadcast problems (bad network connectivity or bug) it can be that one of the trade fee txs is not getting confirmed or has not been broadcast.
After a few days unconfirmed tx gets removed from the bitcoin nodes mempool (size of mempool is a local setting for each node/explorer so it can be that its visible in some explorers but in others not).
In such cases the deposit tx cannot be confirmed as well. Instruct the trader who's tx was not confirmed that he does a SPV resync so that the tx gets cleaned from his local wallet. 
Otherwise there is high risk that the change output gets used for another tx and that will fail then as well. Best to deactivate all open offers to avoid that a taker could take an offer before the wallet is resynced.
The trader who has the confirmed  trade fee tx has lost his trade fee and can request for reimbursement.   

### SPV resync
SPV resync tends to be recommened too often in support. 
SPV resync can be quite heavy and is not a great user experience to do, so only recommend if really needed.
How long a SPV takes depends on the age of the wallet and the number of transactions. 
Old wallets with many transactions can take a day or even more. Usually with a fresh empty wallet its about some minutes up to 1 hour. 
It is faster if the user runs a local BTC node (not pruned). It also depends on internet connection and amount of CPU/RAM. CPU will be maxed out during sync.

SPV resync is needed if there is a transaction in the wallet which never got confirmed. You can check for that by looking up the tx in several block explorers, if its not found in any then its pretty sure that tx got removed from mempool. This is expected in case miner fee was too low or when the tx broadcast failed.
Another reason to do a SPV resync is when there is a message that the wallet contains a `double spend` tx. This is usually not intentional done by the user but due to some errors or reorgs in rare cases.
There might be other more rare error conditions, but for those I recommend to contact the devs before recommending a SPV resync as it will help as well to fix potential bugs.

If users do not get connected to the BTC network do NOT recommend SPV resync. 
Potential reasons are:
- User runs Bitcoin node in not pruned mode
- Bitcoin node is not full synced when Bisq start (Bitcoin node need to always be full synced before starting Bisq)
- User had set custom BTC node in settings or prog arg and has changed his setup
- Connection issues
- Tor issues

### Connection issues
In rare cases Tor can cause problems. If no connection can be created after several restarts and if user is sure to have normal connectivity he should clean the Tor files via `Settings/Network/Tor settings`. He should make a backup first and not do it manually. Deleting the hiddenservice directory would make his node inaccessible as he would create a new onion address! If there are no open offers/trades/disputes and user dont care to lose his local reputation it is not problematic to delete the onion priv key. In some rare cases that can help as well (I saw in rare cases the changing the onion address solved Tor connection issues). 
Tor has some ddos protection which can lead to get flagged an IP address. We see that from time to time on seed nodes and its likely caused by too much traffic or some traffic pattern which Tor interprets as ddos. In such cases changing the IP should help. Using a VPN can help to check if thats the problem (I had it once myself that I could not use Tor anymore and only via VPN it worked). After a few weeks it was ok again so it seems those bans fade out over time. Testing with Tor browser is another option to see if Tor itself has issues (does not cover issues if the onion address or cache files).

If network speed is very low (< 2 Mbit/sec) there might be connection timouts. We improved some data requests already and will improve more in upcoming releases but specially at startup the BTC sync and getData requests can be a bit heavy, specially if Bisq was not used for a while. Doing repeated startups can help as some data gets persisted and at next request less data is required. Using a local Bitcoin node helps as well as the BTC sync is then not adding up to tor traffic.
If the user has not used Bisq since more then a month he should update to latest version and then delete some files in the `db` directory inside the data directory. Then those files get replaced by more recent ones which go shipped with the latest binary. that avoids to need to download the missing data from seed nodes (seed nodes have limits as well of amount of delivered data). Those files are:
- `DaoStateStore`
- `SignedWitnessStore`
- `AccountAgeWitnessStore`
- `BlindVoteStore`
- `ProposalStore`

Please note that deleting or manipulating files inside the db directory is usually not recommended and can cause severe problems! Some files are basically only cache files from public data like the ones above, those are not critical, but other data are local private data and would lead to a corrupted state or failed trades.

### BTC Balance issues
There is a wallet info tool which can be opened with cmd+j. There you see all the raw data of the wallet: Addresses, public and private keys, txs, balance of both BTC and BSQ. Can be useful for investigation.
Another option is the open the emergency wallet tool by cmd+e. There one can withdraw all available funds (not including locked up funds in multisig). The displayed balance there is the sum of
the available balance + reserved in offers balance. 

It is not recommended to use the BTC private keys outside of Bisq. It always can be recovered from Bisq. I am not aware of any case where funds got lost. Spending BTC outside of Bisq can lead to failed taker offer attempts, failed trades or double spends in Bisq and can screw up the wallet.
For those who still want to do it here are the HD wallet paths:
BTC wallet path is: M/44H/0H/1H
BSQ wallet path is: M/44H/142H/1H

BSQ should never be used outside of Bisq as the risk to burn the BSQ is very high. A BSQ tx has very strict rules how it is constructed and small mistakes lead to lost BSQ. 

### Backup
- If a user reports a bug tell user to make a backup of data dir. This can help that relevant log files and rolling backups are still available later. 
Rolling backup means that old files get deleted (e.g. bisq10.log is oldest log file) once the max. number files is reached, so we can only go back a certain depth back in history.
- Tell users to always make a full backup of the data dir. Seed node is only an emergency backup and if there are open trades funds are at risk (e.g. no communication possible with mediator/peer, or manual payout required).
- Every db and wallet file has a backup folder with the rolling backups. They are named with the unix epoch timestamp. Replace an existing file with a backup file should be done only in emergency cases and only if dev gives the advice.

### Start over with a new data directory
See https://bisq.community/t/how-to-switch-to-a-new-data-directory/7583 for more details

### Logs
Log files are pretty verbose and a big hard to interpret for no developers but a few useful information can be easily extraced:
If you search for `VERSION=` you can see the users version. It gets logged at startup. If the user has once updated there must never follow a lower version in flater log entries. Downgrade is not supported and caused severe problems at the segwit update. After 1.5.0 we protect that downgrade is not possible, but will only have affect in next version after 1.5.0.
If you look for an error search for ERROR (all uppercase). But as there are some non critical errors as well (e.g. caused from broken connections) its better to leave that interpretation to devs. If you post errors post the stacktrace as well if it is available or at least a few log lines before and after the error so it is easier to get an idea what caused it.

### References

[1] Bisq donation addresses:
Current Bisq fee receiver address: 38bZBj5peYS3Husdz7AH3gEUiUbYRD951t
Past fee receiver/donation addresses:
1BVxNn3T12veSK6DgqwU4Hdn7QHcDDRag7, 3EtUWqsGThPtjwUczw27YCo6EWvQdaPUyp, 3A8Zc1XioE2HRzYfbb5P8iemCS72M6vRJV

[2] Fee receiver addresses of victims from security incident:
1EKXx73oUhHaUh8JBimtiPGgHfwNmxYKAj
1HpvvMHcoXQsX85CjTsco5ZAAMoGu2Mze9 
3EfRGckBQQuk7cpU7SwatPv8kFD1vALkTU 
13sxMq8mTw7CTSqgGiMPfwo6ZDsVYrHLmR 
19qA2BVPoyXDfHKVMovKG7SoxGY7xrBV8c
19BNi5EpZhgBBWAt5ka7xWpJpX2ZWJEYyq

[3] New receiver address for delayed payout tx (no DAO param change yet done but will  be activated soon):
34VLFgtFKAtwTdZ5rengTT2g2zC99sWQLC

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/bisq-network/support/issues/566
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bisq.network/pipermail/bisq-github/attachments/20201126/07a3aca7/attachment-0001.htm>