[bisq-network/bisq] Migrate to new trade statistics (#4599)

Tue Oct 6 05:02:18 UTC 2020

Replaces #4577

This must not be merged to master before the seed nodes are deployed with that version. We should merge it to a releasebranch and then start setting up the seeds. I put it as draft to avoid accidental merge. But it is open for review.

We replace the old trade statistics which a minimal version and reduce data size from 16.5 MB to 3.9 MB. The old data gets converted to the new format.

Not updated users will not see new trade statistics and old ones only if they have peers with the old version as well. As all seed nodes will run the updated version they will not provide the old statistics. So for those users it will look weird that trade volume goes to zero over time. But I think that is an acceptable price and the alternative would be higher complexity by supporting boths trade statistics still. As 1.4.0 has so many important features I guess most active users will update quickly anyway.

If 2 traders with an old version trade they will publish both the statistics and new nodes will convert them to new trade stats using the hash of the old as hash for the new and so avoiding duplicates.
If 1 of the peers has update and the other not it will be detected via capabilities. In the new version onlt the seller publishes. So if the new version is the buyer he will not publish anyway, so we have only the old version publsihed and get converted to the new version by receiving peers. If the updated user is the seller he will check the capability of the peer and if its an old verison he will no publish so that the buyer with the old version publishes. If both users have updated then only the seller publishes.
So we have only one scenario where both publish (both have not updated) and we solve that with using the hash.

At startup we convert the existing trades stats to the new version (running in a thread, takes about 4 sec). After that we delete the old trade stats. The old trade stats will be excluded from initial data requests and response.
You can view, comment on, or merge this pull request online at:

  https://github.com/bisq-network/bisq/pull/4599

-- Commit Summary --

  * Add TradeStatistics3 and related classes
  * Remove PublishTradeStatistics from buyer protocol
  * Refactor: Move class
  * Refactor: Rename class
  * let seller publish trade statistics only if peer is updated user. If not the peer will publish only.
  * Delete TradeStatistics (version 1)
  * We "hack" TradeStatistics2StorageService to fulfill our needs:
  * Apply TradeStatistics3 to TradeStatisticsManager and some related classes
  * Apply TradeStatistics3 to client classes
  * Adjust tests, remove tests which do not make sense anymore
  * Use TradeStatistics3 in protobuf file
  * Remove resource file
  * Prune mediator and refund agent entries for all entries beside the last 100 we use for the selection algorithm.
  * Add injector.getInstance(TradeStatisticsConverter.class) to BisqExecutable to enforce inclusion.
  * Do conversion in a thread to not block UI thread. takes about 4 seconds on my machine.
  * Add filter for excluding null objects
  * Add check that size is > LOOK_BACK_RANGE

-- File Changes --

    M common/src/main/java/bisq/common/app/Capability.java (3)
    M common/src/main/java/bisq/common/file/JsonFileManager.java (4)
    M core/src/main/java/bisq/core/api/CoreApi.java (4)
    M core/src/main/java/bisq/core/app/BisqExecutable.java (2)
    M core/src/main/java/bisq/core/dao/governance/asset/AssetService.java (6)
    M core/src/main/java/bisq/core/offer/availability/DisputeAgentSelection.java (19)
    M core/src/main/java/bisq/core/proto/CoreProtoResolver.java (3)
    M core/src/main/java/bisq/core/proto/network/CoreNetworkProtoResolver.java (4)
    M core/src/main/java/bisq/core/proto/persistable/CorePersistenceProtoResolver.java (3)
    M core/src/main/java/bisq/core/provider/price/PriceFeedService.java (17)
    M core/src/main/java/bisq/core/setup/CoreNetworkCapabilities.java (3)
    M core/src/main/java/bisq/core/trade/TradeModule.java (4)
    M core/src/main/java/bisq/core/trade/protocol/BuyerProtocol.java (4)
    M core/src/main/java/bisq/core/trade/protocol/SellerProtocol.java (4)
    D core/src/main/java/bisq/core/trade/protocol/tasks/PublishTradeStatistics.java (80)
    A core/src/main/java/bisq/core/trade/protocol/tasks/seller/SellerPublishesTradeStatistics.java (105)
    D core/src/main/java/bisq/core/trade/statistics/TradeStatistics.java (236)
    M core/src/main/java/bisq/core/trade/statistics/TradeStatistics2.java (2)
    M core/src/main/java/bisq/core/trade/statistics/TradeStatistics2StorageService.java (23)
    A core/src/main/java/bisq/core/trade/statistics/TradeStatistics3.java (355)
    A core/src/main/java/bisq/core/trade/statistics/TradeStatistics3StorageService.java (84)
    A core/src/main/java/bisq/core/trade/statistics/TradeStatistics3Store.java (73)
    A core/src/main/java/bisq/core/trade/statistics/TradeStatisticsConverter.java (182)
    M core/src/main/java/bisq/core/trade/statistics/TradeStatisticsForJson.java (52)
    M core/src/main/java/bisq/core/trade/statistics/TradeStatisticsManager.java (138)
    D core/src/test/java/bisq/core/trade/statistics/TradeStatistics2Maker.java (91)
    D core/src/test/java/bisq/core/trade/statistics/TradeStatistics2Test.java (45)
    D core/src/test/java/bisq/core/trade/statistics/TradeStatisticsManagerTest.java (114)
    M daemon/src/main/java/bisq/daemon/grpc/GrpcGetTradeStatisticsService.java (4)
    M desktop/src/main/java/bisq/desktop/main/dao/economy/dashboard/BsqDashboardView.java (32)
    M desktop/src/main/java/bisq/desktop/main/debug/DebugView.java (8)
    M desktop/src/main/java/bisq/desktop/main/market/MarketView.java (32)
    M desktop/src/main/java/bisq/desktop/main/market/trades/TradesChartsView.java (96)
    M desktop/src/main/java/bisq/desktop/main/market/trades/TradesChartsViewModel.java (33)
    M desktop/src/main/java/bisq/desktop/main/offer/MutableOfferDataModel.java (6)
    M desktop/src/main/resources/logback.xml (3)
    M desktop/src/test/java/bisq/desktop/main/market/trades/TradesChartsViewModelTest.java (62)
    M monitor/src/main/java/bisq/monitor/Monitor.java (3)
    M monitor/src/main/java/bisq/monitor/metric/P2PMarketStats.java (6)
    M monitor/src/main/java/bisq/monitor/metric/P2PSeedNodeSnapshot.java (6)
    M p2p/src/main/java/bisq/network/p2p/P2PService.java (10)
    M p2p/src/main/java/bisq/network/p2p/storage/persistence/MapStoreService.java (2)
    D p2p/src/main/resources/TradeStatistics2Store_BTC_MAINNET (3)
    M proto/src/main/proto/grpc.proto (2)
    M proto/src/main/proto/pb.proto (83)

-- Patch Links --

https://github.com/bisq-network/bisq/pull/4599.patch
https://github.com/bisq-network/bisq/pull/4599.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/bisq-network/bisq/pull/4599
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bisq.network/pipermail/bisq-github/attachments/20201005/c9f8fc23/attachment-0001.html>