[bisq-network/bisq] New trade statistics (#4611)

chimp1984 notifications at github.com
Thu Oct 8 23:53:36 UTC 2020


Replaces #4599

Based on #4610

We replace the old trade statistics which a minimal version and reduce data size from 16.5 MB to 3.9 MB. The old data gets converted to the new format.

Not updated users will not see new trade statistics and old ones only if they have peers with the old version as well. As all seed nodes will run the updated version they will not provide the old statistics. So for those users it will look weird that trade volume goes to zero over time. But I think that is an acceptable price and the alternative would be higher complexity by supporting boths trade statistics still. As 1.4.0 has so many important features I guess most active users will update quickly anyway.

If 2 traders with an old version trade they will publish both the statistics and new nodes will convert them to new trade stats using the hash of the old as hash for the new and so avoiding duplicates.
If 1 of the peers has update and the other not it will be detected via capabilities. In the new version onlt the seller publishes. So if the new version is the buyer he will not publish anyway, so we have only the old version publsihed and get converted to the new version by receiving peers. If the updated user is the seller he will check the capability of the peer and if its an old verison he will no publish so that the buyer with the old version publishes. If both users have updated then only the seller publishes.
So we have only one scenario where both publish (both have not updated) and we solve that with using the hash.

At startup we convert the existing trades stats to the new version (running in a thread, takes about 4 sec). After that we delete the old trade stats. The old trade stats will be excluded from initial data requests and response.
You can view, comment on, or merge this pull request online at:

  https://github.com/bisq-network/bisq/pull/4611

-- Commit Summary --

  * Add NO_ADDRESS_PRE_FIX capability.
  * Cleanups
  * Make onRemoved default in interface
  * Add methods for getting peers capabilities
  * Remove verification for address prefix
  * Clear capabilitiesListeners at shutdown
  * Refactor: move SupportedCapabilitiesMessage handling code out to a method
  * Use only node address for equals and hashcode
  * Pass supportedCapabilities to PeerManager. Not further processed yet, will be done in next commits
  * Use getSingleThreadListeningExecutor, cleanups
  * Apply code inspection suggestions
  * Add findPeersCapabilities method
  * Add getDateAsLong method, add setter for capabilities
  * Add findPeersCapabilities method
  * Replace persistedPeers with peerList
  * Use getPersistedPeers for peerList.getList() calls
  * Update common/src/main/java/bisq/common/app/Capability.java
  * Update p2p/src/main/java/bisq/network/p2p/network/Connection.java
  * Fix incorrect handling of decryptedEntries size
  * Refactor: Return early
  * Refactor: Rearrange code, remove unused methods, renamings (no functional change)
  * Refactor: Rename method
  * Use connection.getPeersNodeAddressOptional().isPresent() instead of connection.hasPeersNodeAddress() which does the same internally
  * Fix tests
  * Use a hashset instead of list to avoid duplicates. Filter out my own node from persisted peers.
  * Decrease failedConnectionAttempts onConnection
  * Fix incorrect collection used in == 1 check
  * Add isPresent check
  * Do not log size as we don't want to call potentially expensive toProtoMessage method
  * Copy peers in a new hashset to avoid concurrent modification exc at serialisation
  * Dont reassign param
  * Use custom class MailboxItem instead of Tuple
  * Apply code review suggestions
  * When updating the capability from a reported peer we check if the reported one has higher capabilities, otherwise we ignore it.
  * Apply codacy suggestions
  * Add TradeStatistics3 and related classes
  * Remove PublishTradeStatistics from buyer protocol
  * Refactor: Move class
  * Refactor: Rename class
  * let seller publish trade statistics only if peer is updated user. If not the peer will publish only.
  * Delete TradeStatistics (version 1)
  * We "hack" TradeStatistics2StorageService to fulfill our needs:
  * Apply TradeStatistics3 to TradeStatisticsManager and some related classes
  * Apply TradeStatistics3 to client classes
  * Adjust tests, remove tests which do not make sense anymore
  * Use TradeStatistics3 in protobuf file
  * Remove resource file
  * Prune mediator and refund agent entries for all entries beside the last 100 we use for the selection algorithm.
  * Add injector.getInstance(TradeStatisticsConverter.class) to BisqExecutable to enforce inclusion.
  * Do conversion in a thread to not block UI thread. takes about 4 seconds on my machine.
  * Add filter for excluding null objects
  * Add check that size is > LOOK_BACK_RANGE
  * Add resource file for 1.4.0 (should be updated at release time)
  * Apply codacy suggestions
  * Remove copy&past mistake
  * Republish trade statistics from seller side if peer capability is know. This is not the case without getting PR #4609 merges as well.
  * Remove comment line

-- File Changes --

    M common/src/main/java/bisq/common/app/Capabilities.java (18)
    M common/src/main/java/bisq/common/app/Capability.java (4)
    M common/src/main/java/bisq/common/file/JsonFileManager.java (4)
    M common/src/test/java/bisq/common/app/CapabilitiesTest.java (42)
    M core/src/main/java/bisq/core/api/CoreApi.java (4)
    M core/src/main/java/bisq/core/app/BisqExecutable.java (2)
    M core/src/main/java/bisq/core/dao/governance/asset/AssetService.java (6)
    M core/src/main/java/bisq/core/dao/governance/blindvote/network/RepublishGovernanceDataHandler.java (2)
    M core/src/main/java/bisq/core/dao/node/lite/network/LiteNodeNetworkService.java (2)
    M core/src/main/java/bisq/core/offer/availability/DisputeAgentSelection.java (19)
    M core/src/main/java/bisq/core/proto/CoreProtoResolver.java (3)
    M core/src/main/java/bisq/core/proto/network/CoreNetworkProtoResolver.java (4)
    M core/src/main/java/bisq/core/proto/persistable/CorePersistenceProtoResolver.java (3)
    M core/src/main/java/bisq/core/provider/price/PriceFeedService.java (17)
    M core/src/main/java/bisq/core/setup/CoreNetworkCapabilities.java (4)
    M core/src/main/java/bisq/core/support/dispute/agent/DisputeAgentManager.java (2)
    M core/src/main/java/bisq/core/trade/TradeModule.java (4)
    M core/src/main/java/bisq/core/trade/protocol/BuyerProtocol.java (4)
    M core/src/main/java/bisq/core/trade/protocol/SellerProtocol.java (30)
    M core/src/main/java/bisq/core/trade/protocol/TradeProtocol.java (5)
    D core/src/main/java/bisq/core/trade/protocol/tasks/PublishTradeStatistics.java (80)
    A core/src/main/java/bisq/core/trade/protocol/tasks/seller/SellerPublishesTradeStatistics.java (105)
    D core/src/main/java/bisq/core/trade/statistics/TradeStatistics.java (236)
    M core/src/main/java/bisq/core/trade/statistics/TradeStatistics2.java (2)
    M core/src/main/java/bisq/core/trade/statistics/TradeStatistics2StorageService.java (23)
    A core/src/main/java/bisq/core/trade/statistics/TradeStatistics3.java (355)
    A core/src/main/java/bisq/core/trade/statistics/TradeStatistics3StorageService.java (84)
    A core/src/main/java/bisq/core/trade/statistics/TradeStatistics3Store.java (73)
    A core/src/main/java/bisq/core/trade/statistics/TradeStatisticsConverter.java (182)
    M core/src/main/java/bisq/core/trade/statistics/TradeStatisticsForJson.java (52)
    M core/src/main/java/bisq/core/trade/statistics/TradeStatisticsManager.java (138)
    D core/src/test/java/bisq/core/trade/statistics/TradeStatistics2Maker.java (91)
    D core/src/test/java/bisq/core/trade/statistics/TradeStatistics2Test.java (45)
    D core/src/test/java/bisq/core/trade/statistics/TradeStatisticsManagerTest.java (114)
    M daemon/src/main/java/bisq/daemon/grpc/GrpcGetTradeStatisticsService.java (4)
    M desktop/src/main/java/bisq/desktop/main/dao/economy/dashboard/BsqDashboardView.java (36)
    M desktop/src/main/java/bisq/desktop/main/debug/DebugView.java (8)
    M desktop/src/main/java/bisq/desktop/main/market/MarketView.java (32)
    M desktop/src/main/java/bisq/desktop/main/market/trades/TradesChartsView.java (96)
    M desktop/src/main/java/bisq/desktop/main/market/trades/TradesChartsViewModel.java (33)
    M desktop/src/main/java/bisq/desktop/main/offer/MutableOfferDataModel.java (6)
    M desktop/src/main/resources/logback.xml (3)
    M desktop/src/test/java/bisq/desktop/main/market/trades/TradesChartsViewModelTest.java (65)
    M monitor/src/main/java/bisq/monitor/Monitor.java (3)
    M monitor/src/main/java/bisq/monitor/metric/P2PMarketStats.java (6)
    M monitor/src/main/java/bisq/monitor/metric/P2PNetworkLoad.java (2)
    M monitor/src/main/java/bisq/monitor/metric/P2PSeedNodeSnapshot.java (11)
    M p2p/src/main/java/bisq/network/p2p/DecryptedDirectMessageListener.java (2)
    M p2p/src/main/java/bisq/network/p2p/P2PService.java (332)
    M p2p/src/main/java/bisq/network/p2p/PrefixedSealedAndSignedMessage.java (7)
    M p2p/src/main/java/bisq/network/p2p/network/Connection.java (78)
    M p2p/src/main/java/bisq/network/p2p/network/NetworkNode.java (9)
    M p2p/src/main/java/bisq/network/p2p/network/SynchronizedProtoOutputStream.java (6)
    M p2p/src/main/java/bisq/network/p2p/peers/PeerManager.java (531)
    M p2p/src/main/java/bisq/network/p2p/peers/getdata/RequestDataManager.java (4)
    M p2p/src/main/java/bisq/network/p2p/peers/peerexchange/GetPeersRequestHandler.java (10)
    M p2p/src/main/java/bisq/network/p2p/peers/peerexchange/Peer.java (34)
    M p2p/src/main/java/bisq/network/p2p/peers/peerexchange/PeerExchangeHandler.java (9)
    M p2p/src/main/java/bisq/network/p2p/peers/peerexchange/PeerExchangeManager.java (8)
    M p2p/src/main/java/bisq/network/p2p/peers/peerexchange/PeerList.java (31)
    M p2p/src/main/java/bisq/network/p2p/peers/peerexchange/messages/GetPeersRequest.java (10)
    M p2p/src/main/java/bisq/network/p2p/peers/peerexchange/messages/GetPeersResponse.java (8)
    M p2p/src/main/java/bisq/network/p2p/storage/HashMapChangedListener.java (5)
    M p2p/src/main/java/bisq/network/p2p/storage/P2PDataStorage.java (6)
    M p2p/src/main/java/bisq/network/p2p/storage/persistence/MapStoreService.java (2)
    M p2p/src/main/java/bisq/network/p2p/storage/persistence/StoreService.java (5)
    D p2p/src/main/resources/TradeStatistics2Store_BTC_MAINNET (3)
    A p2p/src/main/resources/TradeStatistics3Store_1.4.0_BTC_MAINNET (3)
    M p2p/src/test/java/bisq/network/p2p/MockNode.java (2)
    M proto/src/main/proto/grpc.proto (2)
    M proto/src/main/proto/pb.proto (83)

-- Patch Links --

https://github.com/bisq-network/bisq/pull/4611.patch
https://github.com/bisq-network/bisq/pull/4611.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/bisq-network/bisq/pull/4611
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bisq.network/pipermail/bisq-github/attachments/20201008/4c9adccb/attachment-0001.html>


More information about the bisq-github mailing list