[bisq-network/bisq] New trade stats (#4577)

[chimp1984]

Replace TradeStatistics2 with a new much smaller TradeStatistics3 object.

It converts existing TradeStatistics2 to the new data as well as data received from the network, which get rebroadcast as new data. 

The data store got reduced from 16.5 MB to 3.6 MB. If is also much better for privacy as it only stores the bare minimum of data and does not link to the offers.

It is still early WIP, missing tests and updates in other modules. Just wanted to share early as it should be deployed with the intitial data request reduction and the persistence redesign.  
You can view, comment on, or merge this pull request online at:

  https://github.com/bisq-network/bisq/pull/4577

-- Commit Summary --

  * Initial tests
  * Version helper uses real Bisq version
  * Tests use static fixtures
  * Migration test succeeds
  * GetMap test succeeds
  * GetMapSinceFilter works
  * updateScenario works
  * freshInstall scenario works
  * put scenario works
  * Introduce SplitStore and Service to handle BL
  * Moved file handling logic to SplitStoreService
  * Activate other split data stores
  * Refactored tests
  * Initial tests for requests
  * Requests are smaller
  * Prevent adding duplicate data
  * Enforce version format and filter faulty ones
  * Test incoming special keys from the future
  * Javadoc cosmetics
  * Init integration test script
  * Persist live database after split
  * Integration test script tests stock
  * Use version history array
  * Integration test script tests new system
  * Use full sync between seednodes
  * Cleanup integration test script
  * Ignore some tests
  * Adjust P2PDataStorage tests
  * Add another migration test
  * Reactivate ignored tests
  * Refactoring and cleaning up
  * Cleanup
  * Refactored test helpers
  * Fixed a potential issue when loading stores
  * Fixed wrong filename
  * Refactoring
  * Make special key platform-independent
  * Fixed persisting wrong data store
  * Renamed data stores
  * Replace TODO comment
  * Fixed a timing issue
  * Added 1.3.6 and 1.3.7 data stores
  * Update history version list
  * Removed deprecated test mechanisms
  * Cleanup imports
  * Included PR feedback
  * Included PR feedback
  * Added 1.3.8 data stores
  * Use Version.VERSION again
  * Merge branch 'master_upstream' into chimp-reduce_initial_request_size
  * Refactor: Add getStore method to be reused by SplitStoreService
  * Use readStore(); instead of store = readStore(getFileName());
  * Apply code inspection suggestions
  * Add mapOfHistoricalStores field
  * Revert critical data stores to current version.
  * Add version field
  * Remove methods in MapStoreService and do instead instance of check in AppendOnlyDataStoreService to call the new method only where it makes sense.
  * Add current version to version list. Makes it more clear that those are all historical immutable data.
  * Remove file
  * Rename SplitStore to PersistableNetworkPayload
  * Rename for better clarity
  * Fix bug with missing replace -> use existing methods in Version for version compare.
  * Revert version to 1.3.7 to not break build
  * Fix param in test. Add Ignore to the 2 test classes handling the new historical data store feature.
  * Fix param in test.
  * Revert changes at tests from prev version
  * Merge branch 'master_upstream' into chimp-reduce_initial_request_size
  * Change dev mode price
  * Remove test files from previous version
  * Add getPersisted method
  * remove test files
  * remove test
  * Fix issues and refactor
  * Cleanup
  * Add data from non-splitStoreService as well
  * Merge branch 'master_upstream' into chimp-reduce_initial_request_size
  * Merge branch 'master_upstream' into chimp-reduce_initial_request_size
  * Merge branch 'master_upstream' into chimp-reduce_initial_request_size
  * Merge branch 'chimp-reduce_initial_request_size' of https://github.com/chimp1984/bisq into chimp-reduce_initial_request_size
  * Cleanups, renaming, add comments
  * Fix mock class to reflect real class better.
  * Fix incorrect test.
  * Remove todo, add system line break
  * Add more logs
  * Add isEmpty check
  * Do not call  super.readFromResources(postFix); as we do not have a not-versioned resource file anymore. Instead use readStore(); to create the live store.
  * Add more logs
  * Remove toProtoNetworkEnvelope call used only for debug logs
  * Only call pruneStore if we have created a new historical store from resources.
  * Fix wrong param order in log
  * Remove dev test data
  * Add more info in log
  * Rename SplitStoreService to HistoricalDataStoreService
  * Add TradeStatistics3
  * Use a set instead of a list
  * Convert TradeStatistics2 to TradeStatistics3
  * Listen to old TradeStatistics2 objects, convert and store them and rebroadcast.

-- File Changes --

    M common/src/main/java/bisq/common/app/Capability.java (3)
    M common/src/main/java/bisq/common/app/Version.java (10)
    M common/src/main/java/bisq/common/storage/Storage.java (11)
    M core/src/main/java/bisq/core/account/sign/SignedWitnessStorageService.java (2)
    M core/src/main/java/bisq/core/account/sign/SignedWitnessStore.java (11)
    M core/src/main/java/bisq/core/account/witness/AccountAgeWitnessStorageService.java (2)
    M core/src/main/java/bisq/core/account/witness/AccountAgeWitnessStore.java (11)
    M core/src/main/java/bisq/core/api/CoreApi.java (4)
    M core/src/main/java/bisq/core/dao/governance/asset/AssetService.java (6)
    M core/src/main/java/bisq/core/dao/governance/blindvote/storage/BlindVoteStore.java (11)
    M core/src/main/java/bisq/core/dao/governance/proposal/ProposalService.java (1)
    M core/src/main/java/bisq/core/dao/governance/proposal/storage/appendonly/ProposalStore.java (11)
    M core/src/main/java/bisq/core/offer/availability/DisputeAgentSelection.java (17)
    M core/src/main/java/bisq/core/proto/CoreProtoResolver.java (3)
    M core/src/main/java/bisq/core/proto/persistable/CorePersistenceProtoResolver.java (3)
    M core/src/main/java/bisq/core/provider/ProvidersRepository.java (5)
    M core/src/main/java/bisq/core/provider/price/PriceFeedService.java (17)
    M core/src/main/java/bisq/core/setup/CoreNetworkCapabilities.java (3)
    M core/src/main/java/bisq/core/support/dispute/agent/DisputeAgentService.java (5)
    M core/src/main/java/bisq/core/trade/Trade.java (1)
    M core/src/main/java/bisq/core/trade/TradeModule.java (2)
    M core/src/main/java/bisq/core/trade/protocol/tasks/PublishTradeStatistics.java (40)
    M core/src/main/java/bisq/core/trade/protocol/tasks/maker/MakerProcessesInputsForDepositTxRequest.java (6)
    M core/src/main/java/bisq/core/trade/protocol/tasks/taker/TakerSendInputsForDepositTxRequest.java (3)
    M core/src/main/java/bisq/core/trade/statistics/TradeStatistics2.java (2)
    M core/src/main/java/bisq/core/trade/statistics/TradeStatistics2StorageService.java (22)
    M core/src/main/java/bisq/core/trade/statistics/TradeStatistics2Store.java (11)
    A core/src/main/java/bisq/core/trade/statistics/TradeStatistics3.java (276)
    A core/src/main/java/bisq/core/trade/statistics/TradeStatistics3StorageService.java (74)
    A core/src/main/java/bisq/core/trade/statistics/TradeStatistics3Store.java (73)
    A core/src/main/java/bisq/core/trade/statistics/TradeStatisticsConverter.java (106)
    M core/src/main/java/bisq/core/trade/statistics/TradeStatisticsForJson.java (1)
    M core/src/main/java/bisq/core/trade/statistics/TradeStatisticsManager.java (102)
    M desktop/src/main/java/bisq/desktop/main/dao/economy/dashboard/BsqDashboardView.java (32)
    M desktop/src/main/java/bisq/desktop/main/market/trades/TradesChartsView.java (96)
    M desktop/src/main/java/bisq/desktop/main/market/trades/TradesChartsViewModel.java (26)
    M desktop/src/main/java/bisq/desktop/main/offer/MutableOfferDataModel.java (6)
    M p2p/src/main/java/bisq/network/p2p/network/Connection.java (40)
    M p2p/src/main/java/bisq/network/p2p/peers/getdata/GetDataRequestHandler.java (12)
    M p2p/src/main/java/bisq/network/p2p/peers/getdata/RequestDataHandler.java (9)
    M p2p/src/main/java/bisq/network/p2p/peers/getdata/messages/GetDataRequest.java (11)
    M p2p/src/main/java/bisq/network/p2p/peers/getdata/messages/GetUpdatedDataRequest.java (24)
    M p2p/src/main/java/bisq/network/p2p/peers/getdata/messages/PreliminaryGetDataRequest.java (34)
    M p2p/src/main/java/bisq/network/p2p/storage/P2PDataStorage.java (153)
    M p2p/src/main/java/bisq/network/p2p/storage/persistence/AppendOnlyDataStoreService.java (8)
    A p2p/src/main/java/bisq/network/p2p/storage/persistence/HistoricalDataStoreService.java (187)
    A p2p/src/main/java/bisq/network/p2p/storage/persistence/PersistableNetworkPayloadStore.java (20)
    M p2p/src/main/java/bisq/network/p2p/storage/persistence/ProtectedDataStoreService.java (6)
    M p2p/src/main/java/bisq/network/p2p/storage/persistence/StoreService.java (26)
    M p2p/src/test/java/bisq/network/p2p/storage/P2PDataStorageBuildGetDataResponseTest.java (13)
    M p2p/src/test/java/bisq/network/p2p/storage/P2PDataStorageRequestDataTest.java (8)
    M p2p/src/test/java/bisq/network/p2p/storage/mocks/AppendOnlyDataStoreServiceFake.java (8)
    M proto/src/main/proto/pb.proto (97)

-- Patch Links --

https://github.com/bisq-network/bisq/pull/4577.patch
https://github.com/bisq-network/bisq/pull/4577.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/bisq-network/bisq/pull/4577
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bisq.network/pipermail/bisq-github/attachments/20200929/626ba8c9/attachment-0001.html>