[bisq-network/bisq] Redesign how we persist json files for explorer nodes (#4670)
chimp1984
notifications at github.com
Mon Oct 19 20:18:21 UTC 2020
Currently we write at each new block all data to disk. This is about 1 GB and causes disk IO issues. We use mutable json data which also have different structure than Bisq DAO model data. This was to fulfill the needs of the original explorer which did not contain any business logic and internal model.
Now we write block chain data as immutable json data and add mutable data structures for the mutable data. Thus disk IO will decrease dramatically. We removed the json wrapper classes and use the Bisq model classes as input for the automatic json conversion. The explorer web app need to maintain its own internal model (it does already according to @softsimon) and create the missing information by lookups of the different models.
The `bisq.core.dao.state.DaoStateService` is the main class where that business logic is implemented.
The main model in Bisq is `bisq.core.dao.state.model.DaoState` and the immutable blockchain models are in `bisq.core.dao.state.model.blockchain`.
Based on release 1.4.0 branch, merged current master into it. Rebased....
You can view, comment on, or merge this pull request online at:
https://github.com/bisq-network/bisq/pull/4670
-- Commit Summary --
* Bump version number for v1.4.0
* Update translations for v1.4.0
* Create a P2WPKH keychain for new btc wallets
* Add a P2WPKH keychain for existing wallets
* AddressEntry: Add boolean segwit flag
* Stop using LegacyAddress for btc addresses
* Fix log msg in BtcCoinSelector
* Comment out segwit BSQ account path
* TradeWalletService: adapt to segwit wallet
* WalletService: adapt to segwit wallet
* New AddressEntry: use different script types
* AddressEntryList: arbitrator entry use P2PKH
* Add segwit/legacy checbox for address creation
* Serialize tx without segwit
* Don't create an extra address at startup
* Don't create a wallet address when not needed
* Remove unused WalletService.findKeyFromPubKeyHash()
* Remove unused import
* Enable reusing unused AVAILABLE entries
* Make codacy happy
* Validate AddressEntry.segwit
* Make it clear segwit is not used for the trade protocol yet.
* BtcWalletService.getFreshAddressEntry(): code clean up
* Construct dummy outputs with LegacyAddress
* setWitness(): Code clean up
* Use try-with-resources
* Improve error handling for P2WPKH
* Switch back to LegacyAddress for fee estimation
* Fix add segwit keychain for encrypted wallet
* Use bitcoinj 0.15.8 (commit a733034)
* Do a backup of the wallet before segwit migration
* Check migratedWalletToSegwit is true
* Merge pull request #4568 from oscarguindzberg/segwitWallet
* Add NO_ADDRESS_PRE_FIX capability.
* Cleanups
* Make onRemoved default in interface
* Add methods for getting peers capabilities
* Remove verification for address prefix
* Clear capabilitiesListeners at shutdown
* Refactor: move SupportedCapabilitiesMessage handling code out to a method
* Use only node address for equals and hashcode
* Pass supportedCapabilities to PeerManager. Not further processed yet, will be done in next commits
* Use getSingleThreadListeningExecutor, cleanups
* Apply code inspection suggestions
* Add findPeersCapabilities method
* Add getDateAsLong method, add setter for capabilities
* Add findPeersCapabilities method
* Replace persistedPeers with peerList
* Use getPersistedPeers for peerList.getList() calls
* Update common/src/main/java/bisq/common/app/Capability.java
* Update p2p/src/main/java/bisq/network/p2p/network/Connection.java
* Fix incorrect handling of decryptedEntries size
* Refactor: Return early
* Refactor: Rearrange code, remove unused methods, renamings (no functional change)
* Refactor: Rename method
* Use connection.getPeersNodeAddressOptional().isPresent() instead of connection.hasPeersNodeAddress() which does the same internally
* Fix tests
* Use a hashset instead of list to avoid duplicates. Filter out my own node from persisted peers.
* Decrease failedConnectionAttempts onConnection
* Fix incorrect collection used in == 1 check
* Add isPresent check
* Do not log size as we don't want to call potentially expensive toProtoMessage method
* Copy peers in a new hashset to avoid concurrent modification exc at serialisation
* Dont reassign param
* Use custom class MailboxItem instead of Tuple
* Apply code review suggestions
* When updating the capability from a reported peer we check if the reported one has higher capabilities, otherwise we ignore it.
* Apply codacy suggestions
* Add TradeStatistics3 and related classes
* Remove PublishTradeStatistics from buyer protocol
* Refactor: Move class
* Refactor: Rename class
* let seller publish trade statistics only if peer is updated user. If not the peer will publish only.
* Delete TradeStatistics (version 1)
* We "hack" TradeStatistics2StorageService to fulfill our needs:
* Apply TradeStatistics3 to TradeStatisticsManager and some related classes
* Apply TradeStatistics3 to client classes
* Adjust tests, remove tests which do not make sense anymore
* Use TradeStatistics3 in protobuf file
* Remove resource file
* Prune mediator and refund agent entries for all entries beside the last 100 we use for the selection algorithm.
* Add injector.getInstance(TradeStatisticsConverter.class) to BisqExecutable to enforce inclusion.
* Do conversion in a thread to not block UI thread. takes about 4 seconds on my machine.
* Add filter for excluding null objects
* Add check that size is > LOOK_BACK_RANGE
* Add resource file for 1.4.0 (should be updated at release time)
* Apply codacy suggestions
* Remove copy&past mistake
* Republish trade statistics from seller side if peer capability is know. This is not the case without getting PR #4609 merges as well.
* Remove comment line
* Remove unused variable
* Cleanups: Remove outdated TODOs, fix typos
* Merge pull request #4610 from chimp1984/remove-address-prefix
* Merge pull request #4611 from chimp1984/new-trade-statistics
* Set memo to tx after tx creation and not on broadcast success, as broadcast success might not get called (even it is broadcast -> pending btcj bug)
* Only show xmr auto conf label for sell offers
* Use hash at conversion also for local data to avoid duplicates.
* Use aesKey in Wallet.toString()
* Merge pull request #4627 from oscarguindzberg/fixWalletToString
* Merge pull request #4626 from chimp1984/avoid-duplicate-trade-stats
* Merge pull request #4625 from chimp1984/only-showXmrAutoConf-for-sell-offers
* Merge pull request #4619 from chimp1984/fix-memo-bug-issue-4616
* Stop PeerGroup only if running
* Change write json files to disk strategy
* Merge pull request #4635 from chimp1984/fix-performance-issues-at-dump-dao-data
* Merge pull request #4631 from oscarguindzberg/fixPeerGroupStop
* Fix bug at mediation with old client
* Merge pull request #4638 from chimp1984/fix-mediation-bug
* Exclude depositTxId as it seems that leads to
* Update resource file.
* Merge pull request #4640 from chimp1984/update-trade-stat-resource-file
* Accept segwit addresses when sending non-BSQ funds
* Merge pull request #4632 from oscarguindzberg/fixSendNonBsq
* Change data response behaviour
* Remove lombok getter for date and rename getDate to getDateAsLong and getTradeDate to getDate.
* Add GetInventory messages
* Improve GetInventoryRequester and GetInventoryRequestManager
* Merge pull request #4642 from chimp1984/change-handling-of-trade-stats-in-data-response
* Merge pull request #4643 from chimp1984/add-get-inventory-msg
* Replace emzy's v2 seednodes with new v3 seednodes
* Resolve conflict in 'core/src/main/resources/btc_mainnet.seednodes'.
* Use toString for NullPointerException
* Add dont show again check box to validation exception popups
* Dont include dead transactions in check for unconfirmed txs chain
* Exclude time-locked txs at isUnconfirmedTransactionsLimitHit
* Merge pull request #4647 from chimp1984/persist-seen-mediator-warnings
* Merge pull request #4648 from chimp1984/dont-include-dead-txs-in-limit-check
* Update bitcoinj checkpoints for v1.4.0
* Only remove offer locally when necessary
* Merge pull request #4651 from ripcurlx/keep-offer-when-canceling
* Update data stores for v1.4.0
* Revert jdk.module.illegalAccess=deny
* Revert to SNAPSHOT version
* Handle Capabilities for encrypted messages (offer availibility request/response)
* Add file name to temp file at write to disk
* Fix min height for trade statistics table
* Republish trade statistics if not found in existing trade stats.
* Merge pull request #4659 from chimp1984/fix-missing-capability-handling-for-encrypted-msg
* Merge pull request #4662 from chimp1984/fix-republish-trade-stats
* Fix register dipute agent test assert
* Add new 'getoffer offer-id' api method
* Wrap Exception in gRPC StatusRuntimeException
* Make URLs in popup notices clickable
* Tidy up getOffer(id)
* Remove unused strings, ref:
* Remove getAllAddressesFromActiveKeys()
* Write block chain data as immutable json data and add mutable data structures for the mutable data
-- File Changes --
M apitest/src/test/java/bisq/apitest/method/MethodTest.java (11)
M apitest/src/test/java/bisq/apitest/method/RegisterDisputeAgentsTest.java (2)
M apitest/src/test/java/bisq/apitest/method/offer/AbstractCreateOfferTest.java (4)
M apitest/src/test/java/bisq/apitest/method/offer/CreateOfferUsingFixedPriceTest.java (67)
M apitest/src/test/java/bisq/apitest/method/offer/CreateOfferUsingMarketPriceMarginTest.java (88)
M apitest/src/test/java/bisq/apitest/method/offer/ValidateCreateOfferTest.java (1)
M build.gradle (4)
M cli/src/main/java/bisq/cli/CliMain.java (16)
M common/src/main/java/bisq/common/app/Capabilities.java (18)
M common/src/main/java/bisq/common/app/Capability.java (4)
M common/src/main/java/bisq/common/app/Version.java (3)
M common/src/main/java/bisq/common/consensus/UsedForTradeContractJson.java (3)
M common/src/main/java/bisq/common/crypto/Encryption.java (1)
M common/src/main/java/bisq/common/crypto/KeyStorage.java (1)
M common/src/main/java/bisq/common/file/JsonFileManager.java (103)
M common/src/main/java/bisq/common/persistence/PersistenceManager.java (2)
M common/src/main/java/bisq/common/proto/network/NetworkEnvelope.java (1)
M common/src/test/java/bisq/common/app/CapabilitiesTest.java (42)
M core/src/main/java/bisq/core/api/CoreApi.java (8)
M core/src/main/java/bisq/core/api/CoreOffersService.java (8)
M core/src/main/java/bisq/core/app/BisqExecutable.java (2)
M core/src/main/java/bisq/core/app/BisqSetup.java (2)
M core/src/main/java/bisq/core/app/WalletAppSetup.java (2)
M core/src/main/java/bisq/core/app/misc/ExecutableForAppWithP2p.java (4)
M core/src/main/java/bisq/core/btc/model/AddressEntry.java (30)
M core/src/main/java/bisq/core/btc/model/AddressEntryList.java (30)
M core/src/main/java/bisq/core/btc/setup/BisqKeyChainGroupStructure.java (12)
M core/src/main/java/bisq/core/btc/setup/WalletConfig.java (104)
M core/src/main/java/bisq/core/btc/setup/WalletsSetup.java (9)
M core/src/main/java/bisq/core/btc/wallet/BsqWalletService.java (22)
M core/src/main/java/bisq/core/btc/wallet/BtcCoinSelector.java (2)
M core/src/main/java/bisq/core/btc/wallet/BtcWalletService.java (72)
M core/src/main/java/bisq/core/btc/wallet/TradeWalletService.java (69)
M core/src/main/java/bisq/core/btc/wallet/WalletService.java (67)
M core/src/main/java/bisq/core/dao/governance/asset/AssetService.java (6)
M core/src/main/java/bisq/core/dao/governance/blindvote/network/RepublishGovernanceDataHandler.java (2)
M core/src/main/java/bisq/core/dao/node/BsqNode.java (6)
M core/src/main/java/bisq/core/dao/node/explorer/ExportJsonFilesService.java (294)
D core/src/main/java/bisq/core/dao/node/explorer/JsonBlock.java (31)
D core/src/main/java/bisq/core/dao/node/explorer/JsonBlocks.java (28)
D core/src/main/java/bisq/core/dao/node/explorer/JsonScriptPubKey.java (43)
D core/src/main/java/bisq/core/dao/node/explorer/JsonSpentInfo.java (35)
D core/src/main/java/bisq/core/dao/node/explorer/JsonTx.java (86)
D core/src/main/java/bisq/core/dao/node/explorer/JsonTxInput.java (33)
D core/src/main/java/bisq/core/dao/node/explorer/JsonTxOutput.java (119)
D core/src/main/java/bisq/core/dao/node/explorer/JsonTxOutputType.java (51)
D core/src/main/java/bisq/core/dao/node/explorer/JsonTxType.java (48)
M core/src/main/java/bisq/core/dao/node/full/FullNode.java (2)
M core/src/main/java/bisq/core/dao/node/lite/LiteNode.java (13)
M core/src/main/java/bisq/core/dao/node/lite/network/LiteNodeNetworkService.java (2)
M core/src/main/java/bisq/core/dao/state/DaoStateService.java (12)
M core/src/main/java/bisq/core/offer/OfferBookService.java (2)
M core/src/main/java/bisq/core/offer/availability/DisputeAgentSelection.java (19)
M core/src/main/java/bisq/core/proto/CoreProtoResolver.java (3)
M core/src/main/java/bisq/core/proto/network/CoreNetworkProtoResolver.java (11)
M core/src/main/java/bisq/core/proto/persistable/CorePersistenceProtoResolver.java (3)
M core/src/main/java/bisq/core/provider/fee/FeeService.java (1)
M core/src/main/java/bisq/core/provider/price/PriceFeedService.java (17)
M core/src/main/java/bisq/core/setup/CoreNetworkCapabilities.java (4)
M core/src/main/java/bisq/core/support/dispute/Dispute.java (2)
M core/src/main/java/bisq/core/support/dispute/agent/DisputeAgentManager.java (2)
M core/src/main/java/bisq/core/support/dispute/arbitration/ArbitrationManager.java (2)
M core/src/main/java/bisq/core/trade/DumpDelayedPayoutTx.java (2)
M core/src/main/java/bisq/core/trade/TradeDataValidation.java (47)
M core/src/main/java/bisq/core/trade/TradeManager.java (13)
M core/src/main/java/bisq/core/trade/TradeModule.java (4)
M core/src/main/java/bisq/core/trade/protocol/BuyerProtocol.java (4)
M core/src/main/java/bisq/core/trade/protocol/SellerProtocol.java (6)
M core/src/main/java/bisq/core/trade/protocol/TradeProtocol.java (5)
D core/src/main/java/bisq/core/trade/protocol/tasks/PublishTradeStatistics.java (80)
A core/src/main/java/bisq/core/trade/protocol/tasks/seller/SellerPublishesTradeStatistics.java (73)
D core/src/main/java/bisq/core/trade/statistics/TradeStatistics.java (236)
M core/src/main/java/bisq/core/trade/statistics/TradeStatistics2.java (37)
M core/src/main/java/bisq/core/trade/statistics/TradeStatistics2StorageService.java (23)
A core/src/main/java/bisq/core/trade/statistics/TradeStatistics3.java (413)
A core/src/main/java/bisq/core/trade/statistics/TradeStatistics3StorageService.java (84)
A core/src/main/java/bisq/core/trade/statistics/TradeStatistics3Store.java (73)
A core/src/main/java/bisq/core/trade/statistics/TradeStatisticsConverter.java (179)
M core/src/main/java/bisq/core/trade/statistics/TradeStatisticsForJson.java (52)
M core/src/main/java/bisq/core/trade/statistics/TradeStatisticsManager.java (181)
M core/src/main/resources/btc_mainnet.seednodes (8)
M core/src/main/resources/i18n/displayStrings.properties (84)
M core/src/main/resources/i18n/displayStrings_de.properties (187)
M core/src/main/resources/i18n/displayStrings_es.properties (217)
M core/src/main/resources/i18n/displayStrings_fa.properties (189)
M core/src/main/resources/i18n/displayStrings_fr.properties (195)
M core/src/main/resources/i18n/displayStrings_ja.properties (287)
M core/src/main/resources/i18n/displayStrings_pt-br.properties (187)
M core/src/main/resources/i18n/displayStrings_pt.properties (187)
M core/src/main/resources/i18n/displayStrings_ru.properties (187)
M core/src/main/resources/i18n/displayStrings_th.properties (187)
M core/src/main/resources/i18n/displayStrings_vi.properties (187)
M core/src/main/resources/i18n/displayStrings_zh-hans.properties (299)
M core/src/main/resources/i18n/displayStrings_zh-hant.properties (303)
M core/src/main/resources/wallet/checkpoints.txt (8)
D core/src/test/java/bisq/core/trade/statistics/TradeStatistics2Maker.java (91)
D core/src/test/java/bisq/core/trade/statistics/TradeStatistics2Test.java (45)
D core/src/test/java/bisq/core/trade/statistics/TradeStatisticsManagerTest.java (114)
M daemon/src/main/java/bisq/daemon/grpc/GrpcGetTradeStatisticsService.java (4)
M daemon/src/main/java/bisq/daemon/grpc/GrpcOffersService.java (19)
M desktop/package/linux/Dockerfile (2)
M desktop/package/linux/package.sh (3)
M desktop/package/linux/release.sh (2)
M desktop/package/macosx/Info.plist (4)
M desktop/package/macosx/create_app.sh (3)
M desktop/package/macosx/finalize.sh (2)
M desktop/package/macosx/insert_snapshot_version.sh (2)
M desktop/package/macosx/replace_version_number.sh (4)
M desktop/package/windows/package.bat (3)
M desktop/package/windows/release.bat (2)
M desktop/src/main/java/bisq/desktop/components/AddressTextField.java (3)
M desktop/src/main/java/bisq/desktop/components/BsqAddressTextField.java (3)
M desktop/src/main/java/bisq/desktop/components/TextFieldWithCopyIcon.java (3)
M desktop/src/main/java/bisq/desktop/main/account/register/AgentRegistrationView.java (2)
M desktop/src/main/java/bisq/desktop/main/dao/economy/dashboard/BsqDashboardView.java (41)
M desktop/src/main/java/bisq/desktop/main/debug/DebugView.java (8)
M desktop/src/main/java/bisq/desktop/main/funds/deposit/DepositView.java (25)
M desktop/src/main/java/bisq/desktop/main/funds/withdrawal/WithdrawalView.java (30)
M desktop/src/main/java/bisq/desktop/main/market/MarketView.java (32)
M desktop/src/main/java/bisq/desktop/main/market/spread/SpreadView.java (2)
M desktop/src/main/java/bisq/desktop/main/market/spread/SpreadViewModel.java (2)
M desktop/src/main/java/bisq/desktop/main/market/trades/TradesChartsView.java (100)
M desktop/src/main/java/bisq/desktop/main/market/trades/TradesChartsViewModel.java (0)
M desktop/src/main/java/bisq/desktop/main/offer/MutableOfferDataModel.java (0)
M desktop/src/main/java/bisq/desktop/main/offer/takeoffer/TakeOfferDataModel.java (0)
M desktop/src/main/java/bisq/desktop/main/offer/takeoffer/TakeOfferView.java (0)
M desktop/src/main/java/bisq/desktop/main/overlays/Overlay.java (0)
M desktop/src/main/java/bisq/desktop/main/overlays/windows/OfferDetailsWindow.java (0)
M desktop/src/main/java/bisq/desktop/main/support/dispute/agent/DisputeAgentView.java (0)
M desktop/src/main/java/bisq/desktop/util/FormBuilder.java (0)
M desktop/src/main/java/bisq/desktop/util/GUIUtil.java (0)
M desktop/src/main/resources/logback.xml (0)
M desktop/src/test/java/bisq/desktop/main/market/trades/TradesChartsViewModelTest.java (0)
M gradle/witness/gradle-witness.gradle (0)
M monitor/src/main/java/bisq/monitor/Monitor.java (0)
M monitor/src/main/java/bisq/monitor/metric/MarketStats.java (0)
M monitor/src/main/java/bisq/monitor/metric/P2PMarketStats.java (0)
M monitor/src/main/java/bisq/monitor/metric/P2PNetworkLoad.java (0)
M monitor/src/main/java/bisq/monitor/metric/P2PSeedNodeSnapshot.java (0)
M monitor/src/main/java/bisq/monitor/metric/PriceNodeStats.java (0)
M monitor/src/main/java/bisq/monitor/metric/TorRoundTripTime.java (0)
M monitor/src/main/java/bisq/monitor/metric/TorStartupTime.java (0)
M p2p/src/main/java/bisq/network/p2p/DecryptedDirectMessageListener.java (0)
M p2p/src/main/java/bisq/network/p2p/P2PService.java (0)
M p2p/src/main/java/bisq/network/p2p/PrefixedSealedAndSignedMessage.java (0)
A p2p/src/main/java/bisq/network/p2p/inventory/GetInventoryRequestHandler.java (0)
A p2p/src/main/java/bisq/network/p2p/inventory/GetInventoryRequestManager.java (0)
A p2p/src/main/java/bisq/network/p2p/inventory/GetInventoryRequester.java (0)
A p2p/src/main/java/bisq/network/p2p/inventory/messages/GetInventoryRequest.java (0)
A p2p/src/main/java/bisq/network/p2p/inventory/messages/GetInventoryResponse.java (0)
M p2p/src/main/java/bisq/network/p2p/network/Connection.java (0)
M p2p/src/main/java/bisq/network/p2p/network/NetworkNode.java (0)
M p2p/src/main/java/bisq/network/p2p/network/SynchronizedProtoOutputStream.java (0)
M p2p/src/main/java/bisq/network/p2p/peers/PeerManager.java (0)
M p2p/src/main/java/bisq/network/p2p/peers/getdata/RequestDataManager.java (0)
M p2p/src/main/java/bisq/network/p2p/peers/peerexchange/GetPeersRequestHandler.java (0)
M p2p/src/main/java/bisq/network/p2p/peers/peerexchange/Peer.java (0)
M p2p/src/main/java/bisq/network/p2p/peers/peerexchange/PeerExchangeHandler.java (0)
M p2p/src/main/java/bisq/network/p2p/peers/peerexchange/PeerExchangeManager.java (0)
M p2p/src/main/java/bisq/network/p2p/peers/peerexchange/PeerList.java (0)
M p2p/src/main/java/bisq/network/p2p/peers/peerexchange/messages/GetPeersRequest.java (0)
M p2p/src/main/java/bisq/network/p2p/peers/peerexchange/messages/GetPeersResponse.java (0)
M p2p/src/main/java/bisq/network/p2p/storage/HashMapChangedListener.java (0)
M p2p/src/main/java/bisq/network/p2p/storage/P2PDataStorage.java (0)
R p2p/src/main/java/bisq/network/p2p/storage/payload/DateSortedTruncatablePayload.java (0)
M p2p/src/main/java/bisq/network/p2p/storage/persistence/MapStoreService.java (0)
M p2p/src/main/java/bisq/network/p2p/storage/persistence/StoreService.java (0)
M p2p/src/main/resources/AccountAgeWitnessStore_BTC_MAINNET (0)
M p2p/src/main/resources/DaoStateStore_BTC_MAINNET (0)
M p2p/src/main/resources/SignedWitnessStore_BTC_MAINNET (0)
D p2p/src/main/resources/TradeStatistics2Store_BTC_MAINNET (0)
A p2p/src/main/resources/TradeStatistics3Store_1.4.0_BTC_MAINNET (0)
M p2p/src/test/java/bisq/network/p2p/MockNode.java (0)
M proto/src/main/proto/grpc.proto (0)
M proto/src/main/proto/pb.proto (0)
M relay/src/main/resources/version.txt (0)
M seednode/src/main/java/bisq/seednode/SeedNodeMain.java (0)
-- Patch Links --
https://github.com/bisq-network/bisq/pull/4670.patch
https://github.com/bisq-network/bisq/pull/4670.diff
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/bisq-network/bisq/pull/4670
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bisq.network/pipermail/bisq-github/attachments/20201019/77338420/attachment-0001.html>
More information about the bisq-github
mailing list