[bisq-network/bisq] [WIP] New persistence model (#4527)

Tue Sep 15 06:50:24 UTC 2020

While reseaching for the app-freeze problem a user reported I looked closer to the changes in our persistence framework. There have been added thread-safety to PersistableEnvelope classes. While trying to figure out if those changes might cause a deadlock in some rare edge cases I came to the conclusion that the overall model is not reflecting well our requirements. Worse it had a lot of resource waste by creating new threads very frequently which got terminated mostly without doing actual work.
Beside that I questioned the need to persist the data that frequently. We read data only at startup (with a few irrelevant exceptions). Writing the data at shutdown would be sufficient for most type of data. The resource heavy data are the DaoState and the Trade statistics as well as SequenceNumberMap which gets updated very frequently. All those data have very low priority and can be recovered in case of a sever crash. For more critical data like disputeList or trades we use a timer which writes to disk a minute after a persist request got called. We could improve that by calling a persistNow just at the moments when critical state changes happen (e.g. in a trade). Atm I left all the write requests as it was before and they basically trigger a request at any data change of an object.
To run those operations on the user thread avoids all the issues with threading and risk of deadlocks. The total performance is for sure better as before as it removed a lot of wasted efforts. If really needed we could introduce a threaded version as well with callback handling, but I guess that is not needed.

The changes are quite heavy over many places. Some are pure refactoring but many are also design changes, like that the data objects are not responsible themselves anymore for persistence but their owner (e.g. trademanager not trade). So that made the overall architecture more clear.

I leave it WIP for now as it needs much more testing and I might add persistNow calls for the critical state changes and might eliminate request calls for  irrelevant data changes. 

You can view, comment on, or merge this pull request online at:

  https://github.com/bisq-network/bisq/pull/4527

-- Commit Summary --

  * Move read code from file manager to storage
  * Remove FileManager class
  * Replace init methods with getPersisted
  * Change API for Storage
  * Use Userthread executor instead of new thread (WIP)
  * Flush to disk at shutdown
  * Refactor: Rename class
  * Refactor: Rename package
  * Refactor: Rename queueUpForSave to persistAtShutDown
  * Add Priority enum
  * Remove removeAndBackupFile call
  * Change PersistenceManager.Priority
  * Refactor: Rename method and variable
  * Add timer to write to disk if data has high prio
  * Change log level

-- File Changes --

    M common/src/main/java/bisq/common/crypto/KeyStorage.java (2)
    R common/src/main/java/bisq/common/persistence/CorruptedDatabaseFilesHandler.java (2)
    R common/src/main/java/bisq/common/persistence/FileUtil.java (21)
    R common/src/main/java/bisq/common/persistence/JsonFileManager.java (8)
    A common/src/main/java/bisq/common/persistence/PersistenceManager.java (306)
    R common/src/main/java/bisq/common/persistence/ResourceNotFoundException.java (2)
    M common/src/main/java/bisq/common/proto/persistable/PersistableEnvelope.java (4)
    M common/src/main/java/bisq/common/proto/persistable/PersistableList.java (62)
    D common/src/main/java/bisq/common/proto/persistable/ThreadedPersistableEnvelope.java (44)
    D common/src/main/java/bisq/common/proto/persistable/UserThreadMappedPersistableEnvelope.java (45)
    D common/src/main/java/bisq/common/proto/persistable/UserThreadMappedPersistableList.java (31)
    D common/src/main/java/bisq/common/storage/FileManager.java (252)
    D common/src/main/java/bisq/common/storage/Storage.java (183)
    M core/src/main/java/bisq/core/account/sign/SignedWitnessStorageService.java (13)
    M core/src/main/java/bisq/core/account/sign/SignedWitnessStore.java (4)
    M core/src/main/java/bisq/core/account/witness/AccountAgeWitnessStorageService.java (13)
    M core/src/main/java/bisq/core/account/witness/AccountAgeWitnessStore.java (4)
    M core/src/main/java/bisq/core/app/AvoidStandbyModeService.java (4)
    M core/src/main/java/bisq/core/app/BisqExecutable.java (29)
    M core/src/main/java/bisq/core/app/BisqHeadlessApp.java (2)
    M core/src/main/java/bisq/core/app/TorSetup.java (2)
    M core/src/main/java/bisq/core/app/misc/AppSetup.java (3)
    M core/src/main/java/bisq/core/app/misc/ExecutableForAppWithP2p.java (49)
    M core/src/main/java/bisq/core/btc/model/AddressEntryList.java (27)
    M core/src/main/java/bisq/core/btc/setup/WalletsSetup.java (2)
    M core/src/main/java/bisq/core/dao/governance/ballot/BallotListService.java (17)
    M core/src/main/java/bisq/core/dao/governance/blindvote/MyBlindVoteList.java (4)
    M core/src/main/java/bisq/core/dao/governance/blindvote/MyBlindVoteListService.java (17)
    M core/src/main/java/bisq/core/dao/governance/blindvote/storage/BlindVoteStorageService.java (8)
    M core/src/main/java/bisq/core/dao/governance/blindvote/storage/BlindVoteStore.java (4)
    M core/src/main/java/bisq/core/dao/governance/bond/reputation/MyReputationList.java (4)
    M core/src/main/java/bisq/core/dao/governance/bond/reputation/MyReputationListService.java (16)
    M core/src/main/java/bisq/core/dao/governance/myvote/MyVoteList.java (4)
    M core/src/main/java/bisq/core/dao/governance/myvote/MyVoteListService.java (19)
    M core/src/main/java/bisq/core/dao/governance/proofofburn/MyProofOfBurnList.java (4)
    M core/src/main/java/bisq/core/dao/governance/proofofburn/MyProofOfBurnListService.java (16)
    M core/src/main/java/bisq/core/dao/governance/proposal/MyProposalList.java (4)
    M core/src/main/java/bisq/core/dao/governance/proposal/MyProposalListService.java (17)
    M core/src/main/java/bisq/core/dao/governance/proposal/storage/appendonly/ProposalStorageService.java (8)
    M core/src/main/java/bisq/core/dao/governance/proposal/storage/appendonly/ProposalStore.java (4)
    M core/src/main/java/bisq/core/dao/governance/proposal/storage/temp/TempProposalStorageService.java (8)
    M core/src/main/java/bisq/core/dao/governance/proposal/storage/temp/TempProposalStore.java (4)
    M core/src/main/java/bisq/core/dao/monitoring/DaoStateMonitoringService.java (4)
    M core/src/main/java/bisq/core/dao/node/explorer/ExportJsonFilesService.java (4)
    M core/src/main/java/bisq/core/dao/state/DaoStateStorageService.java (47)
    M core/src/main/java/bisq/core/dao/state/DaoStateStore.java (4)
    M core/src/main/java/bisq/core/dao/state/model/governance/BallotList.java (4)
    M core/src/main/java/bisq/core/dao/state/unconfirmed/UnconfirmedBsqChangeOutputList.java (4)
    M core/src/main/java/bisq/core/dao/state/unconfirmed/UnconfirmedBsqChangeOutputListService.java (17)
    M core/src/main/java/bisq/core/notifications/alerts/DisputeMsgEvents.java (6)
    M core/src/main/java/bisq/core/offer/OfferBookService.java (2)
    M core/src/main/java/bisq/core/offer/OpenOffer.java (24)
    M core/src/main/java/bisq/core/offer/OpenOfferManager.java (49)
    M core/src/main/java/bisq/core/payment/PaymentAccountList.java (4)
    M core/src/main/java/bisq/core/proto/persistable/CorePersistenceProtoResolver.java (18)
    M core/src/main/java/bisq/core/support/dispute/Dispute.java (27)
    M core/src/main/java/bisq/core/support/dispute/DisputeList.java (73)
    M core/src/main/java/bisq/core/support/dispute/DisputeListService.java (35)
    M core/src/main/java/bisq/core/support/dispute/DisputeManager.java (37)
    M core/src/main/java/bisq/core/support/dispute/agent/MultipleHolderNameDetection.java (4)
    M core/src/main/java/bisq/core/support/dispute/arbitration/ArbitrationDisputeList.java (34)
    M core/src/main/java/bisq/core/support/dispute/arbitration/ArbitrationDisputeListService.java (13)
    M core/src/main/java/bisq/core/support/dispute/arbitration/ArbitrationManager.java (3)
    M core/src/main/java/bisq/core/support/dispute/mediation/MediationDisputeList.java (29)
    M core/src/main/java/bisq/core/support/dispute/mediation/MediationDisputeListService.java (8)
    M core/src/main/java/bisq/core/support/dispute/mediation/MediationManager.java (2)
    M core/src/main/java/bisq/core/support/dispute/refund/RefundDisputeList.java (35)
    M core/src/main/java/bisq/core/support/dispute/refund/RefundDisputeListService.java (8)
    M core/src/main/java/bisq/core/support/dispute/refund/RefundManager.java (2)
    M core/src/main/java/bisq/core/support/traderchat/TraderChatManager.java (1)
    M core/src/main/java/bisq/core/trade/BuyerAsMakerTrade.java (5)
    M core/src/main/java/bisq/core/trade/BuyerAsTakerTrade.java (6)
    M core/src/main/java/bisq/core/trade/BuyerTrade.java (5)
    M core/src/main/java/bisq/core/trade/DumpDelayedPayoutTx.java (2)
    M core/src/main/java/bisq/core/trade/SellerAsMakerTrade.java (5)
    M core/src/main/java/bisq/core/trade/SellerAsTakerTrade.java (6)
    M core/src/main/java/bisq/core/trade/SellerTrade.java (5)
    M core/src/main/java/bisq/core/trade/TradableList.java (72)
    M core/src/main/java/bisq/core/trade/Trade.java (31)
    M core/src/main/java/bisq/core/trade/TradeManager.java (40)
    M core/src/main/java/bisq/core/trade/closed/ClosedTradableManager.java (36)
    M core/src/main/java/bisq/core/trade/failed/FailedTradesManager.java (30)
    M core/src/main/java/bisq/core/trade/statistics/TradeStatistics2StorageService.java (14)
    M core/src/main/java/bisq/core/trade/statistics/TradeStatistics2Store.java (4)
    M core/src/main/java/bisq/core/trade/statistics/TradeStatisticsManager.java (2)
    M core/src/main/java/bisq/core/user/Preferences.java (21)
    M core/src/main/java/bisq/core/user/PreferencesPayload.java (4)
    M core/src/main/java/bisq/core/user/User.java (22)
    M core/src/main/java/bisq/core/user/UserPayload.java (4)
    M core/src/test/java/bisq/core/crypto/EncryptionTest.java (2)
    M core/src/test/java/bisq/core/crypto/SigTest.java (2)
    M core/src/test/java/bisq/core/dao/governance/ballot/BallotListServiceTest.java (4)
    M core/src/test/java/bisq/core/dao/governance/proposal/MyProposalListServiceTest.java (6)
    M core/src/test/java/bisq/core/offer/OpenOfferManagerTest.java (20)
    M core/src/test/java/bisq/core/trade/TradableListTest.java (8)
    M core/src/test/java/bisq/core/user/PreferencesTest.java (15)
    M desktop/src/main/java/bisq/desktop/Navigation.java (14)
    M desktop/src/main/java/bisq/desktop/main/MainViewModel.java (2)
    M desktop/src/main/java/bisq/desktop/main/SharedPresentation.java (2)
    M desktop/src/main/java/bisq/desktop/main/account/content/altcoinaccounts/AltCoinAccountsDataModel.java (2)
    M desktop/src/main/java/bisq/desktop/main/account/content/backup/BackupView.java (2)
    M desktop/src/main/java/bisq/desktop/main/account/content/fiataccounts/FiatAccountsDataModel.java (2)
    M desktop/src/main/java/bisq/desktop/main/overlays/windows/ContractWindow.java (4)
    M desktop/src/main/java/bisq/desktop/main/overlays/windows/DisputeSummaryWindow.java (8)
    M desktop/src/main/java/bisq/desktop/main/overlays/windows/downloadupdate/DownloadTask.java (2)
    M desktop/src/main/java/bisq/desktop/main/portfolio/pendingtrades/PendingTradesDataModel.java (7)
    M desktop/src/main/java/bisq/desktop/main/support/dispute/DisputeView.java (5)
    M desktop/src/main/java/bisq/desktop/main/support/dispute/agent/DisputeAgentView.java (2)
    M desktop/src/main/java/bisq/desktop/main/support/dispute/client/DisputeClientView.java (2)
    M desktop/src/main/java/bisq/desktop/util/GUIUtil.java (54)
    M desktop/src/test/java/bisq/desktop/GuiceSetupTest.java (6)
    M desktop/src/test/java/bisq/desktop/main/portfolio/editoffer/EditOfferDataModelTest.java (2)
    M desktop/src/test/java/bisq/desktop/maker/PreferenceMakers.java (6)
    M monitor/src/main/java/bisq/monitor/metric/P2PMarketStats.java (11)
    M monitor/src/main/java/bisq/monitor/metric/P2PNetworkLoad.java (6)
    M monitor/src/main/java/bisq/monitor/metric/P2PSeedNodeSnapshot.java (11)
    M p2p/src/main/java/bisq/network/p2p/network/TorMode.java (12)
    M p2p/src/main/java/bisq/network/p2p/peers/PeerManager.java (35)
    M p2p/src/main/java/bisq/network/p2p/peers/peerexchange/PeerList.java (13)
    M p2p/src/main/java/bisq/network/p2p/storage/P2PDataStorage.java (22)
    M p2p/src/main/java/bisq/network/p2p/storage/persistence/MapStoreService.java (8)
    M p2p/src/main/java/bisq/network/p2p/storage/persistence/SequenceNumberMap.java (11)
    M p2p/src/main/java/bisq/network/p2p/storage/persistence/StoreService.java (46)
    M p2p/src/test/java/bisq/network/crypto/EncryptionServiceTests.java (2)
    M p2p/src/test/java/bisq/network/p2p/MockNode.java (11)
    M p2p/src/test/java/bisq/network/p2p/storage/TestState.java (39)
    M p2p/src/test/java/bisq/network/p2p/storage/mocks/MapStoreServiceFake.java (4)
    M seednode/src/main/resources/logback.xml (3)
    M statsnode/src/main/resources/logback.xml (3)

-- Patch Links --

https://github.com/bisq-network/bisq/pull/4527.patch
https://github.com/bisq-network/bisq/pull/4527.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/bisq-network/bisq/pull/4527
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bisq.network/pipermail/bisq-github/attachments/20200914/9ad4973b/attachment-0001.html>