[bisq-network/bisq-p2p] Change data persitence (#8)

[Manfred Karrer]

We use at the moment one protobuffer file which gets persisted to disk (PersistableNetworkPayloadMap) for storing data which arrived from the P2P network and are from payload type PersistableNetworkPayload.
There are 2 use cases for those data: Account age witness and trade statistics.
Both are persisted in an append-only style as there is no use case for removing that data. This allows us to only use the hash of the payload as key in a map. For removal of the data we would need to support a signature and store a public key with the data which increases the data size quite a lot (specially for account age witness data we want to keep it as small as possible).

We will have new use cases from the DAO: Proposals and BlindVotes which are essential data for constructing the vote result need to be persisted as well.
We could use a custom data structure inside the DAO domain but then we would need to re-implement already existing features from the p2p network like the delta handling for requests.
At the moment the 2 payload types (account age witness and trade statistics) are stored in the same data structure/file (PersistableNetworkPayloadMap).
For the DAO we would like to have our consensus critical data more explicitly controlled and defined. 
We suggest to use domain specific storage implementations and merge the data from different sources for the P2PDataStorage as the p2p network domain is not aware of the other domains.
That will also gives us more flexibility about the way how the data are persisted (maybe use a high performance DB for those payload types which creates a lot of disk IO) and make it easier to implement solutions for better scalability (e.g. pruning of old trade statistics, or move them to a data provider for on-demand requests once the data gets too much). 

I am working on an implementation for that cahnges at https://github.com/ManfredKarrer/bisq-p2p/commits/storage-provider and would like to ask for feedback (still not complete but basics are in place).

My suggestion uses the AppendOnlyDataStoreService class which maintains a list of storageServices. 
Any domain can add a storage service and the AppendOnlyDataStoreService takes care of merging those data sources for the domain agnostic P2PDataStorage.
We can transition from the current data store to the new system by copying over the existing data entries to the new files and then having 2 new storage files: AccountAgeWitnessMap and TradeStatisticsMap (not impl. yet). 
With the DAO there will be 2 more: ProposalMap and BlindVoteMap.

We added also support for data which can be removed, which would require the public key and use different data types.
Those are handled in PersistedEntryMapService. As there is currently no use case I have not implemented the support for multiple services but it would follow the same concept as in AppendOnlyDataStoreService. I could be probably consolidated to one implementation with a bit more generics support (thought it would get a bit complicate, so I prefer to have a bit duplicated code rather then very obscure generics).
There might be use cases from the DAO for that PersistedEntryMapService but that is still not 100% clear. If there will be no use case we can remove that as well.

@freimair @sqrrm @cbeams 
What do you think about it?




-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/bisq-network/bisq-p2p/issues/8
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bisq.network/pipermail/github/attachments/20180427/47bf2c60/attachment.html>