[bisq-network/bisq] Track p2p data store files using Git LFS (#4114)

[Chris Beams]

The large binary objects in p2p/src/main/resources/ are updated on every
Bisq release with the latest network data to avoid the need for new Bisq
clients to download all of this information from the network, which
would easily overload seed nodes and generally bog down the client.

This approach works well enough for its purposes, but comes with the
siginficant downside of storing all of this binary data in Git history
forever. The current version of these binary objects total about 65M,
and they grow with every release. In aggregate, this has caused the
total size of the repository to grow to 360M, making it cumbersome to
clone over a low-bandwith connection, and slowing down various local Git
operations.

To avoid further exacerbating this problem, this commit sets these files
up to be tracked via Git LFS. There's nothing we can do about the 360M
of files that already exist in history, but we can ensure it doesn't
grow in this unchecked way going forward. For an understanding of how
Git LFS works, see the reference material at [1], and see also the
sample project and README at [2].

We are using GitHub's built-in LFS service here, and it's important to
understand that there are storage and bandwidth limits in place. We have
1G total storage and 1G per month of bandwidth on the free tier. If we
exceed this, we will need to purchase a "data pack" from GitHub, which
will get us to 50G storage and bandwith. These are reasonably priced and
not the end of the world if it becomes necessary. In an attempt to avoid
this, however, the Travis CI build configuration has been updated to
cache Git LFS files, such that they are not re-downloaded on every CI
build, as this would very quickly cause us to exceed the free tier
bandwith limit (see [3] and [4]). With that out of the way, the variable
determining whether we exceed the monthly limit is how many clones we
have every month. Only the latest version of LFS-tracked files are
downloaded on a given clone, and at the current 65M that means we have
about 15 clones per month. This is not very many, and will almost
certainly mean that we exceed the limit. Our options at that point are
to buy a data pack or to run our own LFS server. We would almost
certainly do the former to start.

Tracking these files via LFS means that developers will need to have Git
LFS installed in order to properly synchronize the files. If a developer
does not have LFS installed, cloning will complete successfully and the
build would complete successfully, but the app would fail when trying to
actually load the p2p data store files. For this reason, the build has
been updated to proactively check that the p2p data store files have
been properly synchronized via LFS, and if not, the build fails with a
helpful error message. The docs/build.md instructions have also been
updated accordingly.

It is important that we make this change now, not only to avoid growing
the repository in the way described above as we have been doing now for
many releases, but also because we are now considering adding yet more
binary objects to the repository, as proposed at
https://github.com/bisq-network/projects/issues/25.

[1]: https://git-lfs.github.com
[2]: https://github.com/cbeams/lfs-test
[3]: https://docs-staging.travis-ci.com/user/customizing-the-build/#git-lfs
[4]: https://github.com/travis-ci/travis-ci/issues/8787#issuecomment-394202791


<!--
- make yourself familiar with the CONTRIBUTING.md if you have not already (https://github.com/bisq-network/bisq/blob/master/CONTRIBUTING.md)
- make sure you follow our [coding style guidelines][https://github.com/bisq-network/style/issues)
- pick a descriptive title
- provide some meaningful PR description below
- create the PR
- in case you receive a "Change request" and/or a NACK, please react within 30 days. If not, we will close your PR and it can not be up for compensation.
- After addressing the change request, __please re-request a review!__ Otherwise we might miss your PR as we tend to only look at pull requests tagged with a "review required".
-->

Fixes #replaceWithIssueNr, fixes #replaceWithIssueNr

Your PR description here.
You can view, comment on, or merge this pull request online at:

  https://github.com/bisq-network/bisq/pull/4114

-- Commit Summary --

  * Remove read stdin loop
  * Implement requested changes in PR #4097
  * Track p2p data store files using Git LFS

-- File Changes --

    M .gitattributes (3)
    M .travis.yml (10)
    M build.gradle (14)
    M cli/src/main/java/bisq/cli/app/BisqCliMain.java (111)
    M cli/src/main/java/bisq/cli/app/CliCommand.java (4)
    A cli/src/main/java/bisq/cli/app/CommandParser.java (27)
    M cli/src/main/resources/logback.xml (4)
    M core/src/main/java/bisq/core/grpc/BisqGrpcServer.java (2)
    M docs/build.md (10)
    M gradle/witness/gradle-witness.gradle (2)
    M p2p/src/main/resources/AccountAgeWitnessStore_BTC_MAINNET (0)
    M p2p/src/main/resources/BlindVoteStore_BTC_MAINNET (0)
    M p2p/src/main/resources/DaoStateStore_BTC_MAINNET (274807)
    M p2p/src/main/resources/ProposalStore_BTC_MAINNET (0)
    M p2p/src/main/resources/SignedWitnessStore_BTC_MAINNET (0)
    M p2p/src/main/resources/TradeStatistics2Store_BTC_MAINNET (0)

-- Patch Links --

https://github.com/bisq-network/bisq/pull/4114.patch
https://github.com/bisq-network/bisq/pull/4114.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/bisq-network/bisq/pull/4114
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bisq.network/pipermail/bisq-github/attachments/20200402/febbc8fa/attachment-0001.html>