<p>The large binary objects in p2p/src/main/resources/ are updated on every<br>
Bisq release with the latest network data to avoid the need for new Bisq<br>
clients to download all of this information from the network, which<br>
would easily overload seed nodes and generally bog down the client.</p>
<p>This approach works well enough for its purposes, but comes with the<br>
siginficant downside of storing all of this binary data in Git history<br>
forever. The current version of these binary objects total about 65M,<br>
and they grow with every release. In aggregate, this has caused the<br>
total size of the repository to grow to 360M, making it cumbersome to<br>
clone over a low-bandwith connection, and slowing down various local Git<br>
operations.</p>
<p>To avoid further exacerbating this problem, this commit sets these files<br>
up to be tracked via Git LFS. There's nothing we can do about the 360M<br>
of files that already exist in history, but we can ensure it doesn't<br>
grow in this unchecked way going forward. For an understanding of how<br>
Git LFS works, see the reference material at <a href="https://git-lfs.github.com">1</a>, and see also the<br>
sample project and README at <a href="https://github.com/cbeams/lfs-test">2</a>.</p>
<p>We are using GitHub's built-in LFS service here, and it's important to<br>
understand that there are storage and bandwidth limits in place. We have<br>
1G total storage and 1G per month of bandwidth on the free tier. If we<br>
exceed this, we will need to purchase a "data pack" from GitHub, which<br>
will get us to 50G storage and bandwith. These are reasonably priced and<br>
not the end of the world if it becomes necessary. In an attempt to avoid<br>
this, however, the Travis CI build configuration has been updated to<br>
cache Git LFS files, such that they are not re-downloaded on every CI<br>
build, as this would very quickly cause us to exceed the free tier<br>
bandwith limit (see <a href="https://docs-staging.travis-ci.com/user/customizing-the-build/#git-lfs" rel="nofollow">3</a> and <a href="https://github.com/travis-ci/travis-ci/issues/8787#issuecomment-394202791" data-hovercard-type="issue" data-hovercard-url="/travis-ci/travis-ci/issues/8787/hovercard">4</a>). With that out of the way, the variable<br>
determining whether we exceed the monthly limit is how many clones we<br>
have every month. Only the latest version of LFS-tracked files are<br>
downloaded on a given clone, and at the current 65M that means we have<br>
about 15 clones per month. This is not very many, and will almost<br>
certainly mean that we exceed the limit. Our options at that point are<br>
to buy a data pack or to run our own LFS server. We would almost<br>
certainly do the former to start.</p>
<p>Tracking these files via LFS means that developers will need to have Git<br>
LFS installed in order to properly synchronize the files. If a developer<br>
does not have LFS installed, cloning will complete successfully and the<br>
build would complete successfully, but the app would fail when trying to<br>
actually load the p2p data store files. For this reason, the build has<br>
been updated to proactively check that the p2p data store files have<br>
been properly synchronized via LFS, and if not, the build fails with a<br>
helpful error message. The docs/build.md instructions have also been<br>
updated accordingly.</p>
<p>It is important that we make this change now, not only to avoid growing<br>
the repository in the way described above as we have been doing now for<br>
many releases, but also because we are now considering adding yet more<br>
binary objects to the repository, as proposed at<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="576403913" data-permission-text="Title is private" data-url="https://github.com/bisq-network/projects/issues/25" data-hovercard-type="issue" data-hovercard-url="/bisq-network/projects/issues/25/hovercard" href="https://github.com/bisq-network/projects/issues/25">bisq-network/projects#25</a>.</p>

<p>Fixes #replaceWithIssueNr, fixes #replaceWithIssueNr</p>
<p>Your PR description here.</p>

<hr>

<h4>You can view, comment on, or merge this pull request online at:</h4>
<p>  <a href='https://github.com/bisq-network/bisq/pull/4114'>https://github.com/bisq-network/bisq/pull/4114</a></p>

<h4>Commit Summary</h4>
<ul>
  <li>Remove read stdin loop</li>
  <li>Implement requested changes in PR #4097</li>
  <li>Track p2p data store files using Git LFS</li>
</ul>

<h4>File Changes</h4>
<ul>
  <li>
    <strong>M</strong>
    <a href="https://github.com/bisq-network/bisq/pull/4114/files#diff-fc723d30b02a4cca7a534518111c1a66">.gitattributes</a>
    (3)
  </li>
  <li>
    <strong>M</strong>
    <a href="https://github.com/bisq-network/bisq/pull/4114/files#diff-354f30a63fb0907d4ad57269548329e3">.travis.yml</a>
    (10)
  </li>
  <li>
    <strong>M</strong>
    <a href="https://github.com/bisq-network/bisq/pull/4114/files#diff-c197962302397baf3a4cc36463dce5ea">build.gradle</a>
    (14)
  </li>
  <li>
    <strong>M</strong>
    <a href="https://github.com/bisq-network/bisq/pull/4114/files#diff-a8578edbbfb64f313f4df527913010c8">cli/src/main/java/bisq/cli/app/BisqCliMain.java</a>
    (111)
  </li>
  <li>
    <strong>M</strong>
    <a href="https://github.com/bisq-network/bisq/pull/4114/files#diff-d9177b9a1771e1dea288527a141be530">cli/src/main/java/bisq/cli/app/CliCommand.java</a>
    (4)
  </li>
  <li>
    <strong>A</strong>
    <a href="https://github.com/bisq-network/bisq/pull/4114/files#diff-146c13fe7dbf75f37148656f62dcde51">cli/src/main/java/bisq/cli/app/CommandParser.java</a>
    (27)
  </li>
  <li>
    <strong>M</strong>
    <a href="https://github.com/bisq-network/bisq/pull/4114/files#diff-030077308dff6ee373aca2b0878054fd">cli/src/main/resources/logback.xml</a>
    (4)
  </li>
  <li>
    <strong>M</strong>
    <a href="https://github.com/bisq-network/bisq/pull/4114/files#diff-1e668cbee2d15f677b1eef37b71ba74f">core/src/main/java/bisq/core/grpc/BisqGrpcServer.java</a>
    (2)
  </li>
  <li>
    <strong>M</strong>
    <a href="https://github.com/bisq-network/bisq/pull/4114/files#diff-a60cd9d90730ccc175ef69a2a45d10f2">docs/build.md</a>
    (10)
  </li>
  <li>
    <strong>M</strong>
    <a href="https://github.com/bisq-network/bisq/pull/4114/files#diff-d5a9251f678d7f964871b83c050aa9a7">gradle/witness/gradle-witness.gradle</a>
    (2)
  </li>
  <li>
    <strong>M</strong>
    <a href="https://github.com/bisq-network/bisq/pull/4114/files#diff-d6fc299c16d57d748eb43348f552ad76">p2p/src/main/resources/AccountAgeWitnessStore_BTC_MAINNET</a>
    (0)
  </li>
  <li>
    <strong>M</strong>
    <a href="https://github.com/bisq-network/bisq/pull/4114/files#diff-a621ab9b122b050f076c8033582b374b">p2p/src/main/resources/BlindVoteStore_BTC_MAINNET</a>
    (0)
  </li>
  <li>
    <strong>M</strong>
    <a href="https://github.com/bisq-network/bisq/pull/4114/files#diff-857fec3fff6b5e0030e4f40fa788ff21">p2p/src/main/resources/DaoStateStore_BTC_MAINNET</a>
    (274807)
  </li>
  <li>
    <strong>M</strong>
    <a href="https://github.com/bisq-network/bisq/pull/4114/files#diff-1214836ae5d585a0d998a8903dc4d089">p2p/src/main/resources/ProposalStore_BTC_MAINNET</a>
    (0)
  </li>
  <li>
    <strong>M</strong>
    <a href="https://github.com/bisq-network/bisq/pull/4114/files#diff-f35512c887288e6404f5dff34c8dae71">p2p/src/main/resources/SignedWitnessStore_BTC_MAINNET</a>
    (0)
  </li>
  <li>
    <strong>M</strong>
    <a href="https://github.com/bisq-network/bisq/pull/4114/files#diff-cc4ec18414cd18c51b10b96ee4c8695e">p2p/src/main/resources/TradeStatistics2Store_BTC_MAINNET</a>
    (0)
  </li>
</ul>

<h4>Patch Links:</h4>
<ul>
  <li><a href='https://github.com/bisq-network/bisq/pull/4114.patch'>https://github.com/bisq-network/bisq/pull/4114.patch</a></li>
  <li><a href='https://github.com/bisq-network/bisq/pull/4114.diff'>https://github.com/bisq-network/bisq/pull/4114.diff</a></li>
</ul>

<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />You are receiving this because you are subscribed to this thread.<br />Reply to this email directly, <a href="https://github.com/bisq-network/bisq/pull/4114">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/AJFFTNRNDL6U55AGENSAW5TRKSFBBANCNFSM4L2NFAIA">unsubscribe</a>.<img src="https://github.com/notifications/beacon/AJFFTNWAKZ7IMJ4LJAG2I63RKSFBBA5CNFSM4L2NFAIKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4I2SUWUQ.gif" height="1" width="1" alt="" /></p>
<script type="application/ld+json">[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/bisq-network/bisq/pull/4114",
"url": "https://github.com/bisq-network/bisq/pull/4114",
"name": "View Pull Request"
},
"description": "View this Pull Request on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]</script>