[bisq-network/bisq-markets] Re-implement Markets API service with pre-caching proxy + risq backend (#20)
wiz
notifications at github.com
Fri Nov 8 22:45:55 UTC 2019
## Background
The Bisq Markets API is currently served from a "bare-bones implementation" in PHP that loads and parses JSON files from disk upon every API request, with no caching or database utilized, and the JSON files are periodically updated by a Bisq node running on the same server from the live P2P trade data. Since there is no database or any caching used in this JSON file + PHP code architecture, the original authors specifically mentioned that scalability would eventually become an issue in the README file:
<img width="897" alt="Screen Shot 2019-11-09 at 0 58 23" src="https://user-images.githubusercontent.com/232186/68490766-f35f2f80-028c-11ea-9e27-e33a7837b984.png">
## Issue
As Bisq has grown in popularity, the Markets API now receives thousands of requests every minute. As reported in #19, and shown in my monitoring system's screenshots below, the Markets API server often becomes so overloaded from the uncached PHP / JSON filesystem architecture, that at least once per hour the Markets API server fails to respond to requests under heavy load. Other times when it does respond to API requests, the response time can be quite high, reaching several seconds due to the large backload of requests.
After taking over the Markets API maintainer role about a few weeks ago, I've migrated the site to a more powerful server with double the amount of CPU and RAM, which did help the uptime and response times, but it's obvious that we will eventually need to re-architect the API service from the ground up to solve the underlying scalability problems.
## Proposal
This PR re-implements the Bisq Markets API using a new pre-caching reverse-proxy to be deployed on Google Appengine Java standard runtime, to sit in front of @bodymindarts's new rust based Bisq project called Risq, which serves the Bisq Markets data over a built-in REST API, which gets its data from a dedicated Bisq seednode. The proxy utilizes a key-value datastore, key-value memcache, and HTTP caching to achieve high performance and availability.
## Implementation
The proxy will automatically pre-cache the latest the Markets API data by fetching updates every 60 seconds for the most popular endpoints from the Risq node, so these most popular values will always be served instantly from memcache. The results for other API endpoints and lesser used combinations of arguments will not be pre-cached, but instead cached as they are requested, in order to not request them more than once every 60 seconds. The caching is disabled while in development/testing purposes, and will be enabled in a separate PR later.
Whenever a successful response is obtained, the proxy also saves it in the cloud datastore. This way, if the backend REST API service is unavailable or does not respond for some reason, the proxy will continue serving the most recent known-good data. This ensures maximum uptime and availability and fixes #19.
### Caching and Pre-Caching logic
HTTP cache = 70 seconds
memcache = 70 seconds
datastore = forever
```
if (full query URL key exists in memcache)
return 200 with data
else
if (query to PHP service succeeds)
insert into memcache + datastore
return 200 with data
else // query failed
if (full query URL key exists in datastore)
return 200 with data
else // full query URL key not in datastore
return 502
```
The proxy serves its responses with a HTTP cache headers, which allows it to be cached by Google's internal CDN for that time, and reduces load on the proxy, to minimize running costs. Since trades on Bisq do not happen very frequently, this cache time seems reasonable trade-off for the performance gains.
## CORS
This PR fixes #14 by adding CORS headers to each HTTP response.
## Privacy Concerns
Of course using a fully self-hosted solution is preferable to using Google Appengine, since Google would have a log of all IP addresses that access the proxy hosted on their appengine platform, and Bisq should always protect the privacy of Bisq users as much as possible.
Adding the proxy running on Google Cloud should only a temporary fix until we can re-architect the appengine caching proxy away entirely, and return it to being 100% self-hosted. We have plans for how to accomplish this soon, but it will take some more time. In the meantime, to address this privacy concern we could possibly separate the API requests into 2 categories based on my analysis of the requests to Bisq Markets API:
1. The vast majority of requests originate from other cloud-hosted systems such as coinmarketcap, cryptowatch, etc. which wouldn't be concerned with their privacy. Additionally, if such a service does want to hide their IP address from Google's logging, they can always route their requests through Tor or a VPN.
2. For the actual Bisq users viewing Bisq Markets data directly on Bisq's website, we can proxy these requests on a self-hosted server, so Google never sees their IP addresses.
## Live Demo / Preview
* New Proxy serving API requests: https://bisq-markets.appspot.com/api/volumes?basecurrency=btc
* Bisq Website using Caching Proxy: https://bisq-markets-api-using-caching-proxy.netlify.com/markets/
You can view, comment on, or merge this pull request online at:
https://github.com/bisq-network/bisq-markets/pull/20
-- Commit Summary --
* Replace entire PHP codebase with cloud-based caching reverse-proxy
* Basic GrqphQLQuery
* WIP currencies
* /api/currencies working
* Return int for Currencies.precision field
* Add /markets query
* Add Offers query
* Add Depth query
* Add Ticker query
* Add Trades query
* Add Hloc query
* Add Volumes query
* Simplify offers query. Requires risq v0.3.0
* Merge pull request #3 from bodymindarts/risq-v0.3.0
* Fix trades limit param
* Merge pull request #4 from bodymindarts/fix-trades
* Update backend/frontend hostnames
* Add some testcases
* Fix OffersQuery structure
* Query buy and sell fields
* Merge pull request #5 from bodymindarts/fix-offers-query
* Merge pull request #6 from bodymindarts/buy-and-sell-ticker
* Add appengine configuration to always have min 1 instance running
* Tweak test script to just return HTTP response codes for now
* Fix index out of bounds exception when parsing query string
* Split into 2 testcases
* Handle timestamp=no in volumes
* Fix caching and CORS, update test cases
* Merge pull request #7 from bodymindarts/timestamp-no
* Fix timestamp=no on hloc
* Fix buys/sells for crypto offers (needs risq > 8b6c156)
* Merge pull request #9 from bodymindarts/fix-crypto-offers
* Fix for volume / amount display for altcoin markets
* Merge pull request #10 from bodymindarts/fix-altcoin-markets
* Add market to Trades response (fix null page)
* Merge pull request #11 from bodymindarts/add-market-to-trades
* Add missing org.graalvm.compiler jar dep to pom.xml
* Fix NPE in queryString replacement code
-- File Changes --
M .gitignore (4)
D 404.html (14)
M LICENSE (816)
M README.md (69)
D VERSION (1)
D api/.htaccess (27)
D api/README.md (94)
D api/VERSION (1)
D api/currencies/apidoc.php (50)
D api/currencies/index.php (34)
D api/depth/apidoc.php (47)
D api/depth/index.php (51)
D api/hloc/apidoc.php (60)
D api/hloc/index.php (173)
D api/index.php (173)
D api/markets/apidoc.php (61)
D api/markets/index.php (32)
D api/offers/apidoc.php (116)
D api/offers/index.php (49)
D api/ticker/apidoc.php (89)
D api/ticker/index.php (45)
D api/trades/apidoc.php (67)
D api/trades/index.php (82)
D api/volumes/apidoc.php (54)
D api/volumes/index.php (180)
D bitsquare_index.html (185)
D css/bitsquare/addtoany.css (1)
D css/bitsquare/css.css (392)
D css/bitsquare/style.css (1884)
D css/styles.css (155)
D index-body-summary.php (421)
D index-body.php (598)
D index.php (116)
D lib/btcutil.class.php (57)
D lib/currencies.class.php (87)
D lib/filecache.class.php (97)
D lib/html_table.class.php (211)
D lib/markets.class.php (121)
D lib/offers.class.php (163)
D lib/primary_market.class.php (106)
D lib/settings.class.php (37)
D lib/strict_mode.funcs.php (155)
D lib/summarize_trades.class.php (272)
D lib/summarize_trades_multimarket.class.php (117)
D lib/ticker.class.php (101)
D lib/trades.class.php (237)
A markets (65)
A pom.xml (129)
D settings.json.example (5)
A src/main/java/bisq/markets/api/CachingProxy.java (749)
A src/main/java/bisq/markets/api/GraphQLQuery.java (342)
A src/main/java/bisq/markets/api/GraphQLResponse.java (11)
A src/main/webapp/WEB-INF/appengine-web.xml (39)
A src/main/webapp/WEB-INF/cron.xml (169)
A src/main/webapp/WEB-INF/urlrewrite.xml (38)
A src/main/webapp/WEB-INF/web.xml (62)
D test-ajaxdata.php (569)
D test-cachedata.php (580)
A test/.gitignore (2)
A test/diff-all (12)
A test/test-all (12)
A test/tests (2431)
D widgets/head.html (3)
D widgets/order_details.widget.html (77)
D widgets/timezone-js.html (59)
-- Patch Links --
https://github.com/bisq-network/bisq-markets/pull/20.patch
https://github.com/bisq-network/bisq-markets/pull/20.diff
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/bisq-network/bisq-markets/pull/20
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bisq.network/pipermail/bisq-github/attachments/20191108/9b213abd/attachment-0001.html>
More information about the bisq-github
mailing list