[bisq-contrib] How should we monitor key Bisq network resources?

[Chris Beams]

There are many resources in the larger Bisq network that require monitoring. Here are a few examples:

 - Bisq seed nodes
 - Bisq price nodes
 - The Bitcoin full nodes we’re setting up right now
 - The bisq.network website
 - The Discourse forum software at forum.bisq.io
 - The Mailman service running at lists.bisq.network
 - The Bisq markets webapp and API at market.bisq.io

What kind of monitoring do we need? In many cases, it’s not just uptime monitoring. We also need to be able to run arbitrary checks on these services, e.g. to query what version of Bitcoin Core a node is running, what version of seednode or pricenode software each node is running, and so forth. So these checks might involve RPC calls, HTTP calls or something else..

How do we want to be notified when a resource is down or fails one of these checks? I’d say that, at a minimum, we should be able to route any such alert into a given Slack channel. For example, an alert about a pricenode being down or failing a version check would get routed to the #pricenode channel (which already exists, btw). Pricenode operators should have Slack notifications set up for the #pricenode channel such that they are sure to see alerts like this.

How should we go about setting this up? I wonder if it would make sense to define a “network monitor” role with a certain contract about what resources need to be monitored, and how monitoring alerts should be communicated, and then let the owner of that role set up Nagios or whatever they want to fulfill that contract.

One requirement that we might want to make part of that contract, if feasible, would be that the monitoring solution supports plaintext configuration files that we can manage via Git and GitHub. That way other bonded contributors can “self-serve” by issuing pull requests against, say, the bisq-network/monitoring repository in order to add/change/remove monitoring rules for the resources they are responsible for.

Your thoughts please, especially if you have any expertise in this domain, want to provide this service to the network yourself, or know somebody else who might.

- Chris