[bisq-network/projects] Provide a reliable lightweight monitor with notifications (#45)

[James Cox]

FYI this is what I am working on.  Please let me know if anything is wrong..




### For Wiz' alerting infrastructure:

1. Provide a JSON file that contains details only of CURRENT ACTIVE ALERTS.  If there are no alerts it should be empty.  Format should include timestamp, seednode id, field triggering the alert, value of field, rule name that triggered the alert.


### For use by a web client UI:


2. Provide JSON files of the latest raw data queried from seed nodes.  This is already available as a WIP at http://46.101.179.224:8082/seednode_json and is essentially the same as described in "Example json" for all seednodes.
3. Provide a JSON file of data flagged by the serverside Java analytics: ERROR, WARN, or INFO.  Should include the tag names (same as in [2]), alert level and timestamp.  This will be used by the UI to highlight relevant data elements.
4. Provide an aggregated JSON view of the current seednode data, i.e. values averaged across all seednodes.  I originally implemented this in the prototype GUI but it would make more sense to do it at the server and leave the GUI just to display, no logic.  Identical tag names as [2].
5. Provide a JSON file listing alerts that happened in the past (with timestamps).  A historical log for display in GUI.  Same format as [1] but historical.




### Alerts


Highest priority according to Wiz is there should be **no instances of false alerts**.  To achieve that we need to define upfront exactly what the alerting criteria are (clarify the list posted OP).  Here follows a proposed list based on the original spec:

Hopefully @Wiz will have some suggestions as to which in this list are important, and which are not necessarily so important.





* A seednode is unreachable for > 3 minutes
* A seednode has uptime of > 26 hours
* A seednode's memory usage is over 1.5Gb
* A seednode's numConnections > 2x the seednode's maxConnections
* A seednode's Offer count is < 50% of the 24 hour moving average
* A seednode's Mediator count < 1
* A seednode's RefundAgent count < 1
* Across all seednodes, if there is any deviation of +/- 3 in numBsqBlocks
* Across all seednodes, if there is any deviation of +/- 3 in daoStateChainHeight
* Across all seednodes, if there is any deviation > 30% in SignedWitness
* Across all seednodes, if there is any deviation > 30% in AccountAgeWitness
* Across all seednodes, if there is any deviation > 30% in MailboxStoragePayload
* Across all seednodes, if there is any deviation > 30% in TradeStatistics3
* Across all seednodes, if daoStateHash is not consistent if and only if daoStateChainHeight is the same.  This seems a bit tricky any may need some thought. _[see below] difficult_
* Across all seednodes, Mediator count should be consistent _[see below] difficult_
* Across all seednodes, RefundAgent count should be consistent _[see below] difficult_
* Across all seednodes, Filter count should be consistent _[see below] difficult_
* Across all seednodes, Alert count should be consistent _[see below] difficult_

_**[NB] difficult**_ due to timing of differences between seednodes being updated/queried.  The operations are not atomic so values can differ for a while.  Need some thought on how to implement these cases without generating false alerts.



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/bisq-network/projects/issues/45#issuecomment-716203245
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bisq.network/pipermail/bisq-github/attachments/20201025/1779ba61/attachment.html>