[bisq-network/proposals] [WIP] Finding a unified test framework for Bisq (#74)

Florian Reimair notifications at github.com
Fri Feb 15 15:04:30 UTC 2019

> _This is a Bisq Network proposal. Please familiarize yourself with the [submission and review process](https://docs.bisq.network/proposals.html)._

The whole Bisq project only features 25 test classes, the need for better test coverage is clear. In order to achieve a higher test coverage, we should agree on the used tools/technologies/frameworks for testing before the collection of test frameworks within Bisq gets out of hand.
This goal of this proposal is to foster an understanding on which testing frameworks are available and finally, decide on a small set of tools/technologies/frameworks to be used in Bisq.

# Unit Testing Frameworks

Unit testing frameworks are a convenient way of doing test-driven development when no GUI is involved. As these frameworks are quite mature, integration testing can be done as well. Below, a couple of hand-picked test metrics are discussed. Afterwards, unittest frameworks are evaluated against the metrics. Finally, conclusions are drawn.

## Test Metrics

- **IDE integration** Low effort test-driven development/local testing is only possible when a proper IDE integration exists.
- **Gradle integration** provide effortless automated testing
- **Test classes in Bisq** The number of existing test classes in the Bisq code as per 2019-02-15
- **Parameterized Tests** parameterized tests make it easy to get greater test coverage without having to duplicate code
- **Test Dependencies** Given Test2 depends on Test1 and Test1 fails, Test2 is skipped. Useful in automated testing as it keeps the testing load down in case of a failure. Useful in test design, as basic functionality does not have to be considered for special tests which results in less editing of test code to match altered features.

*your input here*

## Comparison

|   | IDE integration | Gradle integration | Test classes in Bisq | Parameterized Tests | Test Dependencies |
| [JUnit4](https://junit.org/junit4/) | + | + | 18 | o | - |
| [JUnit5 "jupiter"](https://junit.org/junit5/)  | + | + | 7 (only in monitor) | + | - |
| [TestNG](https://testng.org/) | + | + | 0 | + | + |
| [Spock](http://spockframework.org) |?|?|0|?|?|

*your input here*

## Conclusion

JUnit4 is outdated. Spock, being a functional testing framework, seems the most powerful, yet, it is quite different to use than the classic testing frameworks. TestNG allows for resource saving automated tests, is, however, not supported by at least one GUI testing framework.

# GUI Testing

Automated GUI testing is more delicate then simple unit testing. However, given Bisq uses JavaFX only, there are a few testing frameworks which integrate well with unit testing frameworks. Below, a couple of hand-picked test metrics are discussed. Afterwards, available frameworks are evaluated against the metrics. Please be aware, that only frameworks compatible with JavaFX are listed. Finally, conclusions are drawn.

## Test Metrics

- **Supported UnitTest Frameworks** Which unittest frameworks the GUI testing framework in question integrates with.

*your input here*

## Comparison
|   | IDE integration | Gradle integration | Test classes in Bisq | Supported UnitTest frameworks | easy to use |
| [TestFX](https://github.com/TestFX/TestFX) |+|+| 0 | JUnit4, JUnit5, Spock | + |
| [Jemmy](http://wiki.netbeans.org/Jemmy)  |o|o| 0 |-|o|
| [Automaton](https://github.com/renatoathaydes/Automaton) | ? |  ?|?  |?  |?  |

*your input here*

## Conclusion

It looks like TestFX is the only option as of now. Jemmy does rely on a `void main(.)` for running the tests, Automaton seems pretty outdated (last release 2 year ago) and dead (last change 1 year ago).

# Overall Conclusion

As of now, JUnit5 along with TestFX seems to be the winner.

# Aftermath

Having decided on the testing frameworks, contributors can freely create tests. In the long run, efforts required for manual testing should be reduced while increasing test coverage and therefore, the quality of Bisq.

# Call for contribution

I have started a sketch on how testing frameworks can be compared. I also filled in unit test frameworks as I have some experience with them. I, however, do not have any experience with GUI testing frameworks. If you can contribute something to this proposal, please do so! Thanks!

You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bisq.network/pipermail/bisq-github/attachments/20190215/f207ff97/attachment.html>

More information about the bisq-github mailing list