[bisq-network/bisq] API Beta Test Feedback And Next Development Phase (#5407)

[Stan]

**I. Introduction**

A pause in API work gives me a chance look at feedback from recent beta testing, and for a consensus among users and devs about the next development phase.

I think the broad goals are build out the API to expose the features available in the existing desktop application to the new gRPC CLI, and make those features available to new user interfaces and bots via gPRC.  There has also been a lot of back and forth about one or more Bisq devs doing the work of building a new desktop UI and a web + mobile client, or not.  Either way, the choice of using gRPC for client-server data transfer makes it possible for the API to serve clients written in a sufficient variety of programming [languages](https://grpc.io/docs/languages) with this caveat:  gRPC uses the HTTP/2 transport protocol, which most browsers do not support.  By default (I think), we made a decision to move away from HTTP/1.1 and supporting RESTful web clients in the near term.  More will be said about this further down the document and in the ensuing discussion.  

Another point to bring up early in the conversation is an agreement between @cbeams, @sqrrm, and myself:  any new Bisq web client project should be scoped as a prototype, which means it will be small, experimental, and may be discarded in favor of one or more other prototype dev efforts.  Chris Beams also suggested that in the short term, we should create a neatly packed UI prototype and demo it running in a local [Umbrel](https://getumbrel.com/) installation; that may entice a skilled UI developer to commit to delivering a new web client.

Narrower subjects for discussion are outlined below.  Technical issues are mixed in with non-technical, end user related topics.  The technical discussions can be taken elsewhere, but I think some of them need to be mentioned.

_Note:  I try to organize what may be a many threaded discussion within a conventional outline format.  The outline headers could be copied above your comments._  

**II. Bisq dev effort to build new UI(s) requires significant reorganization of existing code**

A. Cons

1. No tangible benefit to end users

    I mention this early to bring attention to the fact that a significant amount of UI related development will not  provide visible or financial benefits to end users, and may seem excessive in terms of BSQ compensation and time. But duplicating code from the desktop app in new client side modules will wind up costing more in the long run.

2. Compensating devs for code refactoring work hard to justify

    Bisq developers try to work on what the team deems valuable and necessary, often measured by what users can see and benefit from.  Developers are also motivated to do work affording quick and visible benefits.  Refactoring work can be drudgery.

B. Pros

1. It has to be done.  I don't think any Bisq dev team member disagrees.  

2. Enables migration from JavaFX desktop application to modular API (that the desktop application will use)

    The benefit of careful, step-by-step refactoring of some existing Bisq UI desktop code for each new API feature is a system of smaller modules, capable of supporting clients (including web/mobile) written in several programming languages.

    _Technical Note: I think defaulting to Gradle for organizing new API modules needs to be discussed among devs.  The future upgrade to JDK 15+ makes it worthwhile to at least take another look at Java modules.  We may stick with Gradle as the build system, but it might not be the best tool for defining the finer grained modules -- what are now Gradle build file sub-projects._

**III. How does gRPC affect decisions about new clients**

A. API's client server data transport protocol is HTTP/2

1. Extra work is needed to make the API server vend Json to RESTful HTTP/1.1 web & mobile clients.

    Support for RESTful clients can be achieved With the help of an externally run HTTP reverse proxy in front of an API server, and coding effort to transform Bisq's protobuf messages into Json.

2. No additional server side work is need to send protobuf messages to HTTP/2 gRPC clients.

    As mentioned in item A.1, a proxy is still needed in front of the API server, but if the [client works with gRPC](https://grpc.io/docs/languages),  no additional work on the server is need to support new desktop and web/mobile clients.

    _Note:  When I say "no additional work on the server is needed", I mean that an existing API method can be invoked from  a gRPC client without the server having to transform data into another data format expected by the client.  Everything sent from the server is a gRPC protobuf message, no matter what kind of gRPC client sent the request._

B. Any new API security mechanism should be supported by gRPC HTTP/2

The current CLI security mechanism is the simplest possible: a cleartext --password=<password> option.  We might want to discuss implementing something stronger, but not as heavy as [macaroons](https://github.com/rescrv/libmacaroons), which Chris Beams and I have experimented with.

**IV. gRPC Web / Mobile UI**

JavaScript clients can make gRPC calls to the API server using [grpc-web](https://github.com/grpc/grpc-web).  I have gone through a tutorial to create a `commonjs` client to invoke some API service methods (pretty easy), and I read in the [grpc-web](https://github.com/grpc/grpc-web) docs  that google closure tools are also compatible (but I don't know anything about google closure tools).

As mentioned in the introduction, there seems to be consensus among devs and end users that a new web client is needed. But development plans should be formed by most core dev team members -- not one dev -- before proceeding with any UI prototype development.

**V. gRPC CLI**

A. The API's CLI should remain the reference client

It is my opinion that most methods exposed to end users should work in the existing gRPC CLI, even when UI development drives the work.  This would be a loose rule to keep the CLI in sync with the API and web UI development.  Some API methods may not be appropriate for the CLI, for example, `registerdisputeagents`.

B. New API methods should be supported in the CLI as soon as they are made available to a new web client

I'm being repetitive, but I think that if and when work on a new UI requires parallel efforts in the API server, do not forget to update the gRPC CLI as well.  I am not saying web UI development should ever have to wait for methods to be implemented in the CLI, but the CLI should never get left far behind the UI in terms of supported features.  It is understood that some web UI features may rely on bi-directional streaming, for example, and will be incompatible with the session-less CLI.

**VI. Bots**

I have experimented with a couple of java bots, and think the API might include a java bot support module in the future.  However,  releasing any java bot support this early in the API's development is not a great idea.  There are many asynchronous activities that occur in offer creation and during a trade;  a bot programmer may make assumptions about offer / trade preparation and state that are not true (yet).  It is also prudent to let the API flush out old problems in the core layers and get them fixed before encouraging bot deployment with too much enthusiasm.

**VII. API still missing basic functionality, can easily reproduce bloom filter problem (Technical)**

A. Unimplemented offer related methods

`editoffer`, `enableoffer`, `disableoffer`, support for trigger price option in `createoffer` and `editoffer`

B. Unimplemented trade related methods

`gettrades` (return user's trades), maybe add add an optional `--short-id` option to `gettrades`, and make it clear in docs/help that `getrrades --short-id=<short-id>` may return more than one trade

C. Bloom filter configuration a problem for larger wallets

The size of the bloom filter plus infrequent filter update broadcasts become a problem after a certain number of wallet transactions are created.  For example, deposit transactions cannot be found.  API tests can easily reproduce the problem, and increasing  the frequency of filter updates removes a strict trade count limit within a single API session.  This may help devs  find a solution.

**VII. Conclusion**

Much this doc is based on my own opinion and observations after working mostly alone on the API, and I realize the inclusion of technical issues here may be annoying. I am trying to get both API user feedback and involvement from other devs about the next development phase from the same discussion.  Call me lazy.

Hypothetically, without any other input on how to proceed with the API work, and with an experienced colleague with good web client skills (specifically with [gRPC on Node.js](https://github.com/grpc/grpc-node)), I would work closely with the UI developer to build a web client prototype.

What do you think?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/bisq-network/bisq/discussions/5407
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bisq.network/pipermail/bisq-github/attachments/20210412/ea1e37f7/attachment-0001.htm>