[bisq-network/bisq] Serialisation format for Bisq 2 (Misq) (Discussion #5927)

[chimp1984]

As we need to decide soon for the serialisation format we are going to use in Bisq 2 (Misq) I wanted to share my research and discuss what we should use.

## Requirements:
- Type safety
- Fast
- Small

## Additional nice-to-have:
- RPC support
- Deterministic serialisation

## Candidates:
- [Protocol Buffers](https://developers.google.com/protocol-buffers/)
- [Cap'n Proto](https://capnproto.org)
- [FlatBuffers](https://google.github.io/flatbuffers/)
- [TLV based custom format](https://en.wikipedia.org/wiki/Type%E2%80%93length%E2%80%93value)

### Protocol Buffers:
#### Pro:
- Widely used
- Used inside Google, so gets top prio support
- Many implementations
- gRPC
- We are familiar with it from Bisq
#### Con:
- Can be slow for large data

### Cap'n Proto
#### Pro:
- Zero-copy deserialization
- Fast
#### Con:
- Smaller dev and language support

### FlatBuffers
#### Pro:
- Zero-copy deserialization
- Fast
- Supports also Protocol Buffers .proto format
- Support for gRPC (not sure how extensive)

#### Con:
- Seems to be developed mainly for file storage in trusted environment (gaming use cases), not for network (from https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html)
- Smaller dev and language support
- "FlatBuffers was definitely designed for languages like C++, so Java and Go may not be as good a fit. A big issue in Java for example is that it can't read UTF-8 strings in-place" (from https://github.com/google/flatbuffers/issues/4385)

### TLV based custom format:
#### Pro:
- Full control on byte level
- Can provide deterministic byte representation 
- Used by LN (https://github.com/lightning/bolts/blob/master/01-messaging.md#type-length-value-format).
#### Con:
- High dev effort
- Risks for implementation bugs
- Risks for security issues

Comparison/discussions:
- https://capnprot o.org/news/2014-06-17-capnproto-flatbuffers-sbe.html
- https://speice.io/2019/09/binary-format-shootout.html
- https://github.com/google/flatbuffers/issues/4385


## Conclusion:
I think Protocol Buffers seems to be the best choice. It does not meet the performance of Cap'n Proto and FlatBuffers but the support for gRPC and the large dev community/support and language support are a big plus. Security considerations are also important specially if not the core language implementation is used. In case of Cap'n Proto and FlatBuffers this work is often done by external developers and there are less eyes on it. Protocol Buffers has achieved inside Google (as well as outside) a huge network effect which is hard to be challenged by the technically maybe superior concepts implemented in Cap'n Proto and FlatBuffers.

Deterministic serialisation is not guaranteed by none of those (beside a TLV based custom format) but it is in practice achieved when avoiding maps (at least in Protocol Buffers ). It could be managed also with a version freeze in case an updated version would cause issues. The requirement for deterministic data is also an exceptional case (DAO) and it could be dealt differently with it as well (would be anyway better but comes with more effort - e.g. use a custom serialisation format where this is a hard requirement).

I have not looked closer to [Simple Binary Encoding](https://real-logic.github.io/simple-binary-encoding/). Seems to be very fast but comes with its own trade-offs. It is used in high frequency trading context, which does not really match our context. 

Any feedback/comments?
 

-- 
Reply to this email directly or view it on GitHub:
https://github.com/bisq-network/bisq/discussions/5927
You are receiving this because you are subscribed to this thread.

Message ID: <bisq-network/bisq/repo-discussions/5927 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bisq.network/pipermail/bisq-github/attachments/20211218/46055a85/attachment.htm>