Consider using Zstd instead of LZ4
Created by: Shnatsel
Recently I've been investigating available compression algorithms (it became a bottleneck in one of my projects) and found that a recent breakthrough in entropy coding - Asymmetric Numeral Systems - has spawned an entire new generation of compression algorithms.
The only two general-purpose algorithms based on it are Apple's LZFSE and Facebook's Zstandard (Zstd).
According to my (admittedly limited) benchmarking, Zstd provides significantly better compression than LZ4 at the same compression speed. It is also much more versatile, with competitive performance in a wide range of compression ratios.
Zstd also comes with a dictionary training mode, which might come in handy (according to On Random-Access Compression blogpost).
I have compared Zstd reference implementation against LZ4 reference implementation. I have failed to obtain comparable results from LZFSE reference implementation.
It would be a shame to use an older compression algorithm in a new filesystem while (seemingly) superior solutions are available.
The drawbacks are:
- increased implementation complexity compared to LZ4
- there is no readily available Zstd implementation in Rust
- possibly something else that I have failed to consider