gkowal/amun-code
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,910 Commits 5 Branches 0 Tags
Commit Graph

26 Commits

Author SHA1 Message Date
Grzegorz Kowal
5efc38b8a1 COMPRESSION, IO: Implement data filters for better compression. 
This commit implements two data filters:
1) "shuffle" - reorganizes bytes to group most to less significant parts
   together: |a1|a2|a3|b1|b2|b3| -> |a1|b1|c1|a2|b2|c2|a3|b3|c3|
2) "bytedelta" - additionally stores differences between subsequent
   values: |a1|a2|a3|b1|b2|b3| -> |a1|b1-a1|c1-b1|a2|b2-a2|c2-b2|a3|b3-a3|c3-b3|

For large datasets, in particular in 3D, the compression ratio can be
significantly better after applying these filters.

Inspired by https://aras-p.info/blog/2023/03/01/Float-Compression-7-More-Filtering-Optimization/

Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2023-07-27 18:35:54 -03:00
Grzegorz Kowal
81de98d9e2 Update the copyright year to 2023. 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2023-02-01 18:36:37 -03:00
Grzegorz Kowal
9829505650 Update copyright year to 2022. 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2022-02-02 09:51:41 -03:00
Grzegorz Kowal
09cbb1b812 COMPRESSION: Declare explicit kind in lz4_iserror() and zstd_iserror(). 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2021-12-03 10:25:21 -03:00
Grzegorz Kowal
1fd1ee7602 IO: Add compression suffix to files individually. 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2021-11-29 12:23:26 -03:00
Grzegorz Kowal
b10b49c8c3 COMPRESSION: Add compression_suffix to store the file extension. 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2021-11-29 12:08:42 -03:00
Grzegorz Kowal
de81332cf8 COMPRESSION: Reset clen in case the compression is unsupported. 
This will indicate that no compression should be used.

Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2021-11-29 08:27:21 -03:00
Grzegorz Kowal
88cdf23078 COMPRESSION, IO: Remove double buffer for compression. 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2021-11-28 23:27:03 -03:00
Grzegorz Kowal
e4ffa62714 COMPRESSION: Add function to determine the compression buffer size. 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2021-11-28 22:32:51 -03:00
Grzegorz Kowal
e6fc8e5d24 COMPRESSION: Pass the input length to compress(). 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2021-11-28 22:19:50 -03:00
Grzegorz Kowal
20282539f9 COMPRESSION: Use ENUM to enumerate compression formats. 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2021-11-28 09:59:20 -03:00
Grzegorz Kowal
2d7a4b0d3c COMPRESSION: Remove unnecessary comments. 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2021-11-19 13:13:43 -03:00
Grzegorz Kowal
fa8cfa8f8f COMPRESSION: Remove the 2GiB limit in compress(). 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2021-02-04 17:40:17 -03:00
Grzegorz Kowal
3d16d5c13d Update copyrights. 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2021-02-04 17:35:04 -03:00
Grzegorz Kowal
e23f97bbd6 COMPRESSION: Remove long lines. 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2020-08-24 22:15:54 -03:00
Grzegorz Kowal
945e627199 COMPRESSION, IO: Replace sizeof() with size() and transfer(). 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2020-08-22 20:29:43 -03:00
Grzegorz Kowal
77122f8854 COMPRESSION: Implement compression level for LZ4 compression. 
Also increase the compression buffer and turn on internal checksum.

Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2020-08-18 18:01:12 -03:00
Grzegorz Kowal
ec8683a7a0 COMPRESSION: Add CRC64 check to LZMA compression. 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2020-08-18 17:58:10 -03:00
Grzegorz Kowal
d048159798 COMPRESSION: Add LZMA/XZ compression format. 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2020-08-13 15:42:07 -03:00
Grzegorz Kowal
f9b6dd47e9 COMPRESSION: Use LZ4 frame for compatibility with command lz4. 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2020-08-13 09:17:09 -03:00
Grzegorz Kowal
b05f186765 COMPRESSION: Use allocated buffer for ZSTD compression. 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2020-08-11 15:36:20 -03:00
Grzegorz Kowal
eb7c8dbd76 COMPRESSION, PYTHON: Add support for LZ4 compression. 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2020-08-11 15:28:20 -03:00
Grzegorz Kowal
af4da9444a IO: Store compression format in binary file attributes. 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2020-08-11 12:00:42 -03:00
Grzegorz Kowal
053392e762 COMPRESSION: Add Zstandard compression support. 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2020-08-10 16:48:31 -03:00
Grzegorz Kowal
08b7e81f6a COMPRESSION, IO: Initialize compression in initialize_io(). 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2020-08-10 15:34:20 -03:00
Grzegorz Kowal
ab3b5d6877 COMPRESSION: Add module to handle data compression. 
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
 2020-08-10 15:12:40 -03:00