e76e875004
Update the copyright year to 2024.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2024-03-07 09:34:43 -03:00
68449d0ccc
IO: Add ACTION and STATUS flags to OPEN statements.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-12-20 18:33:58 -03:00
9455f9be9a
IO: Use action='read' explicitely when opening a file for read-only.
...
Additionally, unify formatting of statements OPEN, READ, WRITE, CLOSE,
INQUIRE.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-12-19 15:55:33 -03:00
bb36b90171
IO: Access the metadata file as read-only while restoring parameters.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-12-19 15:38:40 -03:00
0c8cab74a2
IO, PARAMETERS, SYSTEM: Better parameter restoring.
...
This commit implements a better way (less I/O operations) restoring of
the parameters during the job restart. Some parameters, such as the size
of the block, or the equation system, cannot be changed during the
restart. Such parameters need to be restored from the restart snapshots.
The new way restores these parameters on the MPI master process only,
updates their values if they were changed, and distributes them to other
MPI processes. This way the number of I/O operations is kept to the
minimum (only one process access one file).
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-12-18 09:30:07 -03:00
6847d1c994
PARAMETERS: Rewrite module with use of variable strings.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-12-17 18:09:12 -03:00
58b6a62ff8
IO: Fix wrong dimensions of corners() in store_metablocks_h5().
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-12-14 16:57:35 -03:00
47b3b50ae0
IO: Improve description of subroutine write_binary_xml().
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-14 13:05:24 -03:00
a71a1390ca
IO: Formatting correction.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-14 12:49:42 -03:00
5574b6fcf1
IO: Rewrite storing of restart and regular XML snapshots.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-14 11:50:07 -03:00
8ac5e49def
IO: Enable compression when storing XML restart snapshot.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-12 07:08:48 -03:00
c015d5849a
IO: Add missing item_size for 'complex64' in read_binary_xml().
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-11 19:54:29 -03:00
14fad08f47
IO: Rewrite read_restart_snapshot_xml().
...
This subroutine now used the new XML module to parge XML files. It also
used new subroutine read_binary_xml() to read binary files.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-11 19:41:00 -03:00
64966c3eb3
IO: Add subroutine to read binary XML files.
...
This subroutine is also responsible for detection if binary data are
compressed and encoded, and respectively decompress and decode them.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-11 19:29:08 -03:00
b9c638c07b
COMPRESSION, IO: Rewrite the compression.
...
This commit does not change any functionality. It just prepares for
easier implementation of the compression of the restart snapshots.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-07-29 18:35:45 -03:00
c7d8fdf512
COMPRESSION, IO: Rename compression_<format> to compressor_<format>.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-07-29 16:17:06 -03:00
db19393dd1
COMPRESSION: Make default compression levels more consistent.
...
The default compression levels correspond now to the default levels
defined by each compression format library.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-07-27 23:36:39 -03:00
5efc38b8a1
COMPRESSION, IO: Implement data filters for better compression.
...
This commit implements two data filters:
1) "shuffle" - reorganizes bytes to group most to less significant parts
together: |a1|a2|a3|b1|b2|b3| -> |a1|b1|c1|a2|b2|c2|a3|b3|c3|
2) "bytedelta" - additionally stores differences between subsequent
values: |a1|a2|a3|b1|b2|b3| -> |a1|b1-a1|c1-b1|a2|b2-a2|c2-b2|a3|b3-a3|c3-b3|
For large datasets, in particular in 3D, the compression ratio can be
significantly better after applying these filters.
Inspired by https://aras-p.info/blog/2023/03/01/Float-Compression-7-More-Filtering-Optimization/
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-07-27 18:35:54 -03:00
3ddbe189a0
IO: Fix the restart from XML snapshots with more processes.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-02-15 10:27:41 -03:00
81de98d9e2
Update the copyright year to 2023.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-02-01 18:36:37 -03:00
295453b01c
IO: Distribute files better among the restarting processes.
...
This is an improvement in the case a job restarted with a higher
number of processes. The reading of data blocks is spread among new
processes, so it better balances memory.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2022-12-04 20:16:55 -03:00
8d2edeed12
IO: Fix storing and restroring the forcing coefficients.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2022-05-31 19:30:51 -03:00
9829505650
Update copyright year to 2022.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2022-02-02 09:51:41 -03:00
ce1280d00c
IO: Rebuild the leaf and datablock lists in read_restart_snapshot().
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2022-01-18 15:23:19 -03:00
2b21d29da7
IO: Store/restore cglm.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2022-01-11 12:45:26 -03:00
e977548f41
IO: Replace non-standard SYSTEM with Fortran 2008 EXECUTE_COMMAND_LINE.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-12-17 14:47:25 -03:00
c5d65de44c
IO: XDMF works only with deflate or uncompressed HDF5 files.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-12-03 11:55:30 -03:00
fcce326245
IO: Fix generation of XDMF files.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-12-03 11:54:19 -03:00
34d551e9a6
IO: Stop compiler complaining if HDF5 is off.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-12-03 10:35:56 -03:00
5792fe60a0
IO: Store data type and dimensions for XML-binary format.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-12-02 10:43:03 -03:00
ce1f6cee4a
IO: Tune chunking for compressions in store_dataset_h5().
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-12-02 09:15:35 -03:00
43ad4a7af3
IO: Add SZIP compression to HDF5 snapshots.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-12-01 21:52:41 -03:00
c85831a386
IO: Rewrite HDF5 compression code and add ZFP compression.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-12-01 19:00:24 -03:00
80b9c6d6a2
IO: Compress only variable datasets in HDF5 regular snapshots.
...
Do not compress HDF5 restart snapshots at all.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-12-01 18:59:21 -03:00
a364281184
IO: Fix restore_metablocks_h5() for 3D.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-12-01 18:17:29 -03:00
b85dfafef1
IO: Rewrite file division in read_restart_snapshot_h5().
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-12-01 09:19:06 -03:00
e50c65e7dd
IO: Rewrite file division in read_restart_snapshot_xml().
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-12-01 08:55:39 -03:00
cc48f69803
IO: Fix the restart from the XML files with less processes.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-11-30 22:53:53 -03:00
0ac1fec2d7
IO: Rewrite read_restart_snapshot_xml() a bit.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-11-30 18:22:21 -03:00
863b7c5a85
IO: Rewrite write_binary_xml() so it accepts a pointer instead of array.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-11-30 17:45:33 -03:00
cb2d7b09cf
HASH, IO: Pass pointers to the hash functions instead of arrays.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-11-30 14:23:22 -03:00
b5183dfa0e
IO: Store hashes as integers in read_restart_snapshot_xml().
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-11-30 12:20:14 -03:00
a53601ff92
HASH, IO: Rewrite module HASH.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-11-30 12:04:30 -03:00
1fd1ee7602
IO: Add compression suffix to files individually.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-11-29 12:23:26 -03:00
88cdf23078
COMPRESSION, IO: Remove double buffer for compression.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-11-28 23:27:03 -03:00
9ddf7c7b44
IO: Use compression_bound() to determine the buffer size.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-11-28 22:38:30 -03:00
e6fc8e5d24
COMPRESSION: Pass the input length to compress().
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-11-28 22:19:50 -03:00
a03375122b
IO: Resume jobs from HDF5 restart snapshots for different nghosts.
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-11-25 10:33:55 -03:00
aea03e669e
MESH: Return call status from redistribute_blocks().
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-11-25 08:33:32 -03:00
a1cf8f9192
IO: Rewrite and rename store_snapshot_xml().
...
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2021-11-23 18:03:33 -03:00
