2974 Commits

Author SHA1 Message Date
8253aee8b8 Merge branch 'master' into flux-tubes 2023-12-18 09:35:38 -03:00
0c8cab74a2 IO, PARAMETERS, SYSTEM: Better parameter restoring.
This commit implements a better way (less I/O operations) restoring of
the parameters during the job restart. Some parameters, such as the size
of the block, or the equation system, cannot be changed during the
restart. Such parameters need to be restored from the restart snapshots.
The new way restores these parameters on the MPI master process only,
updates their values if they were changed, and distributes them to other
MPI processes. This way the number of I/O operations is kept to the
minimum (only one process access one file).

Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-12-18 09:30:07 -03:00
66f5fc5c58 PARAMETERS: Read parameters on master and distribute them.
This commit restores the previous way of processing the parameter file.
It is read and processed on the MPI master process, and then the list
of parameters is distributed to other MPI processes. This way only one
process accesses the parameter file, reducing the number of I/O
operations, which can be significant in the case of multiprocess MPI
jobs.

Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-12-17 21:47:38 -03:00
6847d1c994 PARAMETERS: Rewrite module with use of variable strings.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-12-17 18:09:12 -03:00
712933eecc Merge branch 'master' into flux-tubes 2023-12-14 18:21:11 -03:00
58b6a62ff8 IO: Fix wrong dimensions of corners() in store_metablocks_h5().
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-12-14 16:57:35 -03:00
3112b82258 Merge branch 'master' into flux-tubes 2023-09-18 09:25:38 -03:00
5333ebdf37 EVOLUTION: 3S*+ RK methods do not have extended CFL coefficients.
The methods by Ranocha et al. (2021) are error-controlled embedded RK
methods, therefore the CFL value should be set explicitely for these
methods.

Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-09-01 16:36:36 -03:00
62017bc5b3 EVOLUTION: Improve the conservation in SSPRK(10,4) method.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-30 17:37:09 -03:00
a896b7188a USER_PROBLEM: Do not limit B at boundaries in profile 2.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-29 18:43:44 -03:00
488eb72173 Merge branch 'master' into flux-tubes 2023-08-29 17:45:00 -03:00
26022c4ce0 EVOLUTION, FORCING: Add injected flag to update_forcing().
This flag indicates whether the energy injection was performed. If not,
there is no need for the variable update.

Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-29 17:43:03 -03:00
f50ccca3a2 Merge branch 'master' into flux-tubes 2023-08-29 17:23:32 -03:00
55b13fb895 EVOLUTION: Add missing variable update after adding turbulence forcing.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-29 17:09:48 -03:00
48f6618121 Revert "EVOLUTION: Update all blocks after the mesh update."
This reverts commit 8b36ce58e18bdc5af35c03571e77c5566efb2ae0.
2023-08-29 17:06:15 -03:00
b624595682 Merge branch 'master' into flux-tubes 2023-08-29 10:53:47 -03:00
dfe7310408 EVOLUTION: Improve the conservation in the SSPRK3 and SSPRK(4,3) methods.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-29 10:43:42 -03:00
91d3583a78 STATISTICS: Use Kahan summation for statistics integration.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-29 10:32:52 -03:00
8b36ce58e1 EVOLUTION: Update all blocks after the mesh update.
It seems that the selective block update, i.e., update of the blocks
which have been (de)refined only, is not perfect. It violates the
numerical conservation somehow, and requires more investigation.

In order to fix this, perform the update of all data blocks, no matter
if they were refined or not. This makes some blocks to be updated twice,
but resolves the problem of variable conservation.

Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-29 10:15:08 -03:00
39c3cdf7a4 EVOLUTION: Improve the conservation in the SSPRK3(2)4 method.
The 4th step produced errors large enought to give significant
conservation errors.

Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-28 17:53:59 -03:00
bb2d821df6 Merge branch 'master' into flux-tubes 2023-08-14 17:32:01 -03:00
03e99457cc MAKE: Fix mkdeps.sh.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-14 16:42:37 -03:00
47b3b50ae0 IO: Improve description of subroutine write_binary_xml().
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-14 13:05:24 -03:00
a71a1390ca IO: Formatting correction.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-14 12:49:42 -03:00
5574b6fcf1 IO: Rewrite storing of restart and regular XML snapshots.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-14 11:50:07 -03:00
b11a1fad58 XML: Add subroutines to initialize a new tree and add elements.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-14 10:13:40 -03:00
8ac5e49def IO: Enable compression when storing XML restart snapshot.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-12 07:08:48 -03:00
c015d5849a IO: Add missing item_size for 'complex64' in read_binary_xml().
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-11 19:54:29 -03:00
14fad08f47 IO: Rewrite read_restart_snapshot_xml().
This subroutine now used the new XML module to parge XML files. It also
used new subroutine read_binary_xml() to read binary files.

Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-11 19:41:00 -03:00
64966c3eb3 IO: Add subroutine to read binary XML files.
This subroutine is also responsible for detection if binary data are
compressed and encoded, and respectively decompress and decode them.

Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-11 19:29:08 -03:00
2a634faba4 XML: New module to handle XML files.
This module offers a user-friendly interface for parsing XML data and
retrieving elements and attributes.

Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-11 19:26:51 -03:00
fd7ba586a9 COMPRESSION: Implement decompressing subroutine.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-11 16:12:33 -03:00
01656102d4 COMPRESSION: Implement subroutine to decode data.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-11 15:23:40 -03:00
857baa1ce4 COMPRESSION: Add get_compressor_id() and get_encoder_id().
These functions return the compressor and encoder IDs based on their
names.

Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-11 15:09:50 -03:00
0e90f52118 COMPRESSION: Rename cbytes to bytes in compress().
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-07 19:04:53 -03:00
b9750f049b COMPRESSION: Better handle preferences in LZ4 compression.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-07 18:53:42 -03:00
fc6e8134ee COMPRESSION: Slightly rewrite ZSTD compression.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-07 18:31:29 -03:00
3cd8d789a6 COMPRESSION: Add lzma_bound() and better handle LZMA errors.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-08-07 18:14:56 -03:00
65c41105c6 Merge branch 'master' into flux-tubes 2023-08-05 19:02:40 -03:00
e8a220d9a7 PYTHON: Increase amunpy version to 0.9.11.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-07-29 18:40:27 -03:00
f2477396ca PYTHON: Change field 'data_filter' to 'data_encoder'.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-07-29 18:38:46 -03:00
b9c638c07b COMPRESSION, IO: Rewrite the compression.
This commit does not change any functionality. It just prepares for
easier implementation of the compression of the restart snapshots.

Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-07-29 18:35:45 -03:00
44abf23243 COMPRESSION: Add function to check if compression is used.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-07-29 16:29:44 -03:00
c7d8fdf512 COMPRESSION, IO: Rename compression_<format> to compressor_<format>.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-07-29 16:17:06 -03:00
db19393dd1 COMPRESSION: Make default compression levels more consistent.
The default compression levels correspond now to the default levels
defined by each compression format library.

Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-07-27 23:36:39 -03:00
6f0ecb0e42 Merge branch 'master' into flux-tubes 2023-07-27 18:44:45 -03:00
5efc38b8a1 COMPRESSION, IO: Implement data filters for better compression.
This commit implements two data filters:
1) "shuffle" - reorganizes bytes to group most to less significant parts
   together: |a1|a2|a3|b1|b2|b3| -> |a1|b1|c1|a2|b2|c2|a3|b3|c3|
2) "bytedelta" - additionally stores differences between subsequent
   values: |a1|a2|a3|b1|b2|b3| -> |a1|b1-a1|c1-b1|a2|b2-a2|c2-b2|a3|b3-a3|c3-b3|

For large datasets, in particular in 3D, the compression ratio can be
significantly better after applying these filters.

Inspired by https://aras-p.info/blog/2023/03/01/Float-Compression-7-More-Filtering-Optimization/

Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-07-27 18:35:54 -03:00
8c94659cca PYTHON: Increase amunpy version to 0.9.10.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-07-24 12:40:00 -03:00
5245070499 PYTHON: Add support for data filters for AmunXML format.
The supported data filters are 'shuffle' and 'bytedelta'.

Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-07-24 12:34:16 -03:00
d9a535a432 README: Update Build Status website.
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
2023-07-13 22:32:09 -03:00