gkowal/amun-code
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'master' into reconnection

Browse Source
This commit is contained in:
Grzegorz Kowal 2021-12-02 12:03:43 -03:00
commit 34b9ae3022
6 changed files with 1108 additions and 834 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

100
README.md
View File

@ -12,23 +12,26 @@ following features are already implemented:
* hydrodynamic and magnetohydrodynamic set of equations (HD and MHD), * hydrodynamic and magnetohydrodynamic set of equations (HD and MHD),
* both classical and special relativity cases for the above equations, * both classical and special relativity cases for the above equations,
* Cartesian coordinate system, * Cartesian coordinate system so far,
* uniform and adaptive mesh generation and update, * uniform and adaptive mesh generation and update,
* 2nd to 4th order time integration using Strong Stability Preserving * a number of time integration methods, from 2nd to 5th order Runge-Kutta
Runge-Kutta methods, methods: Strong Stability Preserving and Embedded (with the error control),
* 2nd order TVD interpolation with number of limiters and higher order * high order reconstructions: from 2nd to 9th order WENO and MP, both explicit
reconstructions, and compact methods, the 2nd order TVD interpolation has a number of limiters
supported,
* Riemann solvers of Roe- and HLL-types (HLL, HLLC, and HLLD), * Riemann solvers of Roe- and HLL-types (HLL, HLLC, and HLLD),
* standard boundary conditions: periodic, open, reflective, hydrostatic, etc. * standard boundary conditions: periodic, open, reflective, hydrostatic, etc.
* turbulence driving using Alvelius or Ornstein–Uhlenbeck methods, * turbulence driving using Alvelius or Ornstein–Uhlenbeck methods,
* viscous and resistive source terms, * viscous and resistive source terms,
* support for passive scalars (up to 100), * support for passive scalars,
* data stored in internal XML+binary or HDF5 format, * data stored in an internal XML+binary or the HDF5 format,
* support for Zstandard, LZ4, and LZMA compression in XML+binary format, * data integrity of the XML+binary format guaranteed by the XXH64 or XXH3 hashes;
* Python interface to read snapshots in both formats, * support for Zstandard, LZ4, and LZMA compressions in the XML+binary format,
* support for Deflate, SZIP, Zstandard, and ZFP compressions in the HDF5 format,
* easy and consistend Python interface to read snapshots in both formats,
* MPI parallelization, * MPI parallelization,
* completely written in Fortran 2008, * completely written in Fortran 2008,
* simple Makefile or CMake for executable building, * simple Makefile or CMake for building the code executable,
* minimum requirements, only Fortran compiler and Python are required to * minimum requirements, only Fortran compiler and Python are required to
prepare, run, and analyze your simulations. prepare, run, and analyze your simulations.
@ -62,63 +65,59 @@ Requirements
compiler version 9.0 or newer. compiler version 9.0 or newer.
- [NVIDIA HPC](https://developer.nvidia.com/hpc-sdk) compiler version 21.9. - [NVIDIA HPC](https://developer.nvidia.com/hpc-sdk) compiler version 21.9.
Warning: I could not make it run with the included MPI libraries. Warning: I could not make it run with the included MPI libraries.
* Optional, but recommended, [OpenMPI](https://www.open-mpi.org/) for parallel * Recommended, although optional, [OpenMPI](https://www.open-mpi.org/) for
runs, tested with version 1.8 or newer. parallel runs, tested with version 1.8 or newer.
* Optional support for XML-binary format compression requires: * Optional [CMake](https://cmake.org) version 3.16 or newer, for advanced
compilation option selection.
* Optionally, the XML-binary format compression requires:
[LZ4 library](https://lz4.github.io), [LZ4 library](https://lz4.github.io),
[Zstandard library](http://facebook.github.io/zstd/), or [Zstandard library](http://facebook.github.io/zstd/), or
[LZMA library](https://tukaani.org/xz/) [LZMA library](https://tukaani.org/xz/)
[XXHASH library](http://www.xxhash.com/).
* Optional [HDF5 libraries](https://www.hdfgroup.org/solutions/hdf5/), tested * Optional [HDF5 libraries](https://www.hdfgroup.org/solutions/hdf5/), tested
with version 1.10 or newer. The code now uses the new XML-binary snapshot with version 1.10 or newer. The code now uses the new XML-binary snapshot
format. However, if you still want to use older HDF5 snapshot format, you format. However, if you still want to use older HDF5 snapshot format, you
will need these libraries. will need these libraries.
* Optional [CMake](https://cmake.org) version 3.16 or newer, for managing the * Deflate compression is natively supported in HDF5 libraries, however,
build process. optionally these compression formats are supported through filters:
[SZIP](https://support.hdfgroup.org/doc_resource/SZIP/)
[HDF5Plugin-Zstandard](https://github.com/gkowal/HDF5Plugin-Zstandard),
Environment Variables [H5Z-ZFP](https://github.com/LLNL/H5Z-ZFP).
=====================
If you need to use the HDF5 libraries and they are not installed in the default
location, i.e. in the system directory **/usr**, make sure that the environment
variable _HDF5DIR_ is set in your **~/.bashrc** (or **~/.cshrc**) and pointing
to the location where the HDF5 libraries have been installed.
Recommended compilation (using CMake) Recommended compilation (using CMake)
===================================== =====================================
1. Clone the AMUN source code: 1. Clone the AMUN source code:
- from Bitbucket:
`git clone https://grzegorz_kowal@bitbucket.org/amunteam/amun-code.git`,
- from GitLab: - from GitLab:
`git clone https://gitlab.com/gkowal/amun-code.git` `git clone https://gitlab.com/gkowal/amun-code.git`
- from Bitbucket:
`git clone https://grzegorz_kowal@bitbucket.org/amunteam/amun-code.git`,
- or unpack the archive downloaded from page - or unpack the archive downloaded from page
[Downloads](https://bitbucket.org/amunteam/amun-code/downloads/). [Downloads](https://bitbucket.org/amunteam/amun-code/downloads/).
2. Create a directory for compilation in any location, 2. Create the build directory, e.g. `mkdir amun-build && cd amun-build`.
e.g. `mkdir cmake-build && cmake-build`.
3. Call `ccmake <path to amun-code>`, e.g. `ccmake ..`, and press 'c' once. 3. Call `ccmake <path to amun-code>`, e.g. `ccmake ..`, and press 'c' once.
Configure available options. Press 'c' once again, and 'g' to Set available options, if necessary. Press 'c' once again, and 'g' to
generate makefiles. Alternatively, just call `ccmake <path to amun-code>` generate makefiles.
for default options.
4. Compile the code using `make`. The executable file **amun.x** should be 4. Compile the code using `make`. The executable file **amun.x** should be
created. available in a few moments.
Alternative compilation (using `make.conf`) Alternative compilation (using `make`)
=========================================== ===========================================
1. Clone the AMUN source code: 1. Clone the AMUN source code:
- from Bitbucket:
`git clone https://grzegorz_kowal@bitbucket.org/amunteam/amun-code.git`,
- from GitLab: - from GitLab:
`git clone https://gitlab.com/gkowal/amun-code.git` `git clone https://gitlab.com/gkowal/amun-code.git`
- from Bitbucket:
`git clone https://grzegorz_kowal@bitbucket.org/amunteam/amun-code.git`,
- or unpack the archive downloaded from page - or unpack the archive downloaded from page
[Downloads](https://bitbucket.org/amunteam/amun-code/downloads/). [Downloads](https://bitbucket.org/amunteam/amun-code/downloads/).
2. Go to directory **build/hosts/** and copy file **default** to a new file named 2. Go to directory **build/hosts/** and copy file **default** to a new file
exactly as your host name, i.e. `cp default $HOSTNAME`. named exactly as your host name, i.e. `cp default $HOSTNAME`.
3. Customize your compiler and compilation options in your new host file. 3. Customize your compiler and compilation options in your new host file.
4. Go up to directory **build/** and copy file **make.default** to **make.config**. 4. Go up to the directory **build/** and copy file **make.default** to
**make.config**.
5. Customize compilation time options in **make.config**. 5. Customize compilation time options in **make.config**.
6. Compile sources by typing `make` in directory **build/**. The executable file 6. Compile sources by typing `make` in directory **build/**. The executable file
**amun.x** should be created there. **amun.x** should be created there.
@ -147,23 +146,24 @@ where N is the number of processors to use.
Reading data Reading data
============ ============
By default, the code uses new XML+binary snapshot data format. It can also be By default, the code uses the new XML+binary snapshot data format. Parameter
forced by setting parameter **snapshot_format** to **xml**. **snapshot_format** set to either **xml** or **h5** controls which file format
is used.
In order to read produced data in this format, you will need to install the In order to read the data produced in this format, you will need to install the
provided Python module. Simply change to **python/** directory and run Python module AmunPy included in subdirectory **python/amunpy**. Simply go to
`python setup.py install --user` this directory and run
`python ./setup.py install --user`
to install the module in your home directory. to install the module in your home directory.
Import the module in your python script using Import the module in your python script using
`from amunpy import *`, `from amunpy import *`,
and then initiate the interface using and then initiate the interface to the XML+binary snapshots using
`snapshot = AmunXML(<path to the snapshot directory>)` `snapshot = AmunXML(<path to the snapshot directory>)`
and read desired variable using or to the HDF5 files using
`snapshot = AmunH5(<path to any HDF5 snapshot file>)`
and read desired variables using function
`var = snapshot.dataset(<variable>)`. `var = snapshot.dataset(<variable>)`.
The function **dataset()** returns rescaled uniform mesh variable as NumPy The function **dataset()** returns the requested variable mapped on the uniform
array. mesh as a NumPy array.
If you want to read data from HDF5 snapshot, just use
`var = amun_dataset(<snapshot HDF5 file>, <variable>)`.

36
python/amunpy/src/amunpy/amunxml.py
View File

@ -208,14 +208,18 @@ class AmunXML(Amun):
return self.__swap__(dset) return self.__swap__(dset)
def __check_digest(self, filename, digest, data): def __check_digest(self, filename, hash_type, digest, data):
''' '''
Verifies if the provided digest matches the XXH64 hash of data Verifies if the provided digest matches the data.
stored in the given filename.
''' '''
import xxhash import xxhash
if digest.lower() != xxhash.xxh64(data).hexdigest(): failed = False
if hash_type == 'xxh64':
failed = digest.lower() != xxhash.xxh64(data).hexdigest()
elif hash_type == 'xxh3':
failed = digest.lower() != xxhash.xxh3_64(data).hexdigest()
if failed:
print("File '{}' seems to be corrupted! Proceeding anyway...".format(filename)) print("File '{}' seems to be corrupted! Proceeding anyway...".format(filename))
@ -235,7 +239,9 @@ class AmunXML(Amun):
if 'compression_format' in self.binaries[dataset]: if 'compression_format' in self.binaries[dataset]:
if 'compressed_digest' in self.binaries[dataset]: if 'compressed_digest' in self.binaries[dataset]:
self.__check_digest(fname, self.binaries[dataset]['compressed_digest'], stream) htype = self.binaries[dataset]['digest_type']
dhash = self.binaries[dataset]['compressed_digest']
self.__check_digest(fname, htype, dhash, stream)
comp = self.binaries[dataset]['compression_format'] comp = self.binaries[dataset]['compression_format']
if comp == 'zstd': if comp == 'zstd':
@ -248,12 +254,16 @@ class AmunXML(Amun):
raise Exception("Binary file '{}' compressed in unsupported format {}!".format(fname, comp)) raise Exception("Binary file '{}' compressed in unsupported format {}!".format(fname, comp))
if 'digest' in self.binaries[dataset]: if 'digest' in self.binaries[dataset]:
self.__check_digest(fname, self.binaries[dataset]['digest'], data) htype = self.binaries[dataset]['digest_type']
dhash = self.binaries[dataset]['digest']
self.__check_digest(fname, htype, dhash, data)
return numpy.frombuffer(data, dtype=dtype) return numpy.frombuffer(data, dtype=dtype)
else: else:
if 'digest' in self.binaries[dataset]: if 'digest' in self.binaries[dataset]:
self.__check_digest(fname, self.binaries[dataset]['digest'], stream) htype = self.binaries[dataset]['digest_type']
dhash = self.binaries[dataset]['digest']
self.__check_digest(fname, htype, dhash, stream)
return numpy.frombuffer(stream, dtype=dtype) return numpy.frombuffer(stream, dtype=dtype)
else: else:
@ -276,7 +286,9 @@ class AmunXML(Amun):
if 'compression_format' in self.chunks[chunk_number][dataset_name]: if 'compression_format' in self.chunks[chunk_number][dataset_name]:
if 'compressed_digest' in self.chunks[chunk_number][dataset_name]: if 'compressed_digest' in self.chunks[chunk_number][dataset_name]:
self.__check_digest(fname, self.chunks[chunk_number][dataset_name]['compressed_digest'], stream) htype = self.chunks[chunk_number][dataset_name]['digest_type']
dhash = self.chunks[chunk_number][dataset_name]['compressed_digest']
self.__check_digest(fname, htype, dhash, stream)
comp = self.chunks[chunk_number][dataset_name]['compression_format'] comp = self.chunks[chunk_number][dataset_name]['compression_format']
if comp == 'zstd': if comp == 'zstd':
@ -290,12 +302,16 @@ class AmunXML(Amun):
raise Exception("Binary file '{}' compressed in unsupported format {}!".format(fname, comp)) raise Exception("Binary file '{}' compressed in unsupported format {}!".format(fname, comp))
if 'digest' in self.chunks[chunk_number][dataset_name]: if 'digest' in self.chunks[chunk_number][dataset_name]:
self.__check_digest(fname, self.chunks[chunk_number][dataset_name]['digest'], data) htype = self.chunks[chunk_number][dataset_name]['digest_type']
dhash = self.chunks[chunk_number][dataset_name]['digest']
self.__check_digest(fname, htype, dhash, data)
return numpy.frombuffer(data, dtype=dtype) return numpy.frombuffer(data, dtype=dtype)
else: else:
if 'digest' in self.chunks[chunk_number][dataset_name]: if 'digest' in self.chunks[chunk_number][dataset_name]:
self.__check_digest(fname, self.chunks[chunk_number][dataset_name]['digest'], stream) htype = self.chunks[chunk_number][dataset_name]['digest_type']
dhash = self.chunks[chunk_number][dataset_name]['digest']
self.__check_digest(fname, htype, dhash, stream)
return numpy.frombuffer(stream, dtype=dtype) return numpy.frombuffer(stream, dtype=dtype)
else: else:

173
sources/compression.F90
View File

@ -36,15 +36,15 @@ module compression
#ifdef ZSTD #ifdef ZSTD
interface interface
integer(c_size_t) function zstd_bound(srcSize) & integer(c_size_t) function zstd_bound(srcSize) &
bind(C, name="ZSTD_compressBound") bind(C, name="ZSTD_compressBound")
use iso_c_binding, only: c_size_t use iso_c_binding, only: c_size_t
implicit none implicit none
integer(kind=c_size_t), value :: srcSize integer(kind=c_size_t), value :: srcSize
end function zstd_bound end function zstd_bound
integer(c_size_t) function zstd_compress(dst, dstCapacity, & integer(c_size_t) function zstd_compress(dst, dstCapacity, &
src, srcSize, level) & src, srcSize, level) &
bind(C, name="ZSTD_compress") bind(C, name="ZSTD_compress")
use iso_c_binding, only: c_size_t, c_int, c_ptr use iso_c_binding, only: c_size_t, c_int, c_ptr
implicit none implicit none
@ -53,6 +53,12 @@ module compression
integer(kind=c_int) , value :: level integer(kind=c_int) , value :: level
end function zstd_compress end function zstd_compress
integer function zstd_iserror(code) bind(C, name="ZSTD_isError")
use iso_c_binding, only: c_size_t
implicit none
integer(kind=c_size_t), value :: code
end function zstd_iserror
end interface end interface
#endif /* ZSTD */ #endif /* ZSTD */
#ifdef LZ4 #ifdef LZ4
@ -75,6 +81,12 @@ module compression
type(c_ptr) , value :: src, dst, prefsPtr type(c_ptr) , value :: src, dst, prefsPtr
end function lz4_compress end function lz4_compress
integer function lz4_iserror(code) bind(C, name="LZ4F_isError")
use iso_c_binding, only: c_size_t
implicit none
integer(kind=c_size_t), value :: code
end function lz4_iserror
end interface end interface
#endif /* LZ4 */ #endif /* LZ4 */
#ifdef LZMA #ifdef LZMA
@ -94,21 +106,25 @@ module compression
end interface end interface
#endif /* LZMA */ #endif /* LZMA */
! compression parameters
!
integer, save :: compression_format = 0
integer, save :: compression_level = 0
! supported compression formats ! supported compression formats
! !
integer, parameter :: compression_none = 0 enum, bind(c)
integer, parameter :: compression_zstd = 1 enumerator compression_none
integer, parameter :: compression_lz4 = 2 enumerator compression_zstd
integer, parameter :: compression_lzma = 3 enumerator compression_lz4
enumerator compression_lzma
end enum
! compression parameters
!
integer(kind(compression_none)), save :: compression_format = 0
integer , save :: compression_level = 0
character(len=4) , save :: compression_suffix = ''
private private
public :: set_compression, get_compression, compress public :: set_compression, get_compression, compression_bound, compress
public :: compression_suffix
!- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - !- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
! !
@ -131,17 +147,15 @@ module compression
! !
! cformat - the compression format string; ! cformat - the compression format string;
! clevel - the compression level; ! clevel - the compression level;
! suffix - the compressed file suffix;
! !
!=============================================================================== !===============================================================================
! !
subroutine set_compression(cformat, clevel, suffix) subroutine set_compression(cformat, clevel)
implicit none implicit none
character(len=*) , intent(inout) :: cformat character(len=*) , intent(inout) :: cformat
integer , intent(in) :: clevel integer , intent(in) :: clevel
character(len=8) , intent(out) :: suffix
!------------------------------------------------------------------------------- !-------------------------------------------------------------------------------
! !
@ -151,27 +165,27 @@ module compression
cformat = "zstd" cformat = "zstd"
compression_format = compression_zstd compression_format = compression_zstd
compression_level = max(0, min(19, clevel)) compression_level = max(0, min(19, clevel))
suffix = ".zst" compression_suffix = ".zst"
#endif /* ZSTD */ #endif /* ZSTD */
#ifdef LZ4 #ifdef LZ4
case("lz4", "LZ4") case("lz4", "LZ4")
cformat = "lz4" cformat = "lz4"
compression_format = compression_lz4 compression_format = compression_lz4
compression_level = max(1, min(12, clevel)) compression_level = max(1, min(12, clevel))
suffix = ".lz4" compression_suffix = ".lz4"
#endif /* LZ4 */ #endif /* LZ4 */
#ifdef LZMA #ifdef LZMA
case("lzma", "LZMA", "xz", "XZ") case("lzma", "LZMA", "xz", "XZ")
cformat = "lzma" cformat = "lzma"
compression_format = compression_lzma compression_format = compression_lzma
compression_level = max(0, min(9, clevel)) compression_level = max(0, min(9, clevel))
suffix = ".xz" compression_suffix = ".xz"
#endif /* LZMA */ #endif /* LZMA */
case default case default
cformat = "none" cformat = "none"
compression_format = compression_none compression_format = compression_none
compression_level = clevel compression_level = clevel
suffix = "" compression_suffix = ""
end select end select
!------------------------------------------------------------------------------- !-------------------------------------------------------------------------------
@ -203,6 +217,56 @@ module compression
! !
!=============================================================================== !===============================================================================
! !
! function COMPRESSION_BOUND:
! --------------------------
!
! Function returns the minimum buffer size required to perform
! the compression.
!
! Arguments:
!
! ilen - the length of the sequence of bytes to compress;
!
!===============================================================================
!
integer(kind=8) function compression_bound(ilen)
use iso_c_binding, only: c_loc
implicit none
integer(kind=8), intent(in) :: ilen
#ifdef LZ4
integer, dimension(14), target :: prefs = [5, 0, 1, 0, 0, 0, 0, 0, &
0, 0, 0, 0, 0, 0]
#endif /* LZ4 */
!-------------------------------------------------------------------------------
!
select case(compression_format)
#ifdef ZSTD
case(compression_zstd)
compression_bound = zstd_bound(ilen)
#endif /* ZSTD */
#ifdef LZ4
case(compression_lz4)
prefs(5:6) = transfer(ilen, [ 0_4 ])
prefs(9) = compression_level
compression_bound = lz4_bound(ilen, c_loc(prefs))
#endif /* LZ4 */
case default
compression_bound = ilen
end select
return
!-------------------------------------------------------------------------------
!
end function compression_bound
!
!===============================================================================
!
! subroutine COMPRESS: ! subroutine COMPRESS:
! ------------------- ! -------------------
! !
@ -210,80 +274,65 @@ module compression
! !
! Arguments: ! Arguments:
! !
! input - the input sequence of bytes; ! input - the pointer to the input sequence of bytes;
! ilen - the length of input;
! buffer - the buffer where the compression is done;
! clen - the length of buffer in bytes; once the compression was
! successful, it returns the length of the compressed stream;
! !
!=============================================================================== !===============================================================================
! !
subroutine compress(input, output, csize) subroutine compress(input, ilen, buffer, clen)
use iso_c_binding, only: c_int, c_loc use iso_c_binding, only : c_int, c_loc, c_ptr
#ifdef LZ4 #ifdef LZ4
use iso_c_binding, only: c_null_ptr use iso_c_binding, only : c_null_ptr
#endif /* LZ4 */ #endif /* LZ4 */
implicit none implicit none
integer(kind=1), dimension(:), target, intent(in) :: input type(c_ptr) , intent(in) :: input
integer(kind=1), dimension(:), target, intent(out) :: output integer(kind=8), intent(in) :: ilen
integer(kind=8) , target, intent(out) :: csize type(c_ptr) , intent(inout) :: buffer
integer(kind=8), intent(inout) :: clen
#ifdef LZ4 #ifdef LZ4
integer, dimension(14), target :: prefs = [5, 0, 1, 0, 0, 0, 0, 0, & integer, dimension(14), target :: prefs = [5, 0, 1, 0, 0, 0, 0, 0, &
0, 0, 0, 0, 0, 0] 0, 0, 0, 0, 0, 0]
#endif /* LZ4 */ #endif /* LZ4 */
#ifdef LZMA #ifdef LZMA
integer :: ret integer(kind=8), target :: lsize
integer :: ret
#endif /* LZMA */ #endif /* LZMA */
integer(kind=1), dimension(:), allocatable, target :: buffer
!------------------------------------------------------------------------------- !-------------------------------------------------------------------------------
! !
csize = min(size(input, kind=8), size(output, kind=8))
select case(compression_format) select case(compression_format)
#ifdef ZSTD #ifdef ZSTD
case(compression_zstd) case(compression_zstd)
allocate(buffer(zstd_bound(size(input, kind=8)))) clen = zstd_compress(buffer, clen, input, ilen, compression_level)
csize = zstd_compress(c_loc(buffer), size(buffer, kind=8), & if (zstd_iserror(clen) /= 0) clen = 0
c_loc(input), size(input, kind=8), &
compression_level)
if (csize > 0 .and. csize <= size(output, kind=8)) then
output(1:csize) = buffer(1:csize)
else
csize = -1
end if
deallocate(buffer)
#endif /* ZSTD */ #endif /* ZSTD */
#ifdef LZ4 #ifdef LZ4
case(compression_lz4) case(compression_lz4)
prefs(5:6) = transfer(size(input, kind=8), [ 0_4 ]) prefs(5:6) = transfer(ilen, [ 0_4 ])
prefs(9) = compression_level prefs(9) = compression_level
allocate(buffer(lz4_bound(size(input, kind=8), c_loc(prefs)))) clen = lz4_compress(buffer, clen, input, ilen, c_loc(prefs))
csize = lz4_compress(c_loc(buffer), size(buffer, kind=8), & if (lz4_iserror(clen) /= 0) clen = 0
c_loc(input), size(input, kind=8), c_loc(prefs))
if (csize > 0 .and. csize <= size(output, kind=8)) then
output(1:csize) = buffer(1:csize)
else
csize = -1
end if
deallocate(buffer)
#endif /* LZ4 */ #endif /* LZ4 */
#ifdef LZMA #ifdef LZMA
case(compression_lzma) case(compression_lzma)
csize = 0 lsize = 0
allocate(buffer(size(input))) ret = lzma_compress(compression_level, 4, c_null_ptr, &
ret = lzma_compress(compression_level, 4, c_null_ptr, & input, ilen, buffer, c_loc(lsize), clen)
c_loc(input), size(input, kind=8), & if (ret > 0) then
c_loc(buffer), c_loc(csize), size(buffer, kind=8)) clen = 0
if (ret == 0 .and. csize <= size(output, kind=8)) then
output(1:csize) = buffer(1:csize)
else else
csize = -1 clen = lsize
end if end if
deallocate(buffer)
#endif /* LZMA */ #endif /* LZMA */
case default case default
output(1:csize) = input(1:csize) clen = 0
end select end select
!------------------------------------------------------------------------------- !-------------------------------------------------------------------------------

2
sources/forcing.F90
View File

@ -113,7 +113,7 @@ module forcing
! array for driving mode coefficients ! array for driving mode coefficients
! !
complex(kind=8), dimension(:,:), allocatable :: fcoefs complex(kind=8), dimension(:,:), allocatable, target :: fcoefs
! velocity Fourier coefficients in Alfvelius method ! velocity Fourier coefficients in Alfvelius method
! !

287
sources/hash.F90
View File

@ -24,9 +24,10 @@
!! !!
!! module: HASH !! module: HASH
!! !!
!! This module provides 64-bit version of the xxHash64 by Yann Collet. !! This module is an interface to the XXH functions by Yann Collet provided
!! This is a Fortran implementation based on the XXH64 specification !! by the library libxxhash. If this library is not available, an internal
!! published at !! Fortran implementation of the 64-bit version of the xxHash64 is used.
!! The Fortran implementation is based on the XXH64 specification published at
!! https://github.com/Cyan4973/xxHash/blob/dev/doc/xxhash_spec.md !! https://github.com/Cyan4973/xxHash/blob/dev/doc/xxhash_spec.md
!! !!
!! For additional info, see !! For additional info, see
@ -39,58 +40,56 @@ module hash
implicit none implicit none
#ifdef XXHASH #ifdef XXHASH
! interfaces to functions XXH64() and XXH3_64bits() provided ! interfaces to functions XXH64() and XXH3_64bits() provided by
! by the systems library libxxhash ! the library libxxhash
! !
interface interface
integer(c_int64_t) function xxh64_lib(input, length, seed) &
integer(c_int64_t) function xxh64_system(input, length, seed) & bind(C, name="XXH64")
bind(C, name="XXH64")
use iso_c_binding, only: c_ptr, c_size_t, c_int64_t use iso_c_binding, only: c_ptr, c_size_t, c_int64_t
implicit none implicit none
type(c_ptr) , value :: input type(c_ptr) , value :: input
integer(kind=c_size_t), value :: length integer(kind=c_size_t), value :: length
integer(c_int64_t) , value :: seed integer(c_int64_t) , value :: seed
end function xxh64_system end function xxh64_lib
integer(c_int64_t) function xxh3_system(input, length) & integer(c_int64_t) function xxh3_lib(input, length) &
bind(C, name="XXH3_64bits") bind(C, name="XXH3_64bits")
use iso_c_binding, only: c_ptr, c_size_t, c_int64_t use iso_c_binding, only: c_ptr, c_size_t, c_int64_t
implicit none implicit none
type(c_ptr) , value :: input type(c_ptr) , value :: input
integer(kind=c_size_t), value :: length integer(kind=c_size_t), value :: length
end function xxh3_system end function xxh3_lib
end interface end interface
#else /* XXHASH */ #else /* XXHASH */
! hash parameters ! hash parameters
! !
integer(kind=8), parameter :: prime1 = -7046029288634856825_8, & integer(kind=8), parameter :: prime1 = -7046029288634856825_8, &
prime2 = -4417276706812531889_8, & prime2 = -4417276706812531889_8, &
prime3 = 1609587929392839161_8, & prime3 = 1609587929392839161_8, &
prime4 = -8796714831421723037_8, & prime4 = -8796714831421723037_8, &
prime5 = 2870177450012600261_8, & prime5 = 2870177450012600261_8, &
prime6 = 6983438078262162902_8 prime6 = 6983438078262162902_8
#endif /* XXHASH */ #endif /* XXHASH */
! supported hash types ! supported hash types
! !
integer, parameter :: hash_xxh64 = 1 enum, bind(c)
enumerator hash_none
enumerator hash_xxh64
#ifdef XXHASH #ifdef XXHASH
integer, parameter :: hash_xxh3 = 2 enumerator hash_xxh3
#endif /* XXHASH */ #endif /* XXHASH */
end enum
private private
public :: get_hash, check_hash, hash_string, hash_xxh64 public :: hash_info, hash_name, digest, check_digest
#ifdef XXHASH public :: digest_string, digest_integer
public :: hash_xxh3
#endif /* XXHASH */
!- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - !- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
! !
contains contains
! !
!
!=============================================================================== !===============================================================================
!! !!
!!*** PUBLIC SUBROUTINES ***************************************************** !!*** PUBLIC SUBROUTINES *****************************************************
@ -99,122 +98,242 @@ module hash
! !
!=============================================================================== !===============================================================================
! !
! function HASH_STRING: ! subroutine HASH_INFO:
! -------------------- ! --------------------
! !
! Function returns the hash type string. ! Subroutine returns the hash ID and the length in bytes (characters)
! by the provided hash name.
! !
!=============================================================================== !===============================================================================
! !
character(len=8) function hash_string(hash_type) subroutine hash_info(hash_name, hash_id, hash_length)
use helpers, only : print_message
implicit none
character(len=*), intent(in) :: hash_name
integer , intent(out) :: hash_id, hash_length
character(len=*), parameter :: loc = "HASH::hash_info()"
!-------------------------------------------------------------------------------
!
select case(trim(hash_name))
case("xxh64", "XXH64")
hash_id = hash_xxh64
hash_length = 16
#ifdef XXHASH
case("xxh3", "XXH3")
hash_id = hash_xxh3
hash_length = 16
#endif /* XXHASH */
case("none")
hash_id = hash_none
hash_length = 0
case default
call print_message(loc, &
"Hash '" // trim(hash_name) // "' is not supported!")
end select
return
!-------------------------------------------------------------------------------
!
end subroutine hash_info
!
!===============================================================================
!
! function HASH_NAME:
! ------------------
!
! Function returns the hash name by the provided hash ID.
!
!===============================================================================
!
character(len=8) function hash_name(hash_type)
implicit none implicit none
integer, intent(in) :: hash_type integer, intent(in) :: hash_type
integer(kind(hash_none)) :: htype
!------------------------------------------------------------------------------- !-------------------------------------------------------------------------------
! !
htype = hash_type
select case(htype)
case(hash_xxh64)
hash_name = "xxh64"
#ifdef XXHASH #ifdef XXHASH
if (hash_type == hash_xxh3) then case(hash_xxh3)
hash_string = 'xxh3' hash_name = "xxh3"
return
end if
#endif /* XXHASH */ #endif /* XXHASH */
hash_string = 'xxh64' case default
hash_name = "none"
end select
return return
!------------------------------------------------------------------------------- !-------------------------------------------------------------------------------
! !
end function hash_string end function hash_name
! !
!=============================================================================== !===============================================================================
! !
! function GET_HASH: ! function DIGEST:
! ----------------- ! ---------------
! !
! Function calculates the hash for a given sequence of bytes. ! Function calculates the digest for a given sequence of bytes.
! !
! Arguments: ! Arguments:
! !
! input - the input sequence of bytes; ! buffer - the buffer pointer;
! hash_type - the number corresponding to the hash type; ! length - the buffer length;
! hash_id - the hash ID;
! !
!=============================================================================== !===============================================================================
! !
integer(kind=8) function get_hash(input, hash_type) result(hash) integer(kind=8) function digest(buffer, length, hash_id) result(hash)
#ifdef XXHASH use iso_c_binding, only : c_ptr
use iso_c_binding, only: c_loc
#endif /* XXHASH */
implicit none implicit none
integer(kind=1), dimension(:), target, intent(in) :: input type(c_ptr) , intent(in) :: buffer
integer , intent(in) :: hash_type integer(kind=8), intent(in) :: length
integer , intent(in) :: hash_id
#ifdef XXHASH
integer(kind=8) :: length
#endif /* XXHASH */
!------------------------------------------------------------------------------- !-------------------------------------------------------------------------------
! !
#ifdef XXHASH select case(hash_id)
hash = 0 case(hash_xxh64)
length = size(input, kind=8) #ifndef XXHASH
hash = xxh64(buffer, length)
select case(hash_type)
case(hash_xxh3)
hash = xxh3_system(c_loc(input), length)
case default
hash = xxh64_system(c_loc(input), length, hash)
end select
#else /* XXHASH */ #else /* XXHASH */
hash = xxh64(input) hash = xxh64_lib(buffer, length, 0_8)
case(hash_xxh3)
hash = xxh3_lib(buffer, length)
#endif /* XXHASH */ #endif /* XXHASH */
case(hash_none)
hash = 0
end select
return return
!------------------------------------------------------------------------------- !-------------------------------------------------------------------------------
! !
end function get_hash end function digest
! !
!=============================================================================== !===============================================================================
! !
! subroutine CHECK_HASH: ! subroutine CHECK_DIGEST:
! --------------------- ! -----------------------
! !
! Subroutine checks if the provided digest matches the digest of ! Subroutine checks if the provided digest matches the digest of
! the input data. ! the input data.
! !
! Arguments: ! Arguments:
! !
! loc - the location of check; ! loc - the location of check;
! fname - the file name; ! fname - the file name;
! input - the input sequence of bytes; ! buffer - the buffer pointer;
! digest - the data digest to check; ! length - the buffer length;
! hash_type - the number corresponding to the hash type; ! bdigest - the buffer digest to check;
! hash_id - the hash ID;
! !
!=============================================================================== !===============================================================================
! !
subroutine check_hash(loc, fname, input, digest, hash_type) subroutine check_digest(loc, fname, buffer, length, bdigest, hash_id)
use helpers , only : print_message
use iso_c_binding, only : c_ptr
implicit none
character(len=*), intent(in) :: loc, fname
type(c_ptr) , intent(in) :: buffer
integer(kind=8) , intent(in) :: length
integer(kind=8) , intent(in) :: bdigest
integer , intent(in) :: hash_id
!-------------------------------------------------------------------------------
!
if (hash_id == hash_none) return
if (bdigest /= digest(buffer, length, hash_id)) &
call print_message(loc, trim(fname) // " seems to be corrupted!")
!-------------------------------------------------------------------------------
!
end subroutine check_digest
!
!===============================================================================
!
! subroutine DIGEST_STRING:
! ------------------------
!
! Subroutine converts the integer digest to string.
!
! Arguments:
!
! idigest - the digest as integer;
! sdigest - the digest as string;
!
!===============================================================================
!
subroutine digest_string(idigest, sdigest)
use helpers, only : print_message use helpers, only : print_message
implicit none implicit none
character(len=*) , intent(in) :: loc, fname integer(kind=8) , intent(in) :: idigest
integer(kind=1), dimension(:), intent(in) :: input character(len=*), intent(inout) :: sdigest
integer(kind=8) , intent(in) :: digest
integer , intent(in) :: hash_type character(len=*), parameter :: loc = "HASH::digest_string()"
!------------------------------------------------------------------------------- !-------------------------------------------------------------------------------
! !
if (digest /= get_hash(input, hash_type)) & if (len(sdigest) >= 16) then
call print_message(loc, trim(fname) // " seems to be corrupted!") write(sdigest,"(1z16.16)") idigest
else
call print_message(loc, &
"The string is too short to contain the whole digest!")
end if
!------------------------------------------------------------------------------- !-------------------------------------------------------------------------------
! !
end subroutine check_hash end subroutine digest_string
!
!===============================================================================
!
! subroutine DIGEST_INTEGER:
! -------------------------
!
! Subroutine converts the string digest to its integer representation.
!
! Arguments:
!
! sdigest - the digest as string;
! idigest - the digest as integer;
!
!===============================================================================
!
subroutine digest_integer(sdigest, idigest)
implicit none
character(len=*), intent(in) :: sdigest
integer(kind=8) , intent(out) :: idigest
!-------------------------------------------------------------------------------
!
read(sdigest, fmt="(1z16)") idigest
!-------------------------------------------------------------------------------
!
end subroutine digest_integer
! !
!=============================================================================== !===============================================================================
!! !!
@ -232,24 +351,30 @@ module hash
! !
! Arguments: ! Arguments:
! !
! input - the input sequence of bytes; ! buffer - the buffer pointer;
! length - the buffer length;
! !
!=============================================================================== !===============================================================================
! !
integer(kind=8) function xxh64(input) result(hash) integer(kind=8) function xxh64(buffer, length) result(hash)
use iso_c_binding, only : c_ptr, c_f_pointer
implicit none implicit none
integer(kind=1), dimension(:), target, intent(in) :: input type(c_ptr) , intent(in) :: buffer
integer(kind=8), intent(in) :: length
integer(kind=8) :: length
integer(kind=8) :: remaining, offset integer(kind=8) :: remaining, offset
integer(kind=8), dimension(4) :: lane, chunk integer(kind=8), dimension(4) :: lane, chunk
integer(kind=1), dimension(:), pointer :: input
!------------------------------------------------------------------------------- !-------------------------------------------------------------------------------
! !
length = size(input, kind=8) call c_f_pointer(buffer, input, [ length ])
hash = 0_8 hash = 0_8
offset = 1_8 offset = 1_8
remaining = length remaining = length

1344
sources/io.F90
View File

File diff suppressed because it is too large Load Diff