gkowal/amun-code
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'master' into binaries

Browse Source
This commit is contained in:
Grzegorz Kowal 2021-12-02 12:03:19 -03:00
commit 3d9c7c6990
6 changed files with 1108 additions and 834 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

100
README.md
View File

@ -12,23 +12,26 @@ following features are already implemented:
* hydrodynamic and magnetohydrodynamic set of equations (HD and MHD),
* both classical and special relativity cases for the above equations,
* Cartesian coordinate system,
* Cartesian coordinate system so far,
* uniform and adaptive mesh generation and update,
* 2nd to 4th order time integration using Strong Stability Preserving
Runge-Kutta methods,
* 2nd order TVD interpolation with number of limiters and higher order
reconstructions,
* a number of time integration methods, from 2nd to 5th order Runge-Kutta
methods: Strong Stability Preserving and Embedded (with the error control),
* high order reconstructions: from 2nd to 9th order WENO and MP, both explicit
and compact methods, the 2nd order TVD interpolation has a number of limiters
supported,
* Riemann solvers of Roe- and HLL-types (HLL, HLLC, and HLLD),
* standard boundary conditions: periodic, open, reflective, hydrostatic, etc.
* turbulence driving using Alvelius or Ornstein–Uhlenbeck methods,
* viscous and resistive source terms,
* support for passive scalars (up to 100),
* data stored in internal XML+binary or HDF5 format,
* support for Zstandard, LZ4, and LZMA compression in XML+binary format,
* Python interface to read snapshots in both formats,
* support for passive scalars,
* data stored in an internal XML+binary or the HDF5 format,
* data integrity of the XML+binary format guaranteed by the XXH64 or XXH3 hashes;
* support for Zstandard, LZ4, and LZMA compressions in the XML+binary format,
* support for Deflate, SZIP, Zstandard, and ZFP compressions in the HDF5 format,
* easy and consistend Python interface to read snapshots in both formats,
* MPI parallelization,
* completely written in Fortran 2008,
* simple Makefile or CMake for executable building,
* simple Makefile or CMake for building the code executable,
* minimum requirements, only Fortran compiler and Python are required to
prepare, run, and analyze your simulations.
@ -62,63 +65,59 @@ Requirements
compiler version 9.0 or newer.
- [NVIDIA HPC](https://developer.nvidia.com/hpc-sdk) compiler version 21.9.
Warning: I could not make it run with the included MPI libraries.
* Optional, but recommended, [OpenMPI](https://www.open-mpi.org/) for parallel
runs, tested with version 1.8 or newer.
* Optional support for XML-binary format compression requires:
* Recommended, although optional, [OpenMPI](https://www.open-mpi.org/) for
parallel runs, tested with version 1.8 or newer.
* Optional [CMake](https://cmake.org) version 3.16 or newer, for advanced
compilation option selection.
* Optionally, the XML-binary format compression requires:
[LZ4 library](https://lz4.github.io),
[Zstandard library](http://facebook.github.io/zstd/), or
[LZMA library](https://tukaani.org/xz/)
[XXHASH library](http://www.xxhash.com/).
* Optional [HDF5 libraries](https://www.hdfgroup.org/solutions/hdf5/), tested
with version 1.10 or newer. The code now uses the new XML-binary snapshot
format. However, if you still want to use older HDF5 snapshot format, you
will need these libraries.
* Optional [CMake](https://cmake.org) version 3.16 or newer, for managing the
build process.
Environment Variables
=====================
If you need to use the HDF5 libraries and they are not installed in the default
location, i.e. in the system directory **/usr**, make sure that the environment
variable _HDF5DIR_ is set in your **~/.bashrc** (or **~/.cshrc**) and pointing
to the location where the HDF5 libraries have been installed.
* Deflate compression is natively supported in HDF5 libraries, however,
optionally these compression formats are supported through filters:
[SZIP](https://support.hdfgroup.org/doc_resource/SZIP/)
[HDF5Plugin-Zstandard](https://github.com/gkowal/HDF5Plugin-Zstandard),
[H5Z-ZFP](https://github.com/LLNL/H5Z-ZFP).
Recommended compilation (using CMake)
=====================================
1. Clone the AMUN source code:
- from Bitbucket:
`git clone https://grzegorz_kowal@bitbucket.org/amunteam/amun-code.git`,
- from GitLab:
`git clone https://gitlab.com/gkowal/amun-code.git`
- from Bitbucket:
`git clone https://grzegorz_kowal@bitbucket.org/amunteam/amun-code.git`,
- or unpack the archive downloaded from page
[Downloads](https://bitbucket.org/amunteam/amun-code/downloads/).
2. Create a directory for compilation in any location,
e.g. `mkdir cmake-build && cmake-build`.
2. Create the build directory, e.g. `mkdir amun-build && cd amun-build`.
3. Call `ccmake <path to amun-code>`, e.g. `ccmake ..`, and press 'c' once.
Configure available options. Press 'c' once again, and 'g' to
generate makefiles. Alternatively, just call `ccmake <path to amun-code>`
for default options.
Set available options, if necessary. Press 'c' once again, and 'g' to
generate makefiles.
4. Compile the code using `make`. The executable file **amun.x** should be
created.
available in a few moments.
Alternative compilation (using `make.conf`)
Alternative compilation (using `make`)
===========================================
1. Clone the AMUN source code:
- from Bitbucket:
`git clone https://grzegorz_kowal@bitbucket.org/amunteam/amun-code.git`,
- from GitLab:
`git clone https://gitlab.com/gkowal/amun-code.git`
- from Bitbucket:
`git clone https://grzegorz_kowal@bitbucket.org/amunteam/amun-code.git`,
- or unpack the archive downloaded from page
[Downloads](https://bitbucket.org/amunteam/amun-code/downloads/).
2. Go to directory **build/hosts/** and copy file **default** to a new file named
exactly as your host name, i.e. `cp default $HOSTNAME`.
2. Go to directory **build/hosts/** and copy file **default** to a new file
named exactly as your host name, i.e. `cp default $HOSTNAME`.
3. Customize your compiler and compilation options in your new host file.
4. Go up to directory **build/** and copy file **make.default** to **make.config**.
4. Go up to the directory **build/** and copy file **make.default** to
**make.config**.
5. Customize compilation time options in **make.config**.
6. Compile sources by typing `make` in directory **build/**. The executable file
**amun.x** should be created there.
@ -147,23 +146,24 @@ where N is the number of processors to use.
Reading data
============
By default, the code uses new XML+binary snapshot data format. It can also be
forced by setting parameter **snapshot_format** to **xml**.
By default, the code uses the new XML+binary snapshot data format. Parameter
**snapshot_format** set to either **xml** or **h5** controls which file format
is used.
In order to read produced data in this format, you will need to install the
provided Python module. Simply change to **python/** directory and run
`python setup.py install --user`
In order to read the data produced in this format, you will need to install the
Python module AmunPy included in subdirectory **python/amunpy**. Simply go to
this directory and run
`python ./setup.py install --user`
to install the module in your home directory.
Import the module in your python script using
`from amunpy import *`,
and then initiate the interface using
and then initiate the interface to the XML+binary snapshots using
`snapshot = AmunXML(<path to the snapshot directory>)`
and read desired variable using
or to the HDF5 files using
`snapshot = AmunH5(<path to any HDF5 snapshot file>)`
and read desired variables using function
`var = snapshot.dataset(<variable>)`.
The function **dataset()** returns rescaled uniform mesh variable as NumPy
array.
If you want to read data from HDF5 snapshot, just use
`var = amun_dataset(<snapshot HDF5 file>, <variable>)`.
The function **dataset()** returns the requested variable mapped on the uniform
mesh as a NumPy array.

36
python/amunpy/src/amunpy/amunxml.py
View File

@ -208,14 +208,18 @@ class AmunXML(Amun):
return self.__swap__(dset)
def __check_digest(self, filename, digest, data):
def __check_digest(self, filename, hash_type, digest, data):
'''
Verifies if the provided digest matches the XXH64 hash of data
stored in the given filename.
Verifies if the provided digest matches the data.
'''
import xxhash
if digest.lower() != xxhash.xxh64(data).hexdigest():
failed = False
if hash_type == 'xxh64':
failed = digest.lower() != xxhash.xxh64(data).hexdigest()
elif hash_type == 'xxh3':
failed = digest.lower() != xxhash.xxh3_64(data).hexdigest()
if failed:
print("File '{}' seems to be corrupted! Proceeding anyway...".format(filename))
@ -235,7 +239,9 @@ class AmunXML(Amun):
if 'compression_format' in self.binaries[dataset]:
if 'compressed_digest' in self.binaries[dataset]:
self.__check_digest(fname, self.binaries[dataset]['compressed_digest'], stream)
htype = self.binaries[dataset]['digest_type']
dhash = self.binaries[dataset]['compressed_digest']
self.__check_digest(fname, htype, dhash, stream)
comp = self.binaries[dataset]['compression_format']
if comp == 'zstd':
@ -248,12 +254,16 @@ class AmunXML(Amun):
raise Exception("Binary file '{}' compressed in unsupported format {}!".format(fname, comp))
if 'digest' in self.binaries[dataset]:
self.__check_digest(fname, self.binaries[dataset]['digest'], data)
htype = self.binaries[dataset]['digest_type']
dhash = self.binaries[dataset]['digest']
self.__check_digest(fname, htype, dhash, data)
return numpy.frombuffer(data, dtype=dtype)
else:
if 'digest' in self.binaries[dataset]:
self.__check_digest(fname, self.binaries[dataset]['digest'], stream)
htype = self.binaries[dataset]['digest_type']
dhash = self.binaries[dataset]['digest']
self.__check_digest(fname, htype, dhash, stream)
return numpy.frombuffer(stream, dtype=dtype)
else:
@ -276,7 +286,9 @@ class AmunXML(Amun):
if 'compression_format' in self.chunks[chunk_number][dataset_name]:
if 'compressed_digest' in self.chunks[chunk_number][dataset_name]:
self.__check_digest(fname, self.chunks[chunk_number][dataset_name]['compressed_digest'], stream)
htype = self.chunks[chunk_number][dataset_name]['digest_type']
dhash = self.chunks[chunk_number][dataset_name]['compressed_digest']
self.__check_digest(fname, htype, dhash, stream)
comp = self.chunks[chunk_number][dataset_name]['compression_format']
if comp == 'zstd':
@ -290,12 +302,16 @@ class AmunXML(Amun):
raise Exception("Binary file '{}' compressed in unsupported format {}!".format(fname, comp))
if 'digest' in self.chunks[chunk_number][dataset_name]:
self.__check_digest(fname, self.chunks[chunk_number][dataset_name]['digest'], data)
htype = self.chunks[chunk_number][dataset_name]['digest_type']
dhash = self.chunks[chunk_number][dataset_name]['digest']
self.__check_digest(fname, htype, dhash, data)
return numpy.frombuffer(data, dtype=dtype)
else:
if 'digest' in self.chunks[chunk_number][dataset_name]:
self.__check_digest(fname, self.chunks[chunk_number][dataset_name]['digest'], stream)
htype = self.chunks[chunk_number][dataset_name]['digest_type']
dhash = self.chunks[chunk_number][dataset_name]['digest']
self.__check_digest(fname, htype, dhash, stream)
return numpy.frombuffer(stream, dtype=dtype)
else:

173
sources/compression.F90
View File

@ -36,15 +36,15 @@ module compression
#ifdef ZSTD
interface
integer(c_size_t) function zstd_bound(srcSize) &
integer(c_size_t) function zstd_bound(srcSize) &
bind(C, name="ZSTD_compressBound")
use iso_c_binding, only: c_size_t
implicit none
integer(kind=c_size_t), value :: srcSize
end function zstd_bound
integer(c_size_t) function zstd_compress(dst, dstCapacity, &
src, srcSize, level) &
integer(c_size_t) function zstd_compress(dst, dstCapacity, &
src, srcSize, level) &
bind(C, name="ZSTD_compress")
use iso_c_binding, only: c_size_t, c_int, c_ptr
implicit none
@ -53,6 +53,12 @@ module compression
integer(kind=c_int) , value :: level
end function zstd_compress
integer function zstd_iserror(code) bind(C, name="ZSTD_isError")
use iso_c_binding, only: c_size_t
implicit none
integer(kind=c_size_t), value :: code
end function zstd_iserror
end interface
#endif /* ZSTD */
#ifdef LZ4
@ -75,6 +81,12 @@ module compression
type(c_ptr) , value :: src, dst, prefsPtr
end function lz4_compress
integer function lz4_iserror(code) bind(C, name="LZ4F_isError")
use iso_c_binding, only: c_size_t
implicit none
integer(kind=c_size_t), value :: code
end function lz4_iserror
end interface
#endif /* LZ4 */
#ifdef LZMA
@ -94,21 +106,25 @@ module compression
end interface
#endif /* LZMA */
! compression parameters
!
integer, save :: compression_format = 0
integer, save :: compression_level = 0
! supported compression formats
!
integer, parameter :: compression_none = 0
integer, parameter :: compression_zstd = 1
integer, parameter :: compression_lz4 = 2
integer, parameter :: compression_lzma = 3
enum, bind(c)
enumerator compression_none
enumerator compression_zstd
enumerator compression_lz4
enumerator compression_lzma
end enum
! compression parameters
!
integer(kind(compression_none)), save :: compression_format = 0
integer , save :: compression_level = 0
character(len=4) , save :: compression_suffix = ''
private
public :: set_compression, get_compression, compress
public :: set_compression, get_compression, compression_bound, compress
public :: compression_suffix
!- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!
@ -131,17 +147,15 @@ module compression
!
! cformat - the compression format string;
! clevel - the compression level;
! suffix - the compressed file suffix;
!
!===============================================================================
!
subroutine set_compression(cformat, clevel, suffix)
subroutine set_compression(cformat, clevel)
implicit none
character(len=*) , intent(inout) :: cformat
integer , intent(in) :: clevel
character(len=8) , intent(out) :: suffix
!-------------------------------------------------------------------------------
!
@ -151,27 +165,27 @@ module compression
cformat = "zstd"
compression_format = compression_zstd
compression_level = max(0, min(19, clevel))
suffix = ".zst"
compression_suffix = ".zst"
#endif /* ZSTD */
#ifdef LZ4
case("lz4", "LZ4")
cformat = "lz4"
compression_format = compression_lz4
compression_level = max(1, min(12, clevel))
suffix = ".lz4"
compression_suffix = ".lz4"
#endif /* LZ4 */
#ifdef LZMA
case("lzma", "LZMA", "xz", "XZ")
cformat = "lzma"
compression_format = compression_lzma
compression_level = max(0, min(9, clevel))
suffix = ".xz"
compression_suffix = ".xz"
#endif /* LZMA */
case default
cformat = "none"
compression_format = compression_none
compression_level = clevel
suffix = ""
compression_suffix = ""
end select
!-------------------------------------------------------------------------------
@ -203,6 +217,56 @@ module compression
!
!===============================================================================
!
! function COMPRESSION_BOUND:
! --------------------------
!
! Function returns the minimum buffer size required to perform
! the compression.
!
! Arguments:
!
! ilen - the length of the sequence of bytes to compress;
!
!===============================================================================
!
integer(kind=8) function compression_bound(ilen)
use iso_c_binding, only: c_loc
implicit none
integer(kind=8), intent(in) :: ilen
#ifdef LZ4
integer, dimension(14), target :: prefs = [5, 0, 1, 0, 0, 0, 0, 0, &
0, 0, 0, 0, 0, 0]
#endif /* LZ4 */
!-------------------------------------------------------------------------------
!
select case(compression_format)
#ifdef ZSTD
case(compression_zstd)
compression_bound = zstd_bound(ilen)
#endif /* ZSTD */
#ifdef LZ4
case(compression_lz4)
prefs(5:6) = transfer(ilen, [ 0_4 ])
prefs(9) = compression_level
compression_bound = lz4_bound(ilen, c_loc(prefs))
#endif /* LZ4 */
case default
compression_bound = ilen
end select
return
!-------------------------------------------------------------------------------
!
end function compression_bound
!
!===============================================================================
!
! subroutine COMPRESS:
! -------------------
!
@ -210,80 +274,65 @@ module compression
!
! Arguments:
!
! input - the input sequence of bytes;
! input - the pointer to the input sequence of bytes;
! ilen - the length of input;
! buffer - the buffer where the compression is done;
! clen - the length of buffer in bytes; once the compression was
! successful, it returns the length of the compressed stream;
!
!===============================================================================
!
subroutine compress(input, output, csize)
subroutine compress(input, ilen, buffer, clen)
use iso_c_binding, only: c_int, c_loc
use iso_c_binding, only : c_int, c_loc, c_ptr
#ifdef LZ4
use iso_c_binding, only: c_null_ptr
use iso_c_binding, only : c_null_ptr
#endif /* LZ4 */
implicit none
integer(kind=1), dimension(:), target, intent(in) :: input
integer(kind=1), dimension(:), target, intent(out) :: output
integer(kind=8) , target, intent(out) :: csize
type(c_ptr) , intent(in) :: input
integer(kind=8), intent(in) :: ilen
type(c_ptr) , intent(inout) :: buffer
integer(kind=8), intent(inout) :: clen
#ifdef LZ4
integer, dimension(14), target :: prefs = [5, 0, 1, 0, 0, 0, 0, 0, &
integer, dimension(14), target :: prefs = [5, 0, 1, 0, 0, 0, 0, 0, &
0, 0, 0, 0, 0, 0]
#endif /* LZ4 */
#ifdef LZMA
integer :: ret
integer(kind=8), target :: lsize
integer :: ret
#endif /* LZMA */
integer(kind=1), dimension(:), allocatable, target :: buffer
!-------------------------------------------------------------------------------
!
csize = min(size(input, kind=8), size(output, kind=8))
select case(compression_format)
#ifdef ZSTD
case(compression_zstd)
allocate(buffer(zstd_bound(size(input, kind=8))))
csize = zstd_compress(c_loc(buffer), size(buffer, kind=8), &
c_loc(input), size(input, kind=8), &
compression_level)
if (csize > 0 .and. csize <= size(output, kind=8)) then
output(1:csize) = buffer(1:csize)
else
csize = -1
end if
deallocate(buffer)
clen = zstd_compress(buffer, clen, input, ilen, compression_level)
if (zstd_iserror(clen) /= 0) clen = 0
#endif /* ZSTD */
#ifdef LZ4
case(compression_lz4)
prefs(5:6) = transfer(size(input, kind=8), [ 0_4 ])
prefs(5:6) = transfer(ilen, [ 0_4 ])
prefs(9) = compression_level
allocate(buffer(lz4_bound(size(input, kind=8), c_loc(prefs))))
csize = lz4_compress(c_loc(buffer), size(buffer, kind=8), &
c_loc(input), size(input, kind=8), c_loc(prefs))
if (csize > 0 .and. csize <= size(output, kind=8)) then
output(1:csize) = buffer(1:csize)
else
csize = -1
end if
deallocate(buffer)
clen = lz4_compress(buffer, clen, input, ilen, c_loc(prefs))
if (lz4_iserror(clen) /= 0) clen = 0
#endif /* LZ4 */
#ifdef LZMA
case(compression_lzma)
csize = 0
allocate(buffer(size(input)))
ret = lzma_compress(compression_level, 4, c_null_ptr, &
c_loc(input), size(input, kind=8), &
c_loc(buffer), c_loc(csize), size(buffer, kind=8))
if (ret == 0 .and. csize <= size(output, kind=8)) then
output(1:csize) = buffer(1:csize)
lsize = 0
ret = lzma_compress(compression_level, 4, c_null_ptr, &
input, ilen, buffer, c_loc(lsize), clen)
if (ret > 0) then
clen = 0
else
csize = -1
clen = lsize
end if
deallocate(buffer)
#endif /* LZMA */
case default
output(1:csize) = input(1:csize)
clen = 0
end select
!-------------------------------------------------------------------------------

2
sources/forcing.F90
View File

@ -113,7 +113,7 @@ module forcing
! array for driving mode coefficients
!
complex(kind=8), dimension(:,:), allocatable :: fcoefs
complex(kind=8), dimension(:,:), allocatable, target :: fcoefs
! velocity Fourier coefficients in Alfvelius method
!

287
sources/hash.F90
View File

@ -24,9 +24,10 @@
!!
!! module: HASH
!!
!! This module provides 64-bit version of the xxHash64 by Yann Collet.
!! This is a Fortran implementation based on the XXH64 specification
!! published at
!! This module is an interface to the XXH functions by Yann Collet provided
!! by the library libxxhash. If this library is not available, an internal
!! Fortran implementation of the 64-bit version of the xxHash64 is used.
!! The Fortran implementation is based on the XXH64 specification published at
!! https://github.com/Cyan4973/xxHash/blob/dev/doc/xxhash_spec.md
!!
!! For additional info, see
@ -39,58 +40,56 @@ module hash
implicit none
#ifdef XXHASH
! interfaces to functions XXH64() and XXH3_64bits() provided
! by the systems library libxxhash
! interfaces to functions XXH64() and XXH3_64bits() provided by
! the library libxxhash
!
interface
integer(c_int64_t) function xxh64_system(input, length, seed) &
bind(C, name="XXH64")
integer(c_int64_t) function xxh64_lib(input, length, seed) &
bind(C, name="XXH64")
use iso_c_binding, only: c_ptr, c_size_t, c_int64_t
implicit none
type(c_ptr) , value :: input
integer(kind=c_size_t), value :: length
integer(c_int64_t) , value :: seed
end function xxh64_system
end function xxh64_lib
integer(c_int64_t) function xxh3_system(input, length) &
bind(C, name="XXH3_64bits")
integer(c_int64_t) function xxh3_lib(input, length) &
bind(C, name="XXH3_64bits")
use iso_c_binding, only: c_ptr, c_size_t, c_int64_t
implicit none
type(c_ptr) , value :: input
integer(kind=c_size_t), value :: length
end function xxh3_system
end function xxh3_lib
end interface
#else /* XXHASH */
! hash parameters
!
integer(kind=8), parameter :: prime1 = -7046029288634856825_8, &
prime2 = -4417276706812531889_8, &
prime3 = 1609587929392839161_8, &
prime4 = -8796714831421723037_8, &
prime5 = 2870177450012600261_8, &
integer(kind=8), parameter :: prime1 = -7046029288634856825_8, &
prime2 = -4417276706812531889_8, &
prime3 = 1609587929392839161_8, &
prime4 = -8796714831421723037_8, &
prime5 = 2870177450012600261_8, &
prime6 = 6983438078262162902_8
#endif /* XXHASH */
! supported hash types
!
integer, parameter :: hash_xxh64 = 1
enum, bind(c)
enumerator hash_none
enumerator hash_xxh64
#ifdef XXHASH
integer, parameter :: hash_xxh3 = 2
enumerator hash_xxh3
#endif /* XXHASH */
end enum
private
public :: get_hash, check_hash, hash_string, hash_xxh64
#ifdef XXHASH
public :: hash_xxh3
#endif /* XXHASH */
public :: hash_info, hash_name, digest, check_digest
public :: digest_string, digest_integer
!- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!
contains
!
!
!===============================================================================
!!
!!*** PUBLIC SUBROUTINES *****************************************************
@ -99,122 +98,242 @@ module hash
!
!===============================================================================
!
! function HASH_STRING:
! subroutine HASH_INFO:
! --------------------
!
! Function returns the hash type string.
! Subroutine returns the hash ID and the length in bytes (characters)
! by the provided hash name.
!
!===============================================================================
!
character(len=8) function hash_string(hash_type)
subroutine hash_info(hash_name, hash_id, hash_length)
use helpers, only : print_message
implicit none
character(len=*), intent(in) :: hash_name
integer , intent(out) :: hash_id, hash_length
character(len=*), parameter :: loc = "HASH::hash_info()"
!-------------------------------------------------------------------------------
!
select case(trim(hash_name))
case("xxh64", "XXH64")
hash_id = hash_xxh64
hash_length = 16
#ifdef XXHASH
case("xxh3", "XXH3")
hash_id = hash_xxh3
hash_length = 16
#endif /* XXHASH */
case("none")
hash_id = hash_none
hash_length = 0
case default
call print_message(loc, &
"Hash '" // trim(hash_name) // "' is not supported!")
end select
return
!-------------------------------------------------------------------------------
!
end subroutine hash_info
!
!===============================================================================
!
! function HASH_NAME:
! ------------------
!
! Function returns the hash name by the provided hash ID.
!
!===============================================================================
!
character(len=8) function hash_name(hash_type)
implicit none
integer, intent(in) :: hash_type
integer(kind(hash_none)) :: htype
!-------------------------------------------------------------------------------
!
htype = hash_type
select case(htype)
case(hash_xxh64)
hash_name = "xxh64"
#ifdef XXHASH
if (hash_type == hash_xxh3) then
hash_string = 'xxh3'
return
end if
case(hash_xxh3)
hash_name = "xxh3"
#endif /* XXHASH */
hash_string = 'xxh64'
case default
hash_name = "none"
end select
return
!-------------------------------------------------------------------------------
!
end function hash_string
end function hash_name
!
!===============================================================================
!
! function GET_HASH:
! -----------------
! function DIGEST:
! ---------------
!
! Function calculates the hash for a given sequence of bytes.
! Function calculates the digest for a given sequence of bytes.
!
! Arguments:
!
! input - the input sequence of bytes;
! hash_type - the number corresponding to the hash type;
! buffer - the buffer pointer;
! length - the buffer length;
! hash_id - the hash ID;
!
!===============================================================================
!
integer(kind=8) function get_hash(input, hash_type) result(hash)
integer(kind=8) function digest(buffer, length, hash_id) result(hash)
#ifdef XXHASH
use iso_c_binding, only: c_loc
#endif /* XXHASH */
use iso_c_binding, only : c_ptr
implicit none
integer(kind=1), dimension(:), target, intent(in) :: input
integer , intent(in) :: hash_type
#ifdef XXHASH
integer(kind=8) :: length
#endif /* XXHASH */
type(c_ptr) , intent(in) :: buffer
integer(kind=8), intent(in) :: length
integer , intent(in) :: hash_id
!-------------------------------------------------------------------------------
!
#ifdef XXHASH
hash = 0
length = size(input, kind=8)
select case(hash_type)
case(hash_xxh3)
hash = xxh3_system(c_loc(input), length)
case default
hash = xxh64_system(c_loc(input), length, hash)
end select
select case(hash_id)
case(hash_xxh64)
#ifndef XXHASH
hash = xxh64(buffer, length)
#else /* XXHASH */
hash = xxh64(input)
hash = xxh64_lib(buffer, length, 0_8)
case(hash_xxh3)
hash = xxh3_lib(buffer, length)
#endif /* XXHASH */
case(hash_none)
hash = 0
end select
return
!-------------------------------------------------------------------------------
!
end function get_hash
end function digest
!
!===============================================================================
!
! subroutine CHECK_HASH:
! ---------------------
! subroutine CHECK_DIGEST:
! -----------------------
!
! Subroutine checks if the provided digest matches the digest of
! the input data.
!
! Arguments:
!
! loc - the location of check;
! fname - the file name;
! input - the input sequence of bytes;
! digest - the data digest to check;
! hash_type - the number corresponding to the hash type;
! loc - the location of check;
! fname - the file name;
! buffer - the buffer pointer;
! length - the buffer length;
! bdigest - the buffer digest to check;
! hash_id - the hash ID;
!
!===============================================================================
!
subroutine check_hash(loc, fname, input, digest, hash_type)
subroutine check_digest(loc, fname, buffer, length, bdigest, hash_id)
use helpers , only : print_message
use iso_c_binding, only : c_ptr
implicit none
character(len=*), intent(in) :: loc, fname
type(c_ptr) , intent(in) :: buffer
integer(kind=8) , intent(in) :: length
integer(kind=8) , intent(in) :: bdigest
integer , intent(in) :: hash_id
!-------------------------------------------------------------------------------
!
if (hash_id == hash_none) return
if (bdigest /= digest(buffer, length, hash_id)) &
call print_message(loc, trim(fname) // " seems to be corrupted!")
!-------------------------------------------------------------------------------
!
end subroutine check_digest
!
!===============================================================================
!
! subroutine DIGEST_STRING:
! ------------------------
!
! Subroutine converts the integer digest to string.
!
! Arguments:
!
! idigest - the digest as integer;
! sdigest - the digest as string;
!
!===============================================================================
!
subroutine digest_string(idigest, sdigest)
use helpers, only : print_message
implicit none
character(len=*) , intent(in) :: loc, fname
integer(kind=1), dimension(:), intent(in) :: input
integer(kind=8) , intent(in) :: digest
integer , intent(in) :: hash_type
integer(kind=8) , intent(in) :: idigest
character(len=*), intent(inout) :: sdigest
character(len=*), parameter :: loc = "HASH::digest_string()"
!-------------------------------------------------------------------------------
!
if (digest /= get_hash(input, hash_type)) &
call print_message(loc, trim(fname) // " seems to be corrupted!")
if (len(sdigest) >= 16) then
write(sdigest,"(1z16.16)") idigest
else
call print_message(loc, &
"The string is too short to contain the whole digest!")
end if
!-------------------------------------------------------------------------------
!
end subroutine check_hash
end subroutine digest_string
!
!===============================================================================
!
! subroutine DIGEST_INTEGER:
! -------------------------
!
! Subroutine converts the string digest to its integer representation.
!
! Arguments:
!
! sdigest - the digest as string;
! idigest - the digest as integer;
!
!===============================================================================
!
subroutine digest_integer(sdigest, idigest)
implicit none
character(len=*), intent(in) :: sdigest
integer(kind=8) , intent(out) :: idigest
!-------------------------------------------------------------------------------
!
read(sdigest, fmt="(1z16)") idigest
!-------------------------------------------------------------------------------
!
end subroutine digest_integer
!
!===============================================================================
!!
@ -232,24 +351,30 @@ module hash
!
! Arguments:
!
! input - the input sequence of bytes;
! buffer - the buffer pointer;
! length - the buffer length;
!
!===============================================================================
!
integer(kind=8) function xxh64(input) result(hash)
integer(kind=8) function xxh64(buffer, length) result(hash)
use iso_c_binding, only : c_ptr, c_f_pointer
implicit none
integer(kind=1), dimension(:), target, intent(in) :: input
type(c_ptr) , intent(in) :: buffer
integer(kind=8), intent(in) :: length
integer(kind=8) :: length
integer(kind=8) :: remaining, offset
integer(kind=8), dimension(4) :: lane, chunk
integer(kind=1), dimension(:), pointer :: input
!-------------------------------------------------------------------------------
!
length = size(input, kind=8)
call c_f_pointer(buffer, input, [ length ])
hash = 0_8
offset = 1_8
remaining = length

1344
sources/io.F90
View File

File diff suppressed because it is too large Load Diff