Fiber::StorageTransformations::HDF5_Tuning Struct Reference

Tuning parameters related to HDF5 files. More...

#include <StorageTransformations.hpp>

Public Attributes
string	elink_prefix
	The external link prefix as used by HDF5:
int	ELINK_CACHESIZE = 128*1024
	Specifies how many files are kept open via external links such as to avoid re-opening files that are accessed via links.
int	FileSpaceStrategyThreshold = 4096
	The threshold value in bytes at which HDF5 shall start to reclaim unused space in an HDF5 file.
int	sieve_buf_size = 0
	The Sieve buffer is some HDF5-internal look-ahead cache.
int	MDC_initial_size = 32*1024
	Metadata Cache.
int	MDC_max_size = 64*1024
int	MDC_min_size = 1024

Detailed Description

Tuning parameters related to HDF5 files.

See also

https://docs.hdfgroup.org/hdf5/

Member Data Documentation

ELINK_CACHESIZE

int Fiber::StorageTransformations::HDF5_Tuning::ELINK_CACHESIZE = 128*1024

Specifies how many files are kept open via external links such as to avoid re-opening files that are accessed via links.

The file open operation can be a severe performance issue under Windows. The default is 100k files, which may be needed during massive data file processing operations.

elink_prefix

string Fiber::StorageTransformations::HDF5_Tuning::elink_prefix

The external link prefix as used by HDF5:

When the external link link_name is accessed, the library will search for the target file file_name as described below:

If file_name is a relative pathname, the following steps are performed: The library will get the prefix(es) set in the environment variable HDF5_EXT_PREFIX and will try to prepend each prefix to file_name to form a new file_name. If the new file_name does not exist or if HDF5_EXT_PREFIX is not set, the library will get the prefix set via H5Pset_elink_prefix() and prepend it to file_name to form a new file_name. If the new file_name does not exist or no prefix is being set by H5Pset_elink_prefix(), then the path of the file associated with link_loc_id is obtained. This path can be the absolute path or the current working directory plus the relative path of that file when it is created/opened. The library will prepend this path to file_name to form a new file_name. If the new file_name does not exist, then the library will look for file_name and will return failure/success accordingly.

If file_name is an absolute pathname, the library will first try to find file_name. If file_name does not exist, file_name is stripped of directory paths to form a new file_name. The search for the new file_name then follows the same steps as described above for a relative pathname. See examples below illustrating how target_file_name is stripped to form a new file_name.

See also

https://docs.hdfgroup.org/hdf5/v1_14/group___h5_l.html#ga15dfaeb9b1c0b3136533cb97ee45e683

FileSpaceStrategyThreshold

int Fiber::StorageTransformations::HDF5_Tuning::FileSpaceStrategyThreshold = 4096

The threshold value in bytes at which HDF5 shall start to reclaim unused space in an HDF5 file.

Small values may be inefficient as tiny space is reclaimed. Zero turns off the features, thus HDF5 files grow larger. For scenerios where data are only added to an HDF5 file, this features is not needed, but any updates and modifications will benefit from smaller file sizes.

sieve_buf_size

int Fiber::StorageTransformations::HDF5_Tuning::sieve_buf_size = 0

The Sieve buffer is some HDF5-internal look-ahead cache.

Using it has led to strange memory usage issues when reading a dataset and closing it, but the memory occupied by it was not released. This could be avoided by disabling the sieve buffer. If disabling it is "required" with a certain HDF5 version or usage scenario, is yet to be investigated.