Fiber::StorageTransformations::FilterSettings Struct Reference

Classes
struct	ZFP
	https://computing.llnl.gov/projects/zfp More...

Public Types
enum class	SzipCoding { None , Entropy , NearestNeighbor }
enum	Compressor {   None = -1 , BLOSCLZ = 0 , LZ4 = 1 , LZ4HC = 2 ,   SNAPPY = 3 , ZLIB = 4 , ZSTD = 5 }
	Compression mode. More...
enum	Shuffle { NoShuffle = -1 , DOSHUFFLE = 1 , MEMCPYED = 0 , DOBITSHUFFLE = 2 }
enum	AutoSettingStrategy { Uncompressed , BestForReading , BestForWriting , SmallestSize }

Public Member Functions
void	setAutomaticStrategy (AutoSettingStrategy StrategyMode=BestForReading)
	FilterSettings (AutoSettingStrategy StrategyMode)
string	human_readable_description () const
	Provide a description of the current filter settings.

Public Attributes
SzipCoding	SzipCodingMethod = SzipCoding::None
	The following guidelines can be used in determining which option to select:
unsigned int	szip_pixels_per_block = 16
	SZIP compresses data block by block, with a user-tunable block size.
Compressor	BLOSC = None
	blosc_compress() honors different environment variables to control internal parameters without the need of doing that programatically.
Shuffle	ShuffleMode = NoShuffle
int	CompressionLevel = 0
	A number between 0 and 9.
bool	ComputeRangeAttributeOnWriting = false
	Allow to control whether on writing a range information shall be computed and stored.
struct Fiber::StorageTransformations::FilterSettings::ZFP	zfp
	ZFP filter parameters.
bool	doEnableWriteSinglePrecision = false
bool	doEnableNumericalShift = false

Member Enumeration Documentation

Compressor

enum Fiber::StorageTransformations::FilterSettings::Compressor

Compression mode.

Enumerator
BLOSCLZ	BLOSC's own LZ compression mode.

Member Data Documentation

BLOSC

Compressor Fiber::StorageTransformations::FilterSettings::BLOSC = None

blosc_compress() honors different environment variables to control internal parameters without the need of doing that programatically.

Here are the ones supported:

BLOSC_CLEVEL=(INTEGER): This will overwrite the clevel parameter before the compression process starts.

BLOSC_SHUFFLE=[NOSHUFFLE | SHUFFLE | BITSHUFFLE]: This will overwrite the doshuffle parameter before the compression process starts.

BLOSC_TYPESIZE=(INTEGER): This will overwrite the typesize parameter before the compression process starts.

BLOSC_COMPRESSOR=[BLOSCLZ | LZ4 | LZ4HC | SNAPPY | ZLIB]: This will call blosc_set_compressor(BLOSC_COMPRESSOR) before the compression process starts.

BLOSC_NTHREADS=(INTEGER): This will call blosc_set_nthreads(BLOSC_NTHREADS) before the compression process starts.

BLOSC_BLOCKSIZE=(INTEGER): This will call blosc_set_blocksize(BLOSC_BLOCKSIZE) before the compression process starts. NOTE: The blocksize is a critical parameter with important restrictions in the allowed values, so use this with care.

BLOSC_NOLOCK=(ANY VALUE): This will call blosc_compress_ctx() under the hood, with the compressor, blocksize and numinternalthreads parameters set to the same as the last calls to blosc_set_compressor(), blosc_set_blocksize() and blosc_set_nthreads(). BLOSC_CLEVEL, BLOSC_SHUFFLE, BLOSC_TYPESIZE environment vars will also be honored.

ComputeRangeAttributeOnWriting

bool Fiber::StorageTransformations::FilterSettings::ComputeRangeAttributeOnWriting = false

Allow to control whether on writing a range information shall be computed and stored.

Since the data are most likely already in RAM when saving, that would be doable here rather easily. For coordinate fields, the range information corresponds to the bounding box information (without numerical shift if stored in single precision). The implementation of getFilterSettings() may decide to create data if they are not in RAM yet; this should not harm in general because the data need to be created anyway for storing.

szip_pixels_per_block

unsigned int Fiber::StorageTransformations::FilterSettings::szip_pixels_per_block = 16

SZIP compresses data block by block, with a user-tunable block size.

This block size is passed in the parameter pixels_per_block and must be even and not greater than 32, with typical values being 8, 10, 16, or 32. This parameter affects compression ratio; the more pixel values vary, the smaller this number should be to achieve better performance.

In HDF5, compression can be applied only to chunked datasets. If pixels_per_block is bigger than the total number of elements in a dataset chunk, H5Pset_szip will succeed but the subsequent call to H5Dcreate will fail; the conflict can be detected only when the property list is used.

To achieve optimal performance for SZIP compression, it is recommended that a chunk's fastest-changing dimension be equal to N times pixels_per_block where N is the maximum number of blocks per scan line allowed by the SZIP library. In the current version of SZIP, N is set to 128 .

SzipCodingMethod

SzipCoding Fiber::StorageTransformations::FilterSettings::SzipCodingMethod = SzipCoding::None

The following guidelines can be used in determining which option to select:

The entropy coding method is best suited for data that has been processed. The EC method works best for small numbers. The nearest neighbor coding method preprocesses the data then the applies EC method as above.

Other factors may affect results, but the above criteria provides a good starting point for optimizing data compression.

See also

https://support.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetSzip

SZIP compression cannot be applied to compound datatypes, array datatypes, variable-length datatypes, enumerations, or any other user-defined datatypes.

zfp

struct Fiber::StorageTransformations::FilterSettings::ZFP Fiber::StorageTransformations::FilterSettings::zfp

ZFP filter parameters.

https://h5z-zfp.readthedocs.io/en/latest/interfaces.html