The final state of the files we used in this demo is available on Github here. The recording of the Zoom session is here.

Introduction

In this tutorial, we introduce one of the most used HPC I/O programming interfaces. While it is feasible to exchange data through ASCII or encoded binary, they are often application dependent and it is difficult to take into account cross-platform compatibility. One example is the weather forecast workflow, where sensor data comes indirectly from the sensor. They get prepared into formats that are suitable for application ingestion and get preprocessed. After that, data runs through the workflow and during that, generates different products for post-processing and analysis. To ensure that the data can be exchanged seamlessly and in a high-performance manner, I/O libraries such as HDF5 (and a similar library netCDF, which is commonly used in the meteorology community) are used. Not only is the format standardized which can be distributed across organizations, but the format is also self-explanatory, meaning that an application, without any prior knowledge of the file, will be able to decode its content. Furthermore, it handles compatibility issues such as the exchange between big and little-endian (for example, if the data is generated an on AIX machine, then post-processed in an X86 machine). The most important aspect is that it offers high-performance parallel I/O when used together with MPI and a parallel file system (PFS). However, this is out of the scope of this course and will be covered more in our future courses.

Installing the HDF5 Library

Install the HDF5 development libraries and headers on Ubuntu.

$ sudo apt install libhdf5-dev hdfview

NASA's Panoply viewer can be downloaded here.

The HDF5 Format

The Hierarchical Data Format version 5 (HDF5), is an open-source file format that supports large, complex, heterogeneous data.

HDF5 uses a "file directory" like structure that allows you to organize data within the file in many different structured ways, as you might do with files on your computer.

Hierarchical Structure - A file directory within a file

The HDF5 format can be thought of as a file system contained and described within one single file. Think about the files and folders stored on your computer. You might have a data directory with some temperature data for multiple field sites. These temperature data are collected every minute and summarized on an hourly, daily, and weekly basis. Within one HDF5 file, you can store a similar set of data organized in the same way that you might organize files and folders on your computer. However in an HDF5 file, what we call "directories" or "folders" on our computers, are called groups and what we call files on our computer are called datasets.

Skärmavbild 2021-02-24 kl. 09.28.33.png

A dataset is stored in a file in two parts: a header and a data array.

The header contains information that is needed to interpret the array portion of the dataset, as well as metadata (or pointers to metadata) that describes or annotates the dataset. The header information includes the name of the object, its dimensionality, its number-type, information about how the data itself is stored on disk, and other information used by the library to speed up access to the dataset or maintain the file's integrity.

There are four essential classes of information in any header: name, datatype, dataspace, and the storage layout.

A dataset name is a sequence of alphanumeric ASCII characters.

HDF5 allows one to define many different kinds of data types.

There are two categories of data types: atomic datatypes and compound datatypes. Atomic data types can also be system-specific, or NATIVE, and all data types can be named:

Atomic datatypes are those that are not decomposed at the datatype interface level, such as integers and floats.
NATIVE datatypes are system-specific instances of atomic datatypes.
Compound datatypes are made up of atomic datatypes.
Named datatypes are either atomic or compound data types that have been specifically designated to be shared across datasets.

A dataset dataspace describes the dimensionality of the dataset. The dimensions of a dataset can be fixed (unchanging), or they may be unlimited, which means that they are extendible (i.e. they can grow larger).

Properties of a dataspace consist of the rank (number of dimensions) of the data array, the actual sizes of the dimensions of the array, and the maximum sizes of the dimensions of the array.

The HDF5 format makes it possible to store data in a variety of ways. The default storage layout format is contiguous, meaning that data is stored in the same linear way that it is organized in memory. Two other storage layout formats are currently defined for HDF5: compact and chunked. Compact storage is used when the amount of data is small and can be stored directly in the object header. Chunked storage involves dividing the dataset into equal-sized "chunks" that are stored separately.

HDF5 is a Self Describing Format

HDF5 format is self-describing. This means that each file, group, and dataset can have associated metadata that describes exactly what the data are. Following the example above, we can embed information about each site to the file, such as:

The full name and X, Y location of the site
Description of the site.
Any documentation of interest.

Similarly, we might add information about how the data in the dataset were collected, such as descriptions of the sensor used to collect the temperature data. We can also attach information, to each dataset within the site group, about how the averaging was performed and over what time period data are available.

One key benefit of having metadata that is attached to each file, group, and dataset, is that this facilitates automation without the need for a separate (and additional) metadata document. Using a programming language, like C, Fortran or Python we can grab information from the metadata that is already associated with the dataset, and which we might need to process the dataset.

Compressed & Efficient subsetting

The HDF5 format is a compressed format. The size of all data contained within HDF5 is optimized which makes the overall file size smaller. Even when compressed, however, HDF5 files often contain big data and can thus still be quite large.

A powerful attribute of HDF5 is data slicing, by which particular subsets of a dataset can be extracted for processing. This means that the entire dataset doesn't have to be read into memory (RAM); very helpful in allowing us to more efficiently work with very large (gigabytes or more) datasets!

Heterogeneous Data Storage

HDF5 files can store many different types of data within the same file. For example, one group may contain a set of datasets to contain integer (numeric) and text (string) data. Or, one dataset can contain heterogeneous data types (e.g., both text and numeric data in one dataset). This means that HDF5 can store any of the following (and more) in one file:

Temperature and precipitation.
A set of images that cover one or more areas (each image can have specific spatial information associated with it - all in the same file)
A set of images that cover one or more areas (each image can have unique spatial information associated with it)
...

Open Format

The HDF5 format is open and free to use. The supporting libraries (and a free viewer), can be downloaded from the HDF Group website.

The HDF5 API

There are several HDF5 APIs. In this tutorial, we focus on the HDF5 API implemented in C.

The API provides routines for creating HDF5 files, creating and writing groups, datasets, and their attributes to HDF5 files, and reading groups, datasets, and their attributes from HDF5 files.

All C routines in the HDF 5 library begin with a prefix of the form H5*, where * is a single letter indicating the object on which the operation is to be performed:

H5F: File-level access routines.
Example: H5Fopen, which opens an HDF5 file.
H5G: Group functions, for creating and operating on groups of objects.
Example: H5Gset, which sets the working group to the specified group.
H5T: DataType functions, for creating and operating on simple and compound datatypes to be used as the elements in data arrays.
Example: H5Tcopy, which creates a copy of an existing datatype.
H5S: DataSpace functions, which create and manipulate the dataspace in which the elements of a data array are stored.
Example: H5Screate_simple, which creates simple dataspaces.
H5D: Dataset functions, which manipulate the data within datasets and determine how the data is to be stored in the file.
Example: H5Dread, which reads all or part of a dataset into a buffer in memory.
H5P: Property list functions, for manipulating object creation and access properties.
Example: H5Pset_chunk, which sets the number of dimensions and the size of a chunk.
H5A: Attribute access and manipulating routines.
Example: H5Aget_name, which retrieves the name of an attribute.
H5Z: Compression registration routine.
Example: H5Zregister, which registers new compression and uncompression functions for use with the HDF5 library.
H5E: Error handling routines.
Example: H5Eprint, which prints the current error stack.
H5R: Reference routines.
Example: H5Rcreate, which creates a reference.
H5I: Identifier routine.
Example: H5Iget_type, which retrieves the type of an object.

Header files

There are a number of definitions and declarations that should be included with any HDF5 program. These definitions and declarations are contained in several include files. The main include file is hdf5.h. This file includes all of the other files that your program is likely to need. Be sure to include hdf5.h in any program that uses the HDF5 library.

Create an HDF5 File

The following code fragment implements the specified model. If there is a possibility that the file already exists, the user must add the flag H5ACC_TRUNC to the accessing mode to overwrite the previous file's information.

hid_t file;                          /* identifier */
/*
* Create a new file using H5ACC_TRUNC access,
* default file creation properties, and default file
* access properties.
* Then close the file.
*/
file = H5Fcreate(FILE, H5ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
status = H5Fclose(file);

Create and Initialize a dataset for writing to a file

Datatypes and dimensionality (dataspace) are independent objects, which are created separately from any dataset that they might be attached to. Because of this, the creation of a dataset requires, at a minimum, separate definitions of the datatype, dimensionality, and dataset. Hence, to create a dataset the following steps need to be taken:

Create and initialize a dataspace for the dataset to be written.
Define the data type for the dataset to be written.
Create and initialize the dataset itself.

The following code illustrates the creation of these three components of a dataset object (rank is the number of dimensions used in the dataspace.).

hid_t dataset, datatype, dataspace;   /* declare identifiers */

/* 
 * Create dataspace: Describe the size of the array and 
 * create the data space for fixed size dataset. 
 */
dimsf[0] = NX;
dimsf[1] = NY;
dataspace = H5Screate_simple(RANK, dimsf, NULL); 
/*
 * Define datatype for the data in the file.
 * We will store little endian integer numbers.
 */
datatype = H5Tcopy(H5T_NATIVE_INT);
status = H5Tset_order(datatype, H5T_ORDER_LE);
/*
 * Create a new dataset within the file using defined 
 * dataspace and datatype and default dataset creation
 * properties.
 * NOTE: H5T_NATIVE_INT can be used as datatype if conversion
 * to little endian is not needed.
 */
dataset = H5Dcreate(file, DATASETNAME, datatype, dataspace, H5P_DEFAULT);

Deallocate HDF5 Object

The datatype, dataspace, and dataset objects should be released once they are no longer needed by a program. Since each is an independent object, they must be released (or closed) separately. The following lines of code close the datatype, dataspace, and datasets that were created in the preceding section.

H5Tclose(datatype);
H5Dclose(dataset);
H5Sclose(dataspace);

Write a Dataset to a file

Having defined the datatype, dataset, and dataspace parameters, you write out the data with a call to H5Dwrite.

/*
* Write the data to the dataset using default transfer
* properties.
*/
status = H5Dwrite(dataset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL,
	  H5P_DEFAULT, data);

The third and fourth parameters of H5Dwrite in the example describe the dataspaces in memory and in the file, respectively. They are set to the value H5S_ALL to indicate that an entire dataset is to be written.

Reading is analogous to writing. If in the previous example, we wish to read an entire dataset, we would use the same basic calls with the same parameters. Of course, the routine H5Dread would replace H5Dwrite.

Getting information about a dataset

Although reading is analogous to writing, it is often necessary to query a file to obtain information about a dataset. For instance, we often need to know about the data type associated with a dataset, as well as dataspace information (e.g. rank and dimensions). There are several "get" routines for obtaining this information. The following code segment illustrates how we would get this kind of information:

/*
* Get datatype and dataspace identifiers and then query
* dataset class, order, size, rank and dimensions.
*/

datatype  = H5Dget_type(dataset);     /* datatype identifier */ 
class     = H5Tget_class(datatype);
if (class == H5T_INTEGER) printf("Data set has INTEGER type \n");
order     = H5Tget_order(datatype);
if (order == H5T_ORDER_LE) printf("Little endian order \n");

size  = H5Tget_size(datatype);
printf(" Data size is %d \n", size);

dataspace = H5Dget_space(dataset);    /* dataspace identifier */
rank      = H5Sget_simple_extent_ndims(dataspace);
status_n  = H5Sget_simple_extent_dims(dataspace, dims_out);
printf("rank %d, dimensions %d x %d \n", rank, dims_out[0], dims_out[1]);

HDF Tools

The HDF5 library also provides a series of tools for examining HDF5 format data, including the tools h5ls and h5dump. h5ls is analogous to the Unix ls command and enables the user to query the HDF5 namespace in the same way ls queries the Unix file system directory.

The h5dump utility will dump to screen the data stored in the hdf5 file.

Example - Writing Particle Data to an HDF5 File

This is an example of how to write particle data to a file:

#include <hdf5.h>
#include <math.h>

// particle data structure
typedef struct particle3D {
    double x, y, z; // coordinates 
} particle_t;

#define PARTICLE_COUNT 15

int main(int argc, char **argv){

     // define particles
     particle_t particles[PARTICLE_COUNT];

     for (int i=0; i < PARTICLE_COUNT; i++) {
         double t = i*0.2;
         particles[i].x = cos(t);
         particles[i].y = sin(t);
         particles[i].z = t;
     }

     // create HDF5 type layout in memory
     hid_t mtype = H5Tcreate(H5T_COMPOUND, sizeof(particle_t));
     H5Tinsert(mtype, "x coordinate", HOFFSET(particle_t, x), H5T_NATIVE_DOUBLE);
     H5Tinsert(mtype, "y coordinate", HOFFSET(particle_t, y), H5T_NATIVE_DOUBLE);
     H5Tinsert(mtype, "z coordinate", HOFFSET(particle_t, z), H5T_NATIVE_DOUBLE);

     // create data space
     hsize_t dim = PARTICLE_COUNT;
     hid_t space = H5Screate_simple(1, &dim, NULL);

     // create new file with default properties
     hid_t fd = H5Fcreate("particles.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
     // create data set
     int dset = H5Dcreate(fd, "particle data", mtype, space, H5P_DEFAULT,
     H5P_DEFAULT, H5P_DEFAULT);
     // write the entire dataset and close the file
     H5Dwrite(dset, mtype, H5S_ALL, H5S_ALL, H5P_DEFAULT, particles);
     H5Fclose(fd);
}

Compile by providing the path to the header files and the library linking flags.

$ gcc write_particles.c -I/usr/include/hdf5/serial -L/usr/lib/x86_64-linux-gnu/hdf5/serial -lhdf5 -lm

We can show this dataset using utilities such as HDFView.

A small example application

In this example, we will demonstrate how to construct a distributable MNIST dataset in HDF5 from the binary format. We will create a writer to create the datasets together with the labels and a reader that reads the dataset.

Decoding MNIST and writing to HDF5

We use the following code to decode the original MNIST dataset. After that, we follow the sequence:

Define the dimension of the dataset we want to write
Define the churning scheme
Open HDF5 file
Set chunking
Create space
Create a dataset in the space and chunking
Write data to the HDF5 dataset
Clean up

The code will look like the following.

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <string.h>
#include <hdf5.h>

// source: https://github.com/takafumihoriuchi/MNIST_for_C/blob/master/mnist.h

void read_mnist_image(const char *file_path, int data_count, int len_info, int arr_n, double *data, int *info_arr)
{
    int fd;
    unsigned char data_char[arr_n];

    if ((fd = open(file_path, O_RDONLY)) == -1) {
        fprintf(stderr, "couldn't open image file\n");
        exit(-1);
    }

    // read header
    read(fd, info_arr, len_info * sizeof(int));

    // read data
    for (int i = 0; i < data_count; i++) {
        read(fd, data_char, arr_n * sizeof(unsigned char));   
        for (int j = 0; j < arr_n; j++) {
            data[i * arr_n + j] = (double)data_char[j] / 255.0;
        }
    }

    close(fd);
}

void read_mnist_label(const char *file_path, int data_count, int len_info, int arr_n, int *data, int *info_arr)
{
    int fd;
    unsigned char data_char;

    if ((fd = open(file_path, O_RDONLY)) == -1) {
        fprintf(stderr, "couldn't open label file\n");
        exit(-1);
    }

    // read header
    read(fd, info_arr, len_info * sizeof(int));

    // read data
    for (int i = 0; i < data_count; i++) {
        read(fd, &data_char, sizeof(unsigned char));
        data[i] = (int)data_char;
    }

    close(fd);
}


int main(int argc, char *argv[])
{
    herr_t  status;
    int info_image[4];
    int info_label[2];

    // prepare to read data
    double *mnist_image = (double*)malloc(60000 * 784 * sizeof(double));
    int    *mnist_label = (int*)malloc(60000 * sizeof(int));

    // read data from MNIST bin files
    read_mnist_image("./train-images-idx3-ubyte", 60000, 4, 784, mnist_image, info_image);
    read_mnist_label("./train-labels-idx1-ubyte", 60000, 2,   1, mnist_label, info_label);

    // define dataset size: 6k samples, 28x28 each
    hsize_t image_data_dim[3]    = {60000, 28, 28};

    // define chunking, storing 1000 samples per chunk
    hsize_t image_chunk_dims[3]  = {1000, 28, 28};

    // create a new HDF5 file
    hid_t file = H5Fcreate("./mnist.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);

    // set chunking scheme
    hid_t dcpl = H5Pcreate(H5P_DATASET_CREATE);
    status     = H5Pset_chunk(dcpl, 3, image_chunk_dims);

    // create space for dataset in rank 3 with 600000x28x28
    hid_t space = H5Screate_simple(3, image_data_dim, NULL);

    // create a dataset called "Image" in the space created, using float64 in little-endian
    hid_t dset  = H5Dcreate(file, "Image", H5T_IEEE_F64LE, space, H5P_DEFAULT, dcpl, H5P_DEFAULT);

    // write data in double format
    status = H5Dwrite(dset, H5T_NATIVE_DOUBLE, H5S_ALL, H5S_ALL, H5P_DEFAULT, mnist_image);

    // close this dataset
    status = H5Dclose(dset);
    status = H5Sclose(space);
    printf("Written Image dataset...\n");

    // create a new dataset for labels
    hsize_t label_data_dim[1]   = {60000};
    hsize_t label_chunk_dims[1] = {1000};

    // follow same chunking as image data
    status = H5Pset_chunk(dcpl, 1, label_chunk_dims);

    // create space for dataset in rank 1 with 60000
    space  = H5Screate_simple(1, label_data_dim, NULL);

    // create a dataset called "Label" in the space created, using int32 in little-endian
    dset   = H5Dcreate(file, "Label", H5T_STD_I32LE, space, H5P_DEFAULT, dcpl, H5P_DEFAULT);

    // write data in integer format
    status = H5Dwrite(dset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, mnist_label);

    // cleanup
    status = H5Dclose(dset);
    status = H5Sclose(space);
    status = H5Pclose(dcpl);
    status = H5Fclose (file);
    printf("Written Label dataset...\n");

    free(mnist_label);
    free(mnist_image);
}

Compile the code.

$ gcc write_mnist.c -o write_mnist.out -I/usr/include/hdf5/serial -L/usr/lib/x86_64-linux-gnu/hdf5/serial -lhdf5 -lm -g

Create a reader

The reader will follow a similar approach as when writing to a dataset.

#include <stdio.h>
#include <stdlib.h>
#include <hdf5.h>

int main(int argc, char *argv[])
{
    herr_t status;

    // open HDF5 file
    hid_t file  = H5Fopen("./mnist.h5", H5F_ACC_RDONLY, H5P_DEFAULT);

    // open image dataset
    hid_t dset  = H5Dopen(file, "Image", H5P_DEFAULT);

    // get handle to space for dataset
    hid_t space = H5Dget_space(dset);

    // get rank of dataset and prepare to get the dim
    int rank  = H5Sget_simple_extent_ndims(space);

    // get the dim of the dataset
    hsize_t image_dims[rank];
    status = H5Sget_simple_extent_dims(space, image_dims, NULL);

    // compute total memory required to read dataset and allocate memory
    int total_size = 1;
    for (int i = 0; i < rank; i++) total_size *= image_dims[i];
    double *mnist_image = (double*)malloc(total_size* sizeof (double));

    // read dataset in double format
    status = H5Dread(dset, H5T_NATIVE_DOUBLE, H5S_ALL, H5S_ALL, H5P_DEFAULT, mnist_image);

    // close this dataset
    H5Sclose(space);
    H5Dclose(dset);

    // open the label dataset
    dset  = H5Dopen(file, "Label", H5P_DEFAULT);

    // get handle to space that holds the dataset
    space = H5Dget_space(dset);

    // get the rank of the dataset
    rank  = H5Sget_simple_extent_ndims(space);

    // compute total memory required to read dataset and allocate memory
    hsize_t label_dims[rank];
    status = H5Sget_simple_extent_dims(space, label_dims, NULL);
    total_size = 1;
    for (int i = 0; i < rank; i++) total_size *= label_dims[i];
    int *mnist_label = (int*)malloc(total_size* sizeof (int));

    // read dataset in integer
    status = H5Dread(dset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, mnist_label);

    // close dataset and file
    H5Sclose(space);
    H5Dclose(dset);
    H5Fclose(file);

    // printout
    for (long int i = 0; i < image_dims[0]; i++) {
        printf("Label: %d\n\n", mnist_label[i]);
        for (long int j = 0; j < image_dims[1]; j++) {
            for (long int k = 0; k < image_dims[2]; k++) {
                double pixel = mnist_image[(i * (image_dims[1] * image_dims[2])) + image_dims[1] * j + k];
                if (pixel > 0.5)
                    printf("* ");
                else
                    printf(". ");
            }
            printf("\n");
        }
        printf("\n");
    }
}

Compile the code with the following.

$ gcc read_mnist.c -o read_mnist.out -I/usr/include/hdf5/serial -L/usr/lib/x86_64-linux-gnu/hdf5/serial -lhdf5 -lm -g -Wall

We can run the two codes and generate an output file called mnist.h5, and print the output on screen.

$ ./write_mnist.out
Written Image dataset...
Written Label dataset...
$ ./read_mnist.out
........
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 

Label: 8

. . . . . . . . . . . . . . . . . . . . . . . . . . . . 
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 
. . . . . . . . . . . . . . . * * * * * . . . . . . . . 
. . . . . . . . . . . . . * * * * * * * * . . . * * . . 
. . . . . . . . . . . . * * . . . . . . * . . * * . . . 
. . . . . . . . . . . * * . . . . . . . * * * * . . . . 
. . . . . . . . . . . * . . . . . . . . * * * . . . . . 
. . . . . . . . . . * * . . . . . . * * * . . . . . . . 
. . . . . . . . . . * . . . . . * * * . . . . . . . . . 
. . . . . . . . . . * * . . . * * * . . . . . . . . . . 
. . . . . . . . . . * * . * * * . . . . . . . . . . . . 
. . . . . . . . . . * * * * . . . . . . . . . . . . . . 
. . . . . . . . . . * * * . . . . . . . . . . . . . . . 
. . . . . . . . . * * * . . . . . . . . . . . . . . . . 
. . . . . . . * * * . * . . . . . . . . . . . . . . . . 
. . . . . . . * * . . * . . . . . . . . . . . . . . . . 
. . . . . . * * * . . * * . . . . . . . . . . . . . . . 
. . . . . . . * * . . * . . . . . . . . . . . . . . . . 
. . . . . . . * * * * * . . . . . . . . . . . . . . . . 
. . . . . . . . * * * . . . . . . . . . . . . . . . . . 
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 
. . . . . . . . . . . . . . . . . . . . . . . . . . . .

Furthermore, we can inspect the code using tools such as h5dump, HDFView, and Panoply.

By using the self-describing information inside HDF5, an independent viewer can, without any knowledge about the users' application and dataset, perform extraction and visualization.

References

https://www.hdfgroup.org/solutions/hdf5/
MIT - Introduction to HDF5 API: http://web.mit.edu/fwtools_v3.1.0/www/H5.intro.html#Intro-APIs
NSF - Introduction to HDF5 format: https://www.neonscience.org/resources/learning-hub/tutorials/about-hdf5
https://github.com/takafumihoriuchi/MNIST_for_C
https://www.giss.nasa.gov/tools/panoply/
https://www.h5py.org

Zoom Session