ESDS Program

netCDF-4/HDF5 File Format

Summary

This document nominates the netCDF-4/Hierarchical Data Format Version 5 (HDF5) File Format for adoption as a NASA Earth Science Data Systems (ESDS) community standard. It specifies the netCDF-4/HDF5 file format independent of the netCDF I/O libraries designed to read and write netCDF-4/HDF5 data. The netCDF-4/HDF5 file format enables the expansion of the netCDF model, libraries, and machine-independent data format for geoscience data. Together the netCDF interfaces, libraries, and formats support the creation, access, and sharing of scientific data.

With suitable community conventions, the netCDF-4/HDF5 data format can help improve the capability to read and share important scientific data among data providers, data users, and data services.

Status

The netCDF-4/HDF5 File Format is an approved standard recommended for use in NASA's Earth Science Data Systems in March 2011.

Specification Document

netCDF-4/HDF5 File Format

User Resources

Appendix B of the specification document provides guidelines that will enable HDF5 users to create files that will be accessible from netCDF-4.

netCDF Project Web Site

Standards Body

Open Geospatial Consortium (OGC)

NASA Earth Science Community Recommendations for Use

Strengths

netCDF-4 is straightforward to use relative to HDF5, with a lower learning curve. The many tools available enhance ease of use. Use of the HDF5 storage layer in netCDF-4 software provides features for improved performance, such as compression, parallel I/O, relaxed size limits, and the performance benefits of chunking and endianness control.

Weaknesses

Installation and set-up could be improved. HDF5 users point out that more manual intervention is required for installing netCDF-4/HDF5 than for HDF5 alone. NetCDF users point out that for netCDF-4, multiple software libraries must be installed (netCDF, HDF5, possibly other supporting libraries), rather than the one software library required for netCDF-3.

Applicability

netCDF-4 handles many data types and structures needed for Earth science. Those already using HDF tools can access netCDF-4 data using the HDF-5 API rather than netCDF. Those who have not been using HDF tools welcome access to much of the power of HDF via the simpler netCDF API. Community reviewers of the RFC cite many terabytes of data in netCDF-4, with thousands of users.

Limitations

One reviewer noted some internal inconsistencies between the netCDF-4 specification and the DAP library implementation. A more significant issue is that support for Windows users lags significantly behind Linux. This appears to be a problem for both the HDF5 and netCDF-4 software libraries.

Last Updated