Data Product Development Guide for Data Producers

Summary

This Data Product Development Guide (DPDG) for Data Producers was produced for the Earth Observing System Data and Information System (EOSDIS) by the Earth Science Data System Working Group (ESDSWG) responsible for developing the DPDG, to aid in the development of NASA Earth Science data products.

The DPDG was created and subsequently revised to provide guidance for those who develop Earth Science data products, collectively referred to as “data producers.” This guide is primarily intended for developers of Earth Science data products derived from remote sensing data, and particularly for the development of Level 1B through Level 4 products (see Data Processing Levels). However, developers of related data products including Level 0 and 1A satellite data, airborne and ground-based data products, and modeling data will also find useful guidance.

Many changes have been implemented in Version 2. New material has been added regarding data products from airborne and field campaign investigations, cloud-optimized formats and services, and Earthdata Pub, along with mapping the attributes to the Unified Metadata Model profiles and expanding information to include Findability, Accessibility, Interoperability, and Reusability (FAIR) sub-principles.

Status

The Data Product Development Guide (DPDG) for Data Producers version 2.0 is approved as a best practice for NASA Earth Science Data Systems.

The Quick Start Guide to the EOSDIS Data Product Development Guide for Data Producers was published by the ESO as a companion document in June 2021 and updated in April 2022.

NASA Earth Science Community Recommendations for Use

Strengths

The Data Product Development Guide (DPDG) draws upon existing resources such as standards, conventions, best practices, and data format documents to assemble a process for data producers to design and develop Earth Science data products. From requirements and design; to data formats, metadata, and compression; to tools, Digital Object Identifiers (DOIs), and data publication, this document provides a comprehensive set of guidelines and recommendations for data producers working with designated NASA Distributed Active Archive Centers (DAACs) to generate and deliver data products that will serve end user communities. Emphasis has been given to standards and recommended practices formally endorsed by ESCO, findings from ESDSWGs, and recommendations from DAACs and experienced data producers in the creation of this document. DPDG Version 2 also provides guidance for making data products Findable, Accessible, Interoperable, and Reusable (FAIR) and for adhering to NASA’s open science objectives and information policies.

This document highlights the need for close communications between data producers and their assigned DAACs. Examples of areas requiring such communications include product design, user requirements, data formats, product naming, metadata requirements, testing, user feedback, cloud optimization, keyword tagging to facilitate data discovery, versioning, DOIs, and product delivery and publication.

Weaknesses

This guide provides a broad overview, bringing together recommended practices, standards, and conventions in use in the Earth Science community. As these protocols evolve, it will be a challenge to stay abreast of these conventions; evaluating, documenting, and recommending them for use by the Earth Science community; and updating this DPDG as warranted. As Earth Science data products move into the cloud environment and open science gains momentum, significant changes to current practices, standards, and conventions are likely. The process of updating the DPDG has already proven to be time-consuming, and it is critical that a process for maintaining and updating it efficiently be established.

Applicability

This guide is primarily intended for developers of Earth Science data products derived from remote sensing data, and particularly for the development of Level 1B through Level 4 products. However, developers of related data products including Level 0 and 1A satellite, airborne, and ground-based data products will also find the guidance useful.

Much of the document is based on the experience data producers have gained via use of the netCDF file format and the Climate and Forecast (CF) Metadata Conventions, which are prevalent among remote sensing data products. There are, however, strong lessons to be learned from this that apply to the entire data product development process. Many aspects of the data development process, including requirements, design, selecting appropriate data format(s), metadata creation, data compression, data quality assessments, and DOI assignments, are recommended for developing data products derived from non-remote sensing sources as well as remote sensing sources. 

Limitations

This guide provides a thorough process for designing and developing Earth Science data products. Organizational commitment, staff resources, and computing power are essential to carrying out the guidelines presented here. Some data producers may not have the resources required to adhere to these guidelines on their own. It is, therefore, advantageous for data producers to establish a cohesive relationship with the NASA DAACs and to leverage some of the resources that the DAACs have available to support the data product development process.