Requirements for Archiving, Distribution and User Services in Earth Observation System Data and Information System (EOSDIS)
Note: The following document is a marked up version of NASA's Earth Science Data and Information System (ESDIS) Project official baselined document, Requirements for Archiving, Distribution and User Services in Earth Observing System Data and Information System (EOSDIS), 432-10-69, Revision B, March 2017.
The purpose of this document is to provide common requirements for data archiving, data distribution and user services for EOSDIS-supported data. These services are currently being provided by the Distributed Active Archive Centers (DAACs). Other elements of EOSDIS may also be subject to a limited set of these general requirements. The scope of the EOSDIS systems has changed since this requirements document was first written. This document is being updated to reflect these changes.
Keywords: DAAC, ESDIS, EOSDIS, XDS, XDC, LTA, EMS, WIST, EMS
Table of Contents
1.1 Scope and Assumptions
The purpose of this document is to provide common requirements for data archiving, data distribution and user services for EOSDIS-supported data. These services are currently being provided by EOSDIS Data Centers or DAACs. Other elements of EOSDIS may also be subject to a limited set of these general requirements. Rather than refer to specific Archive Centers/DAACs, this document refers to an entity providing such services as an “X Distributed Active Archive Center (XDAAC)”. The data system that provides the functionality to provide the services is referred to as an “X Data System (XDS)”. It is assumed that the reader is familiar with Earth Observing System Data and Information System (EOSDIS) terminology.
The intent of this document is to specify “what” requirements as opposed to any implied implementation or “how”.
This document does not include discussion of network requirements and communications systems. Security requirements are covered by pre-existing security plans tracked by the ESDIS Computer Security Officer. Network requirements are discussed in separate documents.
2 REQUIREMENTS SPECIFICATION FOR ARCHIVE, DISTRIBUTION AND USER SERVICES
The XDAAC has the responsibility to meet the objectives of data product archiving, distribution, and search and ordering for missions assigned to ESDIS. This includes all missions (satellite, aircraft, field campaign, model, in-situ, and others) assigned by NASA Headquarters Earth Science Division to EOSDIS.
Data to be archived are ingested from one or more data production systems and various other mission and external data providers, as necessary, through interfaces described in specific Interface Control Documents (ICDs). Requirements to archive and distribute the data for a given mission extend as long as required by ESDIS. This duration is dependent on the active use of the data by NASA funded investigators and the provisions of long term archiving as determined by ESDIS. XDAACs will need to ensure that the data are transitioned to the appropriate Long-Term Archive when National Aeronautics and Space Administration (NASA) notifies the XDAAC that it is appropriate to do so. The XDS has a search and order/request/access function that 1) provides users with information about the available data products and data-related services (e.g., guide documents), 2) gives users the capability to identify and select their desired information and data products or services before ordering/requesting/accessing, and 3) delivers the requested data to users. The XDS shall provide users with functionalities to manipulate certain data products prior to ordering (e.g., spatial and/or spectral subsetting). The XDAAC shall provide a User Services function to assist users, for example, questions regarding data formats, data usage, system access, etc. While the XDAAC will probably be staffed only on a 40-hour workweek schedule, ESDIS expects the XDS to be available every day on a 24-hour basis with the exceptions of planned downtime.
In addition, the XDAAC provides ESDIS information (typically, log files) related to data ingesting, data archiving, data distribution, and user satisfaction. To facilitate integrated reporting to NASA Headquarters, ESDIS management determines the specific metrics information to be gathered based on the particular needs of the mission and the detailed information needed to monitor performance of the XDAAC. ESDIS collects metrics from the XDSs according to the interface control document (423-47-01) between XDSs and NASA's EOSDIS Metrics System (EMS). Appendix A shows the metrics that the Project routinely uses.
The Project supports and maintains a number of cross-EOSDIS infrastructure components and APIs that together, along with XDS components, define the EOSDIS enterprise. In addition to the EMS, these components include NASA's Common Metadata Repository (CMR), its internal metadata representation called the Unified Metadata Model (UMM), and its Earthdata Search reference client; Global Imagery Browse Services (GIBS) and its Worldview reference client; and Earthdata which includes the Earthdata website, Earthdata Code Collaborative (ECC), Earthdata Login, Earthdata Top Hat along with a number of other reusable components. As new cross-EOSDIS infrastructure components are developed, the XDAAC is expected to participate in the development by providing unique requirements and to participate in systems reviews. XDAACs may also collaborate with each other and it is incumbent on them to provide all necessary interfaces between collaborating XDS systems at each XDAAC.
It is ESDIS policy to release all software as open source following the NASA release process. All software developed for the XDSs shall likewise be released as open source.
It is an ESDIS general policy that all software deployed to the commercial cloud utilize the NASA-Compliant General Application Platform (NGAP) and comply with associated ESDIS processes for cloud application development and deployment.
2.2 Data Ingest
ESDIS policy is that ingest is completed only when the data are available to users and the metadata are in CMR.
- The XDS shall ingest data from each provider in accordance with the interface specification documented in the applicable ICD.
- The XDS shall be capable of ingesting data from multiple providers.
- The XDS shall ensure ingest data integrity through the use of checksums in order to satisfy interface requirements with external systems.
- The XDS shall verify the non-science quality of the data upon ingest for each unique data set.
- The XDS shall provide ingest metrics to the ESDIS Metrics System as described in Appendix A.
- The XDAAC shall support GIBS by providing browse imagery for all higher-level data products where imagery can be generated. ESDIS will approve the XDAAC products for which browse imagery will be generated and the methods used for generation and delivery to GIBS. ESDIS may grant waivers for some products. Browse products may be delivered to GIBS directly from the Science Investigator-led Processing System (SIPS), or be simply passed through the XDAAC to GIBS.
2.3 Data Archive and Distribution Function
Each XDAAC has the responsibility for archiving and distribution of assigned mission-related data products. This includes EOS and other missions’ standard and special mission output products, metadata, and any ancillary/auxiliary or correlative data sets necessary for the production and validation of those output products, as well as data set documentation. Data sources may include SIPSs, EOS Data and Operations System (EDOS), Flight Dynamics System, NASA Sentinel Gateway, other agency interfaces (e.g. NOAA), other DAACs, designated external data providers, and others as appropriate.
The XDAAC archives designated data products and distributes them to users. Some products will be created for distribution through an on-demand processing system and are subject to the same delivery requirements as products that are stored in the archive. The XDAAC distributes data electronically over designated Mission Support Networks and Science Support Networks as defined in the Networks Interface documentation (ICD between EOS Networks and EOSDIS Elements (423-ICD-002). Electronic data transmission must comply with all applicable NASA security standards.
2.3.2 Data Archive
- The XDS shall store all designated data products in an online primary archive system or create products on demand. The XDAAC shall assure that products generated on demand (virtual products) are identical to the corresponding standard products that would otherwise be archived, having undergone initial and periodic ongoing operational and scientific quality assessment. Designated products will be established by ESDIS for each XDAAC and listed in a separate document to be provided to ESDIS by the XDAAC.
- The XDS shall provide the capability of retrieving any data granule stored in the storage system.
- The XDS shall have the capability to archive multiple versions of selected archive data as designated by ESDIS.
- The XDS shall ensure data integrity through the use of checksums in order to satisfy interface requirements with external systems.
- The XDAAC shall obtain and maintain a digital object identifier (DOI) for all standard data product collections distributed to users via the ESDIS-provided process.
- The XDAAC shall maintain catalog metadata for all products that are available from the XDS. The metadata content shall conform to ESDIS Standards Coordination Office (ESCO) approved standards for metadata and be provided in a form that can be easily translated into the Unified Metadata Model (UMM).
- The XDAAC shall maintain archiving, distribution and user services functions for designated data products until notified by ESDIS to stop maintaining these functions.
- As part of the EOS missions data flow for some missions, EDOS maintains a backup of the Level 0 data at their offices in White Sands, NM. For those missions where White Sands is the Level 0 backup, the XDAAC shall be able to interface with the EDOS system and recover Level 0 data (when necessary) from the back-up archive in White Sands, NM.
- The XDAAC shall maintain an off-site backup copy of all other data that would be impossible or difficult to recover in case of loss (e.g., ancillary data, metadata, command history, Science Software, engineering data, calibration data, systems and applications software, selected data products, depending on need). Some off-site backup copies of data may not be directly in XDS control, however, the XDAAC shall be aware of the locations of all backup copies of data. For example, if the backup copy of the data is located at a different DAAC or in a NASA Cloud archive, the XDAAC shall be aware of the location of that backup and have access to it.
- The XDS shall have the capability to restore its archive to avoid permanent loss of archived data.
- The XDS shall allow for new technology integration and scalable systems for archival data.
- The XDS shall allow old versions of data to be removed from the archive.
- The XDS shall provide archive metrics to the EMS. The metrics are described in Appendix A.
2.3.3 Data Distribution
- The XDS shall distribute data to users in accordance with the NASA Earth Science Data and Information Policy that promotes the full and open sharing of all data with the research and applications communities, private industry, academia, and the general public. Data includes standard and special data products, metadata, ancillary/auxiliary data, calibration data, science software source code, browse data, and documentation. ESDIS may grant exceptions to this requirement via waivers for individual data products.
- The XDAAC shall provide data product access on a non-discriminatory basis so that all users are treated equally except where there are restrictions for data products as required and approved by ESDIS.
- The XDAAC shall implement Earthdata Login, following ESDIS procedures, for applications where data from data products are retrieved by humans or machines. This includes direct downloads of data files (e.g. via http/https) as well as retrievals of data through data services such as OPeNDAP. Waivers to this policy shall be approved by ESDIS. User access to some data (such as metadata, science software, browse data, and documentation) shall not require Earthdata Login.
- The XDS shall be capable of responding to users’ requests for data. The XDS shall be capable of providing subscriptions to datasets.
- On an XDS that supports an order-based system, the XDS shall provide automated status information to users regarding their data orders.
- The XDS shall distribute data in ESCO-approved standard formats commonly accepted within the user community for the types of data for which the XDAAC is responsible. Waivers to this policy shall be approved by ESDIS. The ESCO approved standards are listed at /user-resources/standards-and-references.
- The XDS shall distribute data to users via electronic networks.
- The XDS shall distribute data to various data processing systems, instrument teams’ science computing facilities, SIPS, and other DAACs to support product generation and quality assurance in a timely manner to support production schedules.
- The XDS shall provide subsetted data and/or subsetting, reprojection, and format conversion tools appropriate to the XDAAC’s data holdings to ensure efficient distribution of data to users.
- The XDS shall provide distribution metrics to EMS. The metrics are described in the Appendix A.
- The XDS shall ensure data integrity through the use of checksums in order to satisfy interface requirements with external systems.
The performance requirements are based on current performance at the DAACs and agreements between each XDAAC and ESDIS.
- The XDS shall be capable of receiving and archiving the data products resulting from forward processing of data simultaneously with those resulting from reprocessing.
- For an XDS that supports an order-based system, the XDS shall make the data available for electronic pickup by the user for at least 24 hours.
2.3.5 Operational Requirements
- The XDAAC shall notify ESDIS of off-nominal operational instances by the next business day of the occurrence. Off-nominal occurrences include unanticipated downtime of 8 hours, or compromising security incidences.
- The XDAAC shall provide access to network information to ESDIS and shall be IPv6 compliant.
- The XDS shall operate 24 hours a day, 7 days a week with an operational availability of .96 during any given 30-day period. The XDAAC itself may only be staffed on a 40-hour per week basis, but the XDS shall meet this availability requirement around the clock with the exception of scheduled downtime. (Operational availability is defined excluding scheduled downtime.)
- The XDS shall be able to clear, within 48 hours, any backlogs accumulated during scheduled or unscheduled downtimes.
2.4 Information Management
A primary role of information management is to give the users efficient access to the XDAAC-held data products, providing them with all of the information and tools to search, locate, select, and obtain those products and services required to perform their science investigations. These products may be stored in the archives or may entail either higher level processing of an archived product or the placement of an acquisition and processing request. Information management may include XDS-specific, stand-alone client(s) in addition to Earthdata Search. The XDS shall support the export of granule and collection metadata to CMR using appropriate software tools to facilitate cross-XDS access to EOS data holdings. As a best practice, the XDS should use CMR as its own inventory data discovery system.
2.4.1 Website Management
- The XDAAC shall maintain and manage a public Website that provides, at a minimum, access to and information on XDAAC data products, services, documentation, user support, and guidance on how to properly cite data.
- The XDAAC website shall adhere to the Earthdata User Interface (EUI) guidelines for styles and navigational aids and found on cdn.earthdata.nasa.gov/eui/latest/docs/design/.
2.4.2 Metadata Management
- The standard product metadata shall contain all information required of CMR product metadata as defined in the UMM for Collections (UMM-C).
- The XDAAC shall maintain (export, update, delete) its UMM for Granules (UMM-G) metadata in CMR. The XDAAC shall work along with ESDIS to ensure the currency, correctness, completeness, and quality of XDAAC’s metadata in CMR to enable users’ discovery, search, and use of the XDAAC-held data products using the EDSC or other clients using CMR.
- The XDAAC shall manage the accessibility to granule- and/or collection-level data and services in CMR using rules, conditions, permissions, and restrictions as available in CMR.
- The XDAAC shall maintain a CMR metadata provider profile and provider policy information.
- The XDAAC shall support data product access via CMR, which includes rejecting, accepting, canceling, closing and tracking orders or access requests.
- The XDAAC shall work with CMR support staff and coordinate any significant changes to the peak daily rate of metadata provided to CMR.
- The XDAAC shall work with CMR support staff to resolve user community problems (e.g. orders/requests, availability of data holdings for search and order/request/access) and identified metadata quality issues within a ESDIS-specified timeframe.
2.5 User Services
- The XDAAC shall assist users in the resolution of problems in using XDAAC or EOSDIS services, providing scientific information about the data products, resolving problems with formats and other data related issues, and in resolving download failures.
- The XDAAC shall assist ESDIS in conducting user satisfaction surveys. The XDAAC shall participate in the annual ESDIS survey. The XDAAC may propose alternative measures to document customer satisfaction metrics in addition to the survey. ESDIS conducts an annual survey of users to obtain the American Consumer Satisfaction Index (ACSI) for the services users receive from EOSDIS. The ACSI is a metric that NASA reports annually to the Office of Management and Budget. The goal of the performance requirements below is to maintain or increase the ACSI.
2.6 Information Security
- The XDAAC and XDS shall comply with all of NASA’s information technology and information security requirements as in NASA Procedural Requirement (NPR) 2810.1A (Security of Information Technology (Revalidated with Change 1, dated May 19, 2011)) and NASA Policy Directive (NPD) 2810.1E (NASA Information Security Policy, 7/14/2020). The XDAAC shall provide copies of security documentation to appropriate NASA officials and ESDIS.
Appendix A Performance Metrics in Use By ESDIS
- Ingest rates and statistics on all products by XDAAC, Level, Discipline, Mission, Instrument, by time period (including volumes, numbers of granules).
- Archive rates and statistics on all products by XDAAC, Level, Discipline, Mission, Instrument, by time period (including volumes, numbers of granules).
- Product latencies on Near Real-time products by instrument, time period.
- Data distribution metrics by XDAAC shall include volumes, number of files, number of products, number of users, types of products, types of users for given time periods. Various groupings of these metrics are used, for example by: IPhost internet domain (.com, nasa.gov, noaa.gov, umd.edu, etc. as well as individual e-mail addresses used for user satisfaction survey), instrument, mission or campaign, discipline, country, distribution type, registered users.
- Number of user accesses to XDAACs and grouping by user characterization such as country, internet domain.
- Deleted products from archive (by XDAAC, granule, product, mission, instrument, product version, time period).
Appendix B Abbreviations and Acronyms
|ACSI||American Consumer Satisfaction Index|
|API||Application Programming Interface|
|CCB||Configuration Change Board|
|CCR||Configuration Change Request|
|CMR||Common Metadata Repository|
|DAAC||Distributed Active Archive Center|
|DCN||Document Change Notice|
|DOI||Digital Object Identifier|
|EDOS||EOS Data and Operations System|
|EMS||EOSDIS Metrics System|
|EOS||Earth Observing System|
|EOSDIS||Earth Observation System Data and Information System|
|ESDIS||Earth Science Data and Information System|
|GCMD||Global Change Master Directory|
|GIBS||Global Imagery Browse Services|
|HTTP||Hypertext Transfer Protocol|
|ICD||Interface Control Document|
|NASA||National Aeronautics and Space Administration|
|NPD||NASA Policy Directive|
|NPR||NASA Procedural Requirement|
|OPeNDAP||Open-source Project for a Network Data Access Protocol|
|SIPS||Science Investigator-led Processing System|
|UMM||Unified Metadata Model|
|UMM-C||Unified Metadata Model for Collections|
|UMM-G||Unified Metadata Model for Granules|
|URL||Universal Resource Locator|
|XDAAC||X Distributed Active Archive Center|
|XDS||X Data System|