NASA's Earth Science Data Systems (ESDS) Program is committed to making the data it oversees findable, accessible, interoperable, and reusable (FAIR). The more than 100 petabytes (PB) of data in NASA's Earth Observing System Data and Information System (EOSDIS) archive are on the verge of exponential growth from numerous planned high-data-volume missions. Current estimates project that the volume of NASA Earth science data could grow to nearly 600 PB by 2030. In keeping with NASA’s open data policies, data from these missions will be available as early in the mission process as feasible.
Developing a Mission Data Processing System (MDPS) that will ensure the most transparent processing and delivery of future NASA mission and project data is vital. The MDPS is the set of algorithms, software, compute infrastructure, operational procedures, documentation, and teams that process raw instrument data into science quality data products. The MDPS also includes the software tools that support the development of processing algorithms and the validation and analysis of processed data.
The Challenge
In the current NASA Earth science data processing architecture, each mission receives its raw instrument data through the multi-mission Near Space Network and processes data using a mission-specific MDPS. Created data products are then delivered to NASA’s EOSDIS to be archived and distributed.
This siloed data processing architecture creates several difficulties for upcoming high-data volume missions. It is not conducive to facilitating coordinated missions, it creates barriers to enabling broad and early access to science and related software, and it complicates intra-mission and instrument science.
To address these issues, NASA Chief Science Data Officer Kevin Murphy issued a challenge to the broad mission processing community to identify the best model for an open MDPS to support upcoming missions. Specifically, the community was challenged to identify a data processing architecture that not only meets mission science processing objectives and supports Earth system science, but also promotes open science principles and enables data system efficiencies.
Finding the Right Mission Data Processing System
The Multi-Mission Data Processing Study is being conducted to identify the best MDPS for NASA Earth science data processing and delivery. This study comprises several phases:
- Phase 1: Concept Study (October 2021 to March 2023)
- Phase 2: Design Review and Architecture Study (March 2023 through September 2024)
- Phase 3: System Development
- Phase 4: System Implementation
The objective of Phase 1 was to evaluate and assess various system architectures and develop a recommendation for an architecture that would best serve future missions and data users. Two public workshops were conducted between October 2021 and March 2022 that comprised the Open Source Science for the Earth System Observatory Mission Data Processing Study Workshops. The first workshop focused on collecting NASA stakeholder objectives and mission requirements. The second workshop studied practices across NASA and other agencies for developing science data processing systems. A technical trade study was performed along with a programmatic trade study of different architectures. The results from these studies are summarized in the Workshop Final Report.
Phase 2 began in March 2023 and a final report is scheduled for release in September 2024. This follow-on investigation will increase the fidelity of the architecture by establishing the use cases, system requirements, system definition, and potential impact to upcoming missions. Phase 2 will conclude with a final report and a data processing prototype with associated documentation.
Open Results from an Open Study
Open science is a foundational objective of NASA’s Science Mission Directorate (SMD), and is the principle and practice of making research products and processes available to all, while respecting diverse cultures, maintaining security and privacy, and fostering collaborations, reproducibility, and equity.
In keeping with these guidelines, the Multi-Mission Data Processing System Study is fully open and encourages participation by the broader science and data processing communities. It is guided by open science practices and constraints:
- The study features public workshops and open requests for information, and includes broad social media and outreach efforts
- Public workshops enable community participation, engage key stakeholders, and promote diverse and inclusive discussions
- The study solicits input from a broad and diverse set of flight project teams, industry partners, open science experts, and stakeholders across a wide spectrum of the science mission data systems community
- Representatives from upcoming missions participate as members of the study team
- The recommended MDPS is aligned with agency open data, software, algorithm, and publication policies (such as NASA’s SPD-41a)