The Multi-Mission Data Processing System Study is being conducted to identify the best model for an open Mission Data Processing System (MDPS) to support upcoming NASA Earth science missions, particularly the coordinated missions comprising NASA's Earth System Observatory (ESO). The study will identify a data processing architecture that not only meets future mission science processing objectives and supports Earth system science, but also promotes open science principles and enables data system efficiencies.
The study comprises several phases, with each phase building on the results of the previous phase. In keeping with NASA open science guidelines and objectives, all phases in the study are open and encourage participation by the broader science and data processing communities. Phase 2: Design Review and Architecture Study is currently underway.
Back to Multi-Mission Data Processing System Study home page.
Phase 1: Identifying a Recommended Mission Data Processing System
The objective of Phase 1 was to evaluate and assess various system architectures and develop a recommendation for an architecture that would best serve future missions and data users. Two public workshops were conducted. Workshop #1 focused on collecting NASA stakeholder objectives and mission requirements. Workshop #2 studied practices across NASA and other agencies for developing science data processing systems. The information from these workshops was used to inform and identify potential architectures that could meet the study objectives. A technical trade study was performed along with a programmatic trade study of different architectures. The results from these studies were then combined to establish a final recommendation.
The recommendation of the study team is for upcoming missions to develop their own MDPS using a common architecture and services that are provided and managed by an overarching Multi-Mission Organization (MMO). The MMO will establish standards across missions and develop and deliver infrastructure, data catalog, analysis, and (potentially) processing services. This MDPS is designated Type 2 (Managed Services), Variant 4 (encompassing infrastructure, data, catalog, and analysis and processing services) and written as T2V4. To read the full Phase 1 report, please use the link below.
- Workshop #1, October 19-20, 2021
-
We began collecting information with our first virtual two-day workshop on Tuesday, October 19, 2021 (programmatic presentations) and Wednesday, October 20, 2021 (flight project presentations).
The input provided, along with information collected in the second workshop (scheduled for February 2022) on the state-of-the-art in science processing and open science, will be used by the ESO Open Source Processing Study Architecture Working Group to develop and assess potential architectures.
Agenda
Open science is defined as a collaborative culture enabled by technology that empowers the open sharing of data, information and knowledge within the scientific community and the wider public to accelerate scientific research and understanding. Under the guidelines set forth in the 2017 Decadal survey, NASA's new set of Earth-focused missions that form the ESO will adhere to open data, software, algorithm, and publication policies where missions will develop algorithms (in Levels 1–4), software, and documentation in open systems from inception. In order to generate community input and support, this first workshop solicited input from NASA programmatic officials and flight project teams.
Recordings of the workshop are available: ESO Mission: Data Processing Study Workshop
The workshop report is also available: ESO Mission Data Processing Study - Summary of NASA Program Offices and ESO Missions Requirements, Constraints, Recommendations, and Opportunities.
Day 1: Tuesday October 19th, 2021 (1 - 5 PM EST)
Duration EST PST Topic Speaker 0:10 1:00 PM 10:00 AM Welcome and Purpose Andrew Mitchell 0:15 1:10 PM 10:10 AM Earth Science Division (ESD) Opening Remarks Kevin Murphy 0:15 1:25 PM 10:25 AM SMD Scientific Information Policy Steven Crawford 0:15 1:40 PM 10:40 AM ESDS Perspective of Open Science Katie Baynes 0:15 1:55 PM 10:55 AM Overview of the Mission Processing Study Andrew Bingham
0:15 2:10 PM 11:10 AM Q&A with the SAWG 0:10 2:25 PM 11:25 AM Break 0:15 2:35 PM 11:35 AM Flight Projects Program Perspective Charles Webb, Kathleen Boggs 0:15 2:50 PM 11:50 AM Earth Science Technology Office (ESTO) Perspective Pam Millar, Jacqueline Lemoigne-Stewart, Ben Smith
0:15 3:05 PM 12:05 PM R&A Program Perspective Jack Kaye 0:15 3:20 PM 12:20 PM Q&A with the SAWG 0:10 3:35 PM 12:35 PM Break 0:15 3:45 PM 12:45 PM Applied Science Program Perspective Nancy Searby 0:15 4:00 PM 1:00 PM Tsengdar Lee 0:10 4:15 PM 1:15 PM Q&A with the SAWG 0:30 4:25 PM 1:25 PM Open Discussion 4:55 PM 1:55 PM END Day 2: Wednesday October 20th, 2021 (12:20 - 5 PM EST)
Duration EST PST Topic Speaker 0:10 12:20 PM 9:20 AM Recap and Agenda 0:15 12:30 PM 9:30 AM NISAR Programmatic Perspective Gerald Bawden 0:15 12:45 PM 9:45 AM NISAR Project Science Paul Rosen 0:15 1:00 PM 10:00 AM NISAR Project Data Processing Systems Hook Hua 0:10 1:15 PM 10:15 AM Q&A with the SAWG 0:05 1:25 PM 10:25 AM Break 0:15 1:30 PM 10:30 AM Atmosphere Observing System (AOS) Programmatic Perspective Hal Maring 0:15 1:45 PM 10:45 AM AOS Project Science Scott Braun 0:15 2:00 PM 11:00 AM AOS Project Data Processing Systems Robert Wolfe 0:10 2:15 PM 11:15 AM Q&A with the SAWG 0:05 2:25 PM 11:25 AM Break 0:15 2:30 PM 11:30 AM Mass Change (MC) Programmatic Perspective Lucia Tsaoussi 0:15 2:45 PM 11:45 AM MC Project Science David Wiese 0:15 3:00 PM 12:00 PM MC Project Data Processing Systems David Wiese 0:10 3:15 PM 12:15 PM Q&A with the SAWG 0:05 3:25 PM 12:25 PM Break 0:20 3:30 PM 12:30 PM Surface Biology and Geology (SBG) Overview and Project Science Dave Schimel 0:20 3:50 PM 12:50 PM SBG Project Data Processing Systems Jeff Pon 0:05 4:10 PM 1:10 PM SBG Programmatic Perspective Ben Phillips 0:10 4:15 PM 1:15 PM Q&A with the SAWG 0:30 4:25 PM 1:25 PM Open Discussion 4:55 PM 1:55 PM END - Workshop #2, March 1-4, 2022
-
This workshop focused on understanding the state-of-the-art in Big Data processing systems. The workshop was open to the public. A Request For Information (RFI; submission period now closed) sought input from organizations with relevant expertise in Big Data processing and open science to help guide the study and workshop.
Recordings of the workshop are available: ESO Mission: Data Processing Study Workshop.
The workshop report is also available: ESO Mission Data Processing Study: Summary of State-of-the-Practice and State-of-the-Art Mission Data Processing System Architectures.
Please direct questions about the Open Source Science for the Earth System Observatory Mission Science Data Processing Study Workshops to: karen.yuen@jpl.nasa.gov or sara.h.lubkin@nasa.gov.
Agenda
Day 1: Tuesday, March 1, 2022 (12 - 5 pm EST)
Time
ETTime
PSTTitle Speaker Organization 12:00 PM 9:00 AM Welcome and Study Overview Andy Mitchell and Andy Bingham GSFC/JPL 12:15 PM 9:15 AM Logistics Sara Lubkin and Karen Yuen GSFC/JPL 12:20 PM 9:20 AM Opening Remarks Kevin Murphy NASA 12:25 PM 9:25 AM Open Source Science / SPD-41
doi:10.5281/zenodo.6320745Katie Baynes NASA 12:35 PM 9:35 AM Session 1 introduction: Science collaboration approaches Chelle Gentemann Farallon Institute 12:40 PM 9:40 AM SAR data exploration in environmental sciences: challenges and opportunities Naiara Pinto JPL 12:52 PM 9:52 AM Imagining a new NASA computing platform with AI + big data-supported analysis
doi:10.5281/zenodo.6320918Willow Coleman Harvey Mudd College 1:04 PM 10:04 AM Big Data Community Algorithms: Deep Learning for Mapping
doi:10.5281/zenodo.6320982Morteza Karimzadeh University of Colorado 1:16 PM 10:16 AM Cloud Computing Platforms for Processing Geospatial Big Data: Current Status and Challenges. doi:10.5281/zenodo.6307732 Qiusheng Wu University of Tennessee, Knoxville 1:28 PM 10:28 AM Project Jupyter - Lessons and Principles from a Community-Driven Open Source Project
doi:10.5281/zenodo.6320538Fernando Perez University of California, Berkeley 1:40 PM 10:40 AM Fishbowl Discussion Q&A led by the SAWG 2:00 PM 11:00 AM Break 2:10 PM 11:10 AM Session 2 introduction: NASA Earth Systematic Mission Processing Architectures Andy Mitchell GSFC 2:15 PM 11:15 AM Terra MODIS
doi:10.5281/zenodo.6320958Robert Wolfe GSFC 2:30 PM 11:30 AM CLAREO
doi:10.5281/zenodo.6345252Chris Currey LaRC 2:45 PM 11:45 AM ICESat-2
doi:10.5281/zenodo.6320538Tom Neumann GSFC 3:00 PM 12:00 PM PACE
doi:10.5281/zenodo.6313234Sean Bailey and Bryan Franz GSFC 3:15 PM 12:15 PM SWOT
doi:10.5281/zenodo.6315458Oh-Ig Kwoun JPL 3:30 PM 12:30 PM Fishbowl Discussion Q&A led by the SAWG 3:45 PM 12:45 PM Break 3:55 PM 12:55 PM Break Room Topics & Logistics 4:10 PM 1:10 PM Breakout Rooms
1. System development approaches & challenges
2. System operations approaches & challenges
3. Open-sourced science approaches & challenges
4. Data analysis needs for Mission Data Processing System (MDPS)
5. Open source software approaches & challenges
6. MDPS Architectures now and the future4:40 PM 1:40 PM Report out from Breakout rooms 5:10 PM 2:10 PM END Day 2: Wednesday, March 2, 2022 (12 - 4:30 pm EST)
Time
ETTime
PSTTitle Speaker Organization 12:00 PM 9:00 AM Logistics Sara Lubkin and Karen Yuen GSFC/JPL 12:05 PM 9:05 AM Session 3 introduction: NASA Earth System Science Pathfinder mission processing architectures Andy Mitchell GSFC 12:10 PM 9:10 AM OCO-2/OCO-3
doi:10.5281/zenodo.6321343Cecilia Cheng JPL 12:25 PM 9:25 AM EMIT Phil Brodrick JPL 12:40 PM 9:40 AM MAIA
doi:10.5281/zenodo.6323718Jeff Walter LaRC 12:55 PM 9:55 AM Break 1:05 PM 10:05 AM TROPICS
doi:10.5281/zenodo.6323291Jess Braun University of Wisconsin, Madison 1:20 PM 10:20 AM GEDI Scott Luthke GSFC 1:35 PM 10:35 AM Fishbowl Discussion Q&A led by the SAWG 1:50 PM 10:50 AM Session 4 introduction: Non-NASA Earth science mission processing architectures Andy Bingham JPL 1:55 PM 10:55 AM National Oceanic and Atmospheric Administration
doi:10.5281/zenodo.6307369Lihang Zhou and Joseph Mani NOAA 2:10 PM 11:10 AM U.S. Geological Survey Chris Engebretson USGS 2:25 PM 11:25 AM Fishbowl Discussion Q&A led by the SAWG 2:40 PM 11:40 AM Break 2:50 PM 11:50 AM Breakout Rooms
1. System development approaches & challenges
2. System operations approaches & challenges
3. Open-sourced science approaches & challenges
4. Data analysis needs for Mission Data Processing System (MDPS)
5. Open source software approaches & challenges
6. MDPS Architectures now and the future3:20 PM 12:20 PM Report out from Breakout rooms 3:50 PM 12:50 PM Japan Aerospace Exploration Agency
doi:10.5281/zenodo.6319381Osamu Ochia JAXA 4:05 PM 1:05 PM END Day 3: Thursday, March 3, 2022 (12 - 5:20 pm EST)
Time
ETTime
PSTTitle Speaker Organization 12:00 PM 9:00 AM Logistics 12:05 PM 9:05 AM National Remote Sensing Center, India Space Research Organization (ISRO)
doi:10.5281/zenodo.6302928Uzair Mujeeb ISRO 12:20 PM 9:20 AM Italian Space Agency (ASI) Francesco Tataranni ASI 12:35 PM 9:35 AM German Aerospace Center (DLR)
doi:10.5281/zenodo.6315122Conrad Albrecht DLR 12:50 PM 9:50 AM European Space Agency (ESA) Anca Anghelea, Klaus Scipal, and Clement Albinet ESA 1:05 PM 10:05 AM Fishbowl Discussion Q&A led by the SAWG 1:20 PM 10:20 AM Break 1:30 PM 10:30 AM Session 5 introduction: Non-Earth science mission processing architectures Luke Dahl 1:35 PM 10:35 AM Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST)
doi:10.5281/zenodo.6311597KT Lim Vera C. Rubin Observatory 1:50 PM 10:50 AM Nancy Grace Roman Space Telescope
doi:10.5281/zenodo.6326463Megan Soey Space Telescope Science Institute (STScI) 2:05 PM 11:05 AM IPAC Missions Gregory Dubois-Felsmann Caltech 2:20 PM 11:20 AM Fishbowl Discussion Q&A led by the SAWG 2:35 PM 11:35 AM Break 2:45 PM 11:45 AM Session 6 introduction: System interfaces and standards Andy Bingham 2:50 PM 11:50 AM Earth Science Data and Information System (ESDIS) Sara Lubkin GSFC 3:05 PM 12:05 PM NASA Transform to Open Science (TOPS)
doi:10.5281/zenodo.6324320Chelle Gentemann Farallon Institute 3:20 PM 12:20 PM NASA's High-End Computing Capability: Growing to Support Science Data Processing for the Earth System Observatory Missions
doi:10.5281/zenodo.6330014Jon Jenkins Ames Research Center 3:35 PM 12:35 PM Data System Standards Ingo Simonis OGC 3:50 PM 12:50 PM Fishbowl Discussion Q&A led by the SAWG 4:05 PM 1:05 PM Break 4:05 PM 1:05 PM Breakout Rooms
1. System development approaches & challenges
2. System operations approaches & challenges
3. Open-sourced science approaches & challenges
4. Data analysis needs for Mission Data Processing System (MDPS)
5. Open source software approaches & challenges
6. MDPS Architectures now and the future4:35 PM 1:35 PM Report out from Breakout rooms 5:05 PM 2:05 PM END Day 4: Friday, March 4, 2022 (12 - 4 pm EST)
Time
ETTime
PSTTitle Speaker Organization 12:00 PM 9:00 AM Logistics 12:05 PM 9:05 AM Session 7 introduction: Other Big Data processing system architectures Luke Dahl 12:10 PM 9:10 AM CyVerse Tyson Swetnam, Edwin Skidmore University of Arizona 12:25 PM 9:25 AM Pangeo
doi:10.5281/zenodo.6330215Joe Hamman Carbon Plan 12:40 PM 9:40 AM Alsaka SAR Facility Kirk Hogenson Alaska Satellite Facility 12:55 PM 9:55 AM Raytheon
doi:10.5281/zenodo.6325581Shawn Miller Raytheon 1:10 PM 10:10 AM Element 84 STAC
doi:10.5281/zenodo.6326572Dan Pilone Element 84 1:25 PM 10:25 AM Break 1:35 PM 10:35 AM Fishbowl Discussion Q&A led by the SAWG 1:50 PM 10:50 AM Red Hat Dan McGuan Red Hat 2:05 PM 11:05 AM Amazon Web Services (AWS) AWS 2:20 PM 11:20 AM Multi-Mission Algorithm and Analysis Platform (MAAP)
doi:10.5281/zenodo.6345081George Chang JPL 2:35 PM 11:35 AM OpenNEX
doi:10.5281/zenodo.6345019Jennifer Dungan and Ian Brosnan ARC 2:50 PM 11:50 AM Break 3:00 PM 12:00 PM Unity (Science Data System as a Service)
doi:2014/54386Hook Hua and Laura Jewell JPL 3:15 PM 12:15 PM Science Data Analytics Platform (SDAP)
doi:10.5281/zenodo.6310363Nga Chung and Thomas Huang JPL 3:30 PM 12:30 PM NASA Earth Information System (EIS) Alexey Shiklomanov GSFC 3:45 PM 12:45 PM Fishbowl Discussion Q&A led by the SAWG 4:00 PM 1:00 PM END of Workshop - Phase 1 Study Participants
-
Name Focus Area Organization Required Expertise Elias Sayfi Co-Chair JPL - Flight mission data processing system architect.
- Community builder
Natasha Stavros Co-Chair University of Colorado, Boulder - Science data processing system architect
- Community builder
Hook Hua SBG and NISAR Science Processing Representative JPL - Science data processing systems
- Level 3 (L3) requirements and mission objectives
Curt Tilmes AOS Science Processing Representative Goddard/Ames Research Center - Science data processing systems
- L3 requirements and mission objectives
Bernie Bienstock MC Science Processing Representative JPL - Science data processing systems
- L3 requirements and mission objectives
Wenying Su Applied Science Algorithm Developer Langley Research Center - Expertise developing L3+ algorithms for science applications
- Experience integrating algorithms into a pipeline.
- Experience working in a collaborative development environment.
Andy Michaelis Data Processing Workflow Community-focused Developer Ames - Expertise in modern open-source development practices, cloud-based orchestration, GitHub, Continuous Integration.
- Experience with big data applications (science preferred, but not necessary)
Lesley Ott Geophysical Numerical Modeler Goddard - Expertise with HPC
- Expertise with big data assimilation and operations
Qing Yue Instrument Algorithm Developer JPL - L2 algorithm development
- Science data processing systems
Evelyn Ho Science Processing Architect from a comparable big-data science-based organization or project Earth Science Data Information System (ESDIS) Science Investigator-led Processing Systems (SIPS) Chris Engebretson Science Processing Architect from a comparable big-data science-based organization or project USGS Adrian Parker Science Processing Architect from a comparable big-data science-based organization or project NOAA Sean Harkin Science Processing Architect from a comparable big-data science-based organization or project Marshall Space Flight Center