ESDS Program

Multi-Mission Data Processing System Study, Phase 1: Concept Study

The Multi-Mission Data Processing System Study is being conducted to identify the best model for an open Mission Data Processing System (MDPS) to support upcoming NASA Earth science missions, particularly the coordinated missions comprising NASA's Earth System Observatory (ESO). The study will identify a data processing architecture that not only meets future mission science processing objectives and supports Earth system science, but also promotes open science principles and enables data system efficiencies.

The study comprises several phases, with each phase building on the results of the previous phase. In keeping with NASA open science guidelines and objectives, all phases in the study are open and encourage participation by the broader science and data processing communities. Phase 2: Design Review and Architecture Study is currently underway.

Back to Multi-Mission Data Processing System Study home page.

Phase 1: Identifying a Recommended Mission Data Processing System

The objective of Phase 1 was to evaluate and assess various system architectures and develop a recommendation for an architecture that would best serve future missions and data users. Two public workshops were conducted. Workshop #1 focused on collecting NASA stakeholder objectives and mission requirements. Workshop #2 studied practices across NASA and other agencies for developing science data processing systems. The information from these workshops was used to inform and identify potential architectures that could meet the study objectives. A technical trade study was performed along with a programmatic trade study of different architectures. The results from these studies were then combined to establish a final recommendation.

The recommendation of the study team is for upcoming missions to develop their own MDPS using a common architecture and services that are provided and managed by an overarching Multi-Mission Organization (MMO). The MMO will establish standards across missions and develop and deliver infrastructure, data catalog, analysis, and (potentially) processing services. This MDPS is designated Type 2 (Managed Services), Variant 4 (encompassing infrastructure, data, catalog, and analysis and processing services) and written as T2V4. To read the full Phase 1 report, please use the link below.

Image
Rectangular flow graphic showing the four phases of the mission processing system evaluation.
Phase 1 was a detailed analysis of Mission Data Processing System (MDPS) architectures to find the best combination to support multi-mission use and individual mission needs.
Workshop #1, October 19-20, 2021

We began collecting information with our first virtual two-day workshop on Tuesday, October 19, 2021 (programmatic presentations) and Wednesday, October 20, 2021 (flight project presentations).

The input provided, along with information collected in the second workshop (scheduled for February 2022) on the state-of-the-art in science processing and open science, will be used by the ESO Open Source Processing Study Architecture Working Group to develop and assess potential architectures.

Agenda

Open science is defined as a collaborative culture enabled by technology that empowers the open sharing of data, information and knowledge within the scientific community and the wider public to accelerate scientific research and understanding. Under the guidelines set forth in the 2017 Decadal survey, NASA's new set of Earth-focused missions that form the ESO will adhere to open data, software, algorithm, and publication policies where missions will develop algorithms (in Levels 1–4), software, and documentation in open systems from inception. In order to generate community input and support, this first workshop solicited input from NASA programmatic officials and flight project teams.

Recordings of the workshop are available: ESO Mission: Data Processing Study Workshop

The workshop report is also available: ESO Mission Data Processing Study - Summary of NASA Program Offices and ESO Missions Requirements, Constraints, Recommendations, and Opportunities.

 

Day 1: Tuesday October 19th, 2021 (1 - 5 PM EST)

Duration EST PST Topic Speaker
0:10 1:00 PM 10:00 AM Welcome and Purpose Andrew Mitchell
0:15 1:10 PM 10:10 AM Earth Science Division (ESD) Opening Remarks Kevin Murphy
0:15 1:25 PM 10:25 AM SMD Scientific Information Policy Steven Crawford
0:15 1:40 PM 10:40 AM ESDS Perspective of Open Science Katie Baynes
0:15 1:55 PM 10:55 AM Overview of the Mission Processing Study

Andrew Bingham

0:15 2:10 PM 11:10 AM Q&A with the SAWG  
0:10 2:25 PM 11:25 AM Break  
0:15 2:35 PM 11:35 AM Flight Projects Program Perspective Charles Webb, Kathleen Boggs
0:15 2:50 PM 11:50 AM Earth Science Technology Office (ESTO) Perspective

Pam Millar, Jacqueline Lemoigne-Stewart, Ben Smith

0:15 3:05 PM 12:05 PM R&A Program Perspective Jack Kaye
0:15 3:20 PM 12:20 PM Q&A with the SAWG  
0:10 3:35 PM 12:35 PM Break

 

0:15 3:45 PM 12:45 PM Applied Science Program Perspective Nancy Searby
0:15 4:00 PM 1:00 PM

High Performance Computing (HPC) Perspective

Tsengdar Lee
0:10 4:15 PM 1:15 PM Q&A with the SAWG  
0:30 4:25 PM 1:25 PM Open Discussion  
  4:55 PM 1:55 PM END  

 

Day 2: Wednesday October 20th, 2021 (12:20 - 5 PM EST)

Duration EST PST Topic Speaker
0:10 12:20 PM 9:20 AM Recap and Agenda  
0:15 12:30 PM 9:30 AM NISAR Programmatic Perspective Gerald Bawden
0:15 12:45 PM 9:45 AM NISAR Project Science Paul Rosen
0:15 1:00 PM 10:00 AM NISAR Project Data Processing Systems Hook Hua
0:10 1:15 PM 10:15 AM Q&A with the SAWG  
0:05 1:25 PM 10:25 AM Break  
0:15 1:30 PM 10:30 AM Atmosphere Observing System (AOS) Programmatic Perspective Hal Maring
0:15 1:45 PM 10:45 AM AOS Project Science Scott Braun
0:15 2:00 PM 11:00 AM AOS Project Data Processing Systems Robert Wolfe
0:10 2:15 PM 11:15 AM Q&A with the SAWG  
0:05 2:25 PM 11:25 AM Break  
0:15 2:30 PM 11:30 AM Mass Change (MC) Programmatic Perspective Lucia Tsaoussi
0:15 2:45 PM 11:45 AM MC Project Science David Wiese 
0:15 3:00 PM 12:00 PM MC Project Data Processing Systems David Wiese
0:10 3:15 PM 12:15 PM Q&A with the SAWG  
0:05 3:25 PM 12:25 PM Break  
0:20 3:30 PM 12:30 PM Surface Biology and Geology (SBG) Overview and Project Science Dave Schimel
0:20 3:50 PM 12:50 PM SBG Project Data Processing Systems Jeff Pon
0:05 4:10 PM 1:10 PM SBG Programmatic Perspective Ben Phillips
0:10 4:15 PM 1:15 PM Q&A with the SAWG  
0:30 4:25 PM 1:25 PM Open Discussion  
  4:55 PM 1:55 PM END  
Workshop #2, March 1-4, 2022

This workshop focused on understanding the state-of-the-art in Big Data processing systems. The workshop was open to the public. A Request For Information (RFI; submission period now closed) sought input from organizations with relevant expertise in Big Data processing and open science to help guide the study and workshop.

Recordings of the workshop are available: ESO Mission: Data Processing Study Workshop.

The workshop report is also available: ESO Mission Data Processing Study: Summary of State-of-the-Practice and State-of-the-Art Mission Data Processing System Architectures.

Please direct questions about the Open Source Science for the Earth System Observatory Mission Science Data Processing Study Workshops to: karen.yuen@jpl.nasa.gov or sara.h.lubkin@nasa.gov.

Agenda

Day 1: Tuesday, March 1, 2022 (12 - 5 pm EST)

Time
ET
Time
PST
Title Speaker Organization
12:00 PM 9:00 AM Welcome and Study Overview Andy Mitchell and Andy Bingham GSFC/JPL
12:15 PM 9:15 AM Logistics Sara Lubkin and Karen Yuen GSFC/JPL
12:20 PM 9:20 AM Opening Remarks Kevin Murphy NASA
12:25 PM 9:25 AM Open Source Science / SPD-41
doi:10.5281/zenodo.6320745
Katie Baynes NASA
12:35 PM 9:35 AM Session 1 introduction: Science collaboration approaches Chelle Gentemann Farallon Institute
12:40 PM 9:40 AM SAR data exploration in environmental sciences: challenges and opportunities Naiara Pinto JPL
12:52 PM 9:52 AM Imagining a new NASA computing platform with AI + big data-supported analysis
doi:10.5281/zenodo.6320918

 

Willow Coleman Harvey Mudd College
1:04 PM 10:04 AM Big Data Community Algorithms: Deep Learning for Mapping
doi:10.5281/zenodo.6320982
Morteza Karimzadeh University of Colorado
1:16 PM 10:16 AM Cloud Computing Platforms for Processing Geospatial Big Data: Current Status and Challenges. doi:10.5281/zenodo.6307732 Qiusheng Wu University of Tennessee, Knoxville
1:28 PM 10:28 AM Project Jupyter - Lessons and Principles from a Community-Driven Open Source Project
doi:10.5281/zenodo.6320538
Fernando Perez University of California, Berkeley
1:40 PM 10:40 AM Fishbowl Discussion Q&A led by the SAWG  
2:00 PM 11:00 AM Break    
2:10 PM 11:10 AM Session 2 introduction: NASA Earth Systematic Mission Processing Architectures Andy Mitchell GSFC
2:15 PM 11:15 AM Terra MODIS
doi:10.5281/zenodo.6320958
Robert Wolfe GSFC
2:30 PM 11:30 AM CLAREO
doi:10.5281/zenodo.6345252
Chris Currey LaRC
2:45 PM 11:45 AM ICESat-2
doi:10.5281/zenodo.6320538
Tom Neumann GSFC
3:00 PM 12:00 PM PACE
doi:10.5281/zenodo.6313234
Sean Bailey and Bryan Franz GSFC
3:15 PM 12:15 PM SWOT
doi:10.5281/zenodo.6315458
Oh-Ig Kwoun JPL
3:30 PM 12:30 PM Fishbowl Discussion Q&A led by the SAWG  
3:45 PM 12:45 PM Break    
3:55 PM 12:55 PM Break Room Topics & Logistics    
4:10 PM 1:10 PM Breakout Rooms
1. System development approaches & challenges
2. System operations approaches & challenges
3. Open-sourced science approaches & challenges
4. Data analysis needs for Mission Data Processing System (MDPS)
5. Open source software approaches & challenges
6. MDPS Architectures now and the future
   
4:40 PM 1:40 PM Report out from Breakout rooms    
5:10 PM 2:10 PM END    

 

Day 2: Wednesday, March 2, 2022 (12 - 4:30 pm EST)

Time
ET
Time
PST
Title Speaker Organization
12:00 PM 9:00 AM Logistics Sara Lubkin and Karen Yuen GSFC/JPL
12:05 PM 9:05 AM Session 3 introduction: NASA Earth System Science Pathfinder mission processing architectures Andy Mitchell GSFC
12:10 PM 9:10 AM OCO-2/OCO-3
doi:10.5281/zenodo.6321343
Cecilia Cheng JPL
12:25 PM 9:25 AM EMIT Phil Brodrick JPL
12:40 PM 9:40 AM MAIA
doi:10.5281/zenodo.6323718
Jeff Walter LaRC
12:55 PM 9:55 AM Break    
1:05 PM 10:05 AM TROPICS
doi:10.5281/zenodo.6323291
Jess Braun University of Wisconsin, Madison
1:20 PM 10:20 AM GEDI Scott Luthke GSFC
1:35 PM 10:35 AM Fishbowl Discussion Q&A led by the SAWG  
1:50 PM 10:50 AM Session 4 introduction: Non-NASA Earth science mission processing architectures Andy Bingham JPL
1:55 PM 10:55 AM National Oceanic and Atmospheric Administration
doi:10.5281/zenodo.6307369
Lihang Zhou and Joseph Mani NOAA
2:10 PM 11:10 AM U.S. Geological Survey Chris Engebretson USGS
2:25 PM 11:25 AM Fishbowl Discussion Q&A led by the SAWG  
2:40 PM 11:40 AM Break    
2:50 PM 11:50 AM Breakout Rooms
1. System development approaches & challenges
2. System operations approaches & challenges
3. Open-sourced science approaches & challenges
4. Data analysis needs for Mission Data Processing System (MDPS)
5. Open source software approaches & challenges
6. MDPS Architectures now and the future
   
3:20 PM 12:20 PM Report out from Breakout rooms    
3:50 PM 12:50 PM Japan Aerospace Exploration Agency
doi:10.5281/zenodo.6319381
Osamu Ochia JAXA
4:05 PM 1:05 PM END    

 

Day 3: Thursday, March 3, 2022 (12 - 5:20 pm EST)

Time
ET
Time
PST
Title Speaker Organization
12:00 PM 9:00 AM Logistics    
12:05 PM 9:05 AM National Remote Sensing Center, India Space Research Organization (ISRO)
doi:10.5281/zenodo.6302928
Uzair Mujeeb ISRO
12:20 PM 9:20 AM Italian Space Agency (ASI) Francesco Tataranni ASI
12:35 PM 9:35 AM German Aerospace Center (DLR)
doi:10.5281/zenodo.6315122
Conrad Albrecht DLR
12:50 PM 9:50 AM European Space Agency (ESA) Anca Anghelea, Klaus Scipal, and Clement Albinet ESA
1:05 PM 10:05 AM Fishbowl Discussion Q&A led by the SAWG  
1:20 PM 10:20 AM Break    
1:30 PM 10:30 AM Session 5 introduction: Non-Earth science mission processing architectures Luke Dahl  
1:35 PM 10:35 AM Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST)
doi:10.5281/zenodo.6311597
KT Lim Vera C. Rubin Observatory
1:50 PM 10:50 AM Nancy Grace Roman Space Telescope
doi:10.5281/zenodo.6326463
Megan Soey Space Telescope Science Institute (STScI)
2:05 PM 11:05 AM IPAC Missions Gregory Dubois-Felsmann Caltech
2:20 PM 11:20 AM Fishbowl Discussion Q&A led by the SAWG  
2:35 PM 11:35 AM Break    
2:45 PM 11:45 AM Session 6 introduction: System interfaces and standards Andy Bingham  
2:50 PM 11:50 AM Earth Science Data and Information System (ESDIS)

doi:10.5281/zenodo.6402353

Sara Lubkin GSFC
3:05 PM 12:05 PM NASA Transform to Open Science (TOPS)
doi:10.5281/zenodo.6324320
Chelle Gentemann Farallon Institute
3:20 PM 12:20 PM NASA's High-End Computing Capability: Growing to Support Science Data Processing for the Earth System Observatory Missions
doi:10.5281/zenodo.6330014
Jon Jenkins Ames Research Center
3:35 PM 12:35 PM Data System Standards Ingo Simonis OGC
3:50 PM 12:50 PM Fishbowl Discussion Q&A led by the SAWG  
4:05 PM 1:05 PM Break    
4:05 PM 1:05 PM Breakout Rooms
1. System development approaches & challenges
2. System operations approaches & challenges
3. Open-sourced science approaches & challenges
4. Data analysis needs for Mission Data Processing System (MDPS)
5. Open source software approaches & challenges
6. MDPS Architectures now and the future
   
4:35 PM 1:35 PM Report out from Breakout rooms    
5:05 PM 2:05 PM END    

 

Day 4: Friday, March 4, 2022 (12 - 4 pm EST)

Time
ET
Time
PST
Title Speaker Organization
12:00 PM 9:00 AM Logistics    
12:05 PM 9:05 AM Session 7 introduction: Other Big Data processing system architectures Luke Dahl  
12:10 PM 9:10 AM CyVerse Tyson Swetnam, Edwin Skidmore University of Arizona
12:25 PM 9:25 AM Pangeo
doi:10.5281/zenodo.6330215
Joe Hamman Carbon Plan
12:40 PM 9:40 AM Alsaka SAR Facility

doi:10.5281/zenodo.6354724

Kirk Hogenson Alaska Satellite Facility
12:55 PM 9:55 AM Raytheon
doi:10.5281/zenodo.6325581
Shawn Miller Raytheon
1:10 PM 10:10 AM Element 84 STAC
doi:10.5281/zenodo.6326572
Dan Pilone Element 84
1:25 PM 10:25 AM Break    
1:35 PM 10:35 AM Fishbowl Discussion Q&A led by the SAWG  
1:50 PM 10:50 AM Red Hat Dan McGuan Red Hat
2:05 PM 11:05 AM Amazon Web Services (AWS)   AWS
2:20 PM 11:20 AM Multi-Mission Algorithm and Analysis Platform (MAAP)
doi:10.5281/zenodo.6345081
George Chang JPL
2:35 PM 11:35 AM OpenNEX
doi:10.5281/zenodo.6345019
Jennifer Dungan and Ian Brosnan ARC
2:50 PM 11:50 AM Break    
3:00 PM 12:00 PM Unity (Science Data System as a Service)
doi:2014/54386
Hook Hua and Laura Jewell JPL
3:15 PM 12:15 PM Science Data Analytics Platform (SDAP)
doi:10.5281/zenodo.6310363
Nga Chung and Thomas Huang JPL
3:30 PM 12:30 PM NASA Earth Information System (EIS)

doi.10.5281/zenodo.6353694

Alexey Shiklomanov GSFC
3:45 PM 12:45 PM Fishbowl Discussion Q&A led by the SAWG  
4:00 PM 1:00 PM END of Workshop    
Phase 1 Study Participants
Name Focus Area Organization Required Expertise
Elias Sayfi Co-Chair JPL
  • Flight mission data processing system architect.
  • Community builder
Natasha Stavros Co-Chair University of Colorado, Boulder
  • Science data processing system architect
  • Community builder
Hook Hua SBG and NISAR Science Processing Representative JPL
  • Science data processing systems
  • Level 3 (L3) requirements and mission objectives
Curt Tilmes AOS Science Processing Representative Goddard/Ames Research Center
  • Science data processing systems
  • L3 requirements and mission objectives
Bernie Bienstock MC Science Processing Representative JPL
  • Science data processing systems
  • L3 requirements and mission objectives
Wenying Su Applied Science Algorithm Developer Langley Research Center
  • Expertise developing L3+ algorithms for science applications
  • Experience integrating algorithms into a pipeline.
  • Experience working in a collaborative development environment.
Andy Michaelis Data Processing Workflow Community-focused Developer Ames
  • Expertise in modern open-source development practices, cloud-based orchestration, GitHub, Continuous Integration.
  • Experience with big data applications (science preferred, but not necessary)
Lesley Ott Geophysical Numerical Modeler Goddard
  • Expertise with HPC
  • Expertise with big data assimilation and operations
Qing Yue Instrument Algorithm Developer JPL
  • L2 algorithm development
  • Science data processing systems
Evelyn Ho Science Processing Architect from a comparable big-data science-based organization or project Earth Science Data Information System (ESDIS) Science Investigator-led Processing Systems (SIPS)  
Chris Engebretson Science Processing Architect from a comparable big-data science-based organization or project USGS  
Adrian Parker Science Processing Architect from a comparable big-data science-based organization or project NOAA  
Sean Harkin Science Processing Architect from a comparable big-data science-based organization or project Marshall Space Flight Center  

View Workshop Recordings

Read the Workshop Final Report

Last Updated