Near Real-Time versus Standard Products

Aqua MODIS Cloud Top Temperature over the coast of Kenya and Tanzania on March 16, 2021
Aqua MODIS Cloud Top Temperature over the coast of Kenya and Tanzania on March 16, 2021 at 1105UTC. The clouds over land show sensitivity to different GDAS ancillary input data.

Science quality, or higher-level “standard” data products are an internally consistent, well-calibrated record of the Earth’s geophysical properties to support science. They are made available within 8-40 hours of satellite observation.

If latency is not a primary concern, users are encouraged to use the standard science products, which are created using the best available ancillary, calibration and ephemeris information.

Standard products corresponding to the Land, Atmosphere Near real-time Capability for Earth Observations (EO) (LANCE) near real-time (NRT) products are archived as follows:

  • AIRS and MLS standard products are available through NASA's Goddard Earth Sciences Data and Information Services Center (GES DISC).
  • AMSR SIPS generates AMSR2 standard science quality data products and they are available at NASA's National Snow and Ice Distributed Active Archive Center (NSIDC DAAC).
  • Lightning Imaging Sensor on ISS (ISS LIS) standard products are available from NASA’s Global Hydrometeorology Resource Center DAAC ( GHRC DAAC)
  • Multi-angle Imaging SpectroRadiometer (MISR) standard products are available from NASA's Atmospheric Sciences Data Center (ASDC).
  • Moderate Resolution Imaging Spectroradiometer (MODIS) standard atmosphere and L1 products are available through NASA's Level-1 and Atmosphere Archive and Distribution System (LAADS) DAAC (LAADS DAAC).
  • MODIS standard land products are available from NASA's Land Processes DAAC (LP DAAC) and NSIDC DAAC (snow and ice).
  • OMI standard products are available through GES DISC.
  • OMPS standard products are available through GES DISC.
  • VIIRS land products standard are available from LP DAAC and NSIDC DAAC (snow and ice).
  • VIIRS-Atmosphere standard products are available through NASA's Level 1 and Atmosphere Archive and Distribution System Distributed Active Archive Center (LAADS DAAC) via NASA's Suomi NPP Land SIPS.

Comparisons Between the NRT and Standard Products

AIRS NRT

In order to generate data products within 3 hours of observation time, a number of changes have been made to standard processing approach to expedite the availability of input data sets.

The Atmospheric Infrared Sounder (AIRS) NRT data products are identical to the AIRS routine products except for the following. The AIRS NRT products are produced:

  • The NRT granules are produced without previous or subsequent granules if those granules are not available within 5 minutes (previous and subsequent granules are used for calibration and are generally present in 5 minutes).
  • Predictive ephemeris/attitude data are used (in contrast, definitive ephemeris/attitude data are used for processing standard products).
  • Nominally, a forecast surface pressure is used; if this is unavailable, a surface climatology is then used.
  • No ice cloud properties retrievals are performed (ice cloud properties are a new retrieval in the Version 6 support product AIRX2SUP).

These differences have the following implications for the AIRS NRT Level 1B and Level 2 products.

Level 1B - The atmospheric surface pressure is not used in Level 1B processing, the AIRS Level 1b radiance data produced in the NRT system are nearly identical to the routinely processed AIRS data. There may be small differences in the geolocation (i.e., longitude and latitude). Also, since the NRT system does not wait as long for the previous or subsequent granule data to be present before processing there may be small differences in the radiances of NRT data that were processed without the previous or subsequent granules. This can happen when AIRS is entering or leaving the range of a downlink station (usually are 17 times/day). The reason for the differences is because the standard product uses the space view offset from the previous and subsequent granules for calibration purposes.

Level 2 - The AIRS Level 2 NRT retrieval products can differ from the routine products because of small differences in assumed surface pressure and/or differences in the radiances. Although it is rare, if the forecast surface pressure is not available when the NRT data are produced the NRT algorithm assumes a surface pressure based on a Digital Elevation Map (DEM) rather than the more accurate forecast surface pressure. The differences in the assumed surface pressure tend to be small (~ 10 mbar) near the equator but can become larger (~ 70 mbar) near synoptic weather events. The pressure differences can also lead to differences in the retrieved temperature and water vapor.

For more information, please read the following document, AIRS NRT data products: May 10, 2010, describing the difference between AIRS NRT and Standard Products.

Back to top

AMSR2 NRT

The LANCE system provides access to near real-time data (less than 3 hours from observation). Users, operational agencies and researchers utilize these products for a wide range of purposes (weather forecasting to monitoring natural hazards). These users often need data much sooner than routine science processing allows and are willing to trade science quality for timely access. Science quality products should be used for latency independent research and applications.

The AMSR SIPS generates AMSR2 standard science quality data products and they are available at NSIDC DAAC.

Back to top

LIS NRT (ISS)

Near real-time (NRT), non-quality controlled (NQC) standard data products, and quality controlled (QC) standard data products from the LIS (ISS) are available from NASA's Global Hydrometeorology Resource Center (GHRC DAAC) and the Land, Atmosphere Near real-time Capability for EO (LANCE) system. These data are available in both HDF-4 and netCDF-4 formats, with corresponding browse images in GIF format.

LANCE LIS (ISS) NRT data are available rapidly after an observation (generally within two minutes), and are an excellent resource for applications requiring low data latency, such as tracking on-going severe storms or tracking lightning over oceans and other data-sparse regions. Standard data products, on the other hand, are created daily after all raw observations for the day have been acquired and are more complete than NRT data.

The LIS (ISS) Quality Controlled (QC) standard data products have extensive processing, quality assurance reviews, and validation and should be used in scientific research.

Back to top

MISR NRT

Differences between MISR Level 1 NRT Radiances and Level 1 Standard Products

MISR Level 1 NRT radiance products contain the same content as the standard products. However, changes were made to the code to allow the L1 processing to deal with the session-based MISR L0 data and the Terra attitude data that goes along with this L0 data.

Level 1 Near Real-Time

The MISR NRT Level 1 data products consist of radiance measurements organized in 10-50 minute sessions representing a portion of a single MISR orbit. Measurements are obtained throughout the 380 km MISR swath and projected onto a Space-Oblique Mercator (SOM) map grid. The Ellipsoid-projected and Terrain-projected top-of-atmosphere (TOA) radiance products provide measurements respectively resampled onto the WGS84 reference ellipsoid and onto the topography of a digital elevation model. Spatial sampling is 275 m in all bands of MISR’s nadir camera, and in the red band of the other eight (off-nadir) cameras. The remaining channels are sampled at 1.1 km.

Level 2 Near Real-Time

MISR Level 2 NRT Products are geophysical measurements derived in near real time from Level 1 NRT products. MISR Level 2 NRT Cloud Motion Vector (CMV) datasets are distributed as BUFR and HDF format files within 3 hours of satellite overflight. Product data is organized in 10-50 minute sessions representing a portion of a single MISR orbit. Parameters are projected onto a SOM map grid and defined throughout the 380 km wide MISR swath. At present, available Level 2 NRT products include two formats of the MISR Level 2 CMV product, comprising height-resolved cloud motions representative of the wind field. The CMV product is analogous to the MISR Level 3 CMV product offered in standard production.

MISR BUFR format files have been verified to be compatible with the ECMWF BUFRDC 000400 library and the NCEP BUFRLIB v10-2-3 library.

Differences between MISR Level 2 NRT CMV and Level 3 Standard Products

MISR Level 2 NRT CMV products are intended to contain nearly equivalent science information as L3 MISR CMV product files produced with monthly and seasonal file granularity. The cloud-tracking and reconstruction algorithms of each are equivalent, except for the following details:

  • Whereas the L3 CMV product includes in the product only wind speeds up to speeds of 50 ms-1, the Level 2 NRT CMV product includes wind speeds up to 100 ms-1. Wind speeds greater than 50 ms-1 were originally excluded from the L3 CMV product because they can only be obtained for wind orientations aligned with the ground track of the satellite. The potential sampling bias associated with this limitation is less relevant for rapid response applications.
  • For data sessions in which fewer than 1000 valid retrievals are obtained prior to quality filtering, all retrievals will be omitted on the basis that insufficient data is available to perform necessary checks on the accuracy of camera pointing information.

Level 2 NRT CMV products provide a nominal 75% of the coverage provided by Level 3 CMV products.

Level 3 CMV product datasets are distributed as NetCDF format files comprising data collected over the course of a month, season, or year.

Level 2 NRT CMVs are derived from data sessions typically comprising less than the full orbit employed by standard process.

Level 2 NRT CMV product datasets are generated by a separate implementation of the same algorithms used for standard products that yields nearly equivalent results. Differences are due to the nature of NRT processing and fall within the expected uncertainty of the retrieval.

For more information about Level 2 CMVs, please read section 2 of the following document, JPL D-74995 Data Product Specification for the MISR Cloud Motion Vector Product.

Additional information about the difference between the MISR NRT and standard products is available in the Data Quality Summary for L2 CMV NRT Product and L3 CMV Product.

Differences between MISR Level 2 NRT Aerosols and Level 2 Standard Products

A paper that describes the development of the new NRT aerosol product and summarizes the differences between the NRT and the standard aerosol products has been accepted for publication in the journal Atmospheric Measurement Techniques and is available at https://amt.copernicus.org/preprints/amt-2021-71/.

The MISR NRT aerosol products contain information on atmospheric aerosols, including optical depth, single scattering albedo, Angstrom exponent, and aerosol size. The MISR NRT retrieval strategy builds upon the MISR standard aerosol algorithm. NRT employs the same ancillary datasets as the initial FIRSTLOOK version of the standard product that is available within 2 days from acquisition. The key difference between the standard and NRT processing is in cloud screening: the NRT algorithm does not have access to cloud classifiers that are available in the standard algorithm, and which are generated during Level 1 and Level 2 cloud processing. As a result, in the NRT algorithm, a threshold on the Aerosol Retrieval Confidence Index (ARCI) was adjusted to 0.18; the equivalent ARCI threshold in the FIRSTLOOK and standard processing is 0.15. Information about this product and details about product quality are available on the MISR Data and Information Table.

Back to top

MLS NRT

In order to generate data products within 3 hours of observation time, a number of changes have been made to standard processing approach to expedite the availability of input data sets.

For detailed information on MLS NRT data product quality, please see the user guide.

The MLS NRT algorithm uses a simplified fast forward model to meet NRT data latency requirements and are therefore not as accurate as the retrievals that constitute the standard MLS products. Nevertheless the results are scientifically useful in selected regions of the earth's atmosphere provided that the data are screened according to the recommendations in the MLS NRT user guide.

The NRT data are typically available within 3 hours of observation and are broken into files containing about 15 minutes of data. The 7 most recent days of MLS NRT data are kept online for users to download. After this time users should download the standard, full production Level-2 data products.

Back to top

MODIS NRT

In order to generate data products within 3 hours of observation time, a number of changes have been made to standard processing approach to expedite the availability of input data sets.

Near Real-Time Approach

  • The delivery of the Level 0 (L0) is expedited from the EOS Data and Operations System (EDOS). NRT processing acquires L0 files at the end of each downlink session (session-based files) within 10-30 minutes of real-time, whereas standard forward processing acquires 2-hour L0 files for Aqua and Terra within 7-8 hours of real-time.
  • For geolocation MODIS Terra and MODIS Aqua differ in their approach. For MODIS Terra the attitude and ephemeris data entrained in the L0 data are used rather than the flight data (FD) product for standard processing. The Aqua sensor (unlike Terra) lacks the ability to generate definitive ephemeris and attitude data, and therefore, NASA's Flight Dynamics Facility (FDF) calculates predicted satellite ephemeris data and uploads it to the spacecraft once per day. This data is later returned to the ground stations as Ground-Based Attitude Determination (GBAD) data, which is also time-coordinated with the other instruments. This GBAD ephemeris and attitude data is used to process Aqua MODIS standard products.
  • Most NRT Product Generation Executive (PGE) codes use the same science algorithm used in the standard processing. However, certain Level 2 (L2) codes have modified production rules to relax the requirements for ancillary data products.
  • For Level 3, rolling products such as MOD141N/Q1N, MxD09A1N, MCD43A1N/A2/A2N download this PDF to explain the differences between the NRT and standard "n-day" products.
Image
image
In this side-by-side comparison of a standard and near-real time Land Surface Reflectance granule over the Midwest there appears to be no difference between the products. However, under close examination the near-real time view shows slightly more haze West of the Great Lakes.

To meet the latency requirement, most L2 NRT Atmosphere PGEs (Cloud Mask/Profile, Cloud, and Aerosol) are allowed to use ancillary input products (e.g. GDAS, NISE, OZONE, TOAST) from an earlier time period, and sometime even several days prior to the current data day while the operational processing could be delayed until the best ancillary data is available. Use of ancillaries from different data days can result in significant difference in the daily cloud and aerosol product generated at NRT compared to the operational products from MODAPS. The L2 Fire, Snow, and Sea Ice from NRT are nearly identical to the operational product as these PGEs do not make use of ancillary data products. A special version of the L2 Land Surface Reflectance for near real-time use was developed by the Principal Investigator (PI); this code uses the National Oceanic and Atmospheric Administration Global Forecast System (GFS) ancillary product as opposed to Global Data Assimilation System (GDAS) used in the standard processing version. Difference in L2 surface reflectance product from this change to processing code in NRT processing isn’t significant.

Image
image
MODIS Cloud top tempurature over the Midwest. Thin clouds over Lake Superior show sensitivity to GDAS ancillary data.

The near real-time PGE codes were extensively science tested and all products were compared to the baseline products generated by the standard processing. This comparison was made by the atmosphere and land Quality Assessment (QA) (Land Data Operational Products Evaluation [LDOPE]) staff and by the PIs. All PGEs have been validated for use in LANCE-MODIS by the PIs. This process will be repeated as new versions of the PGEs become available and are considered for inclusion in LANCE-MODIS.

Side by Side Comparison

Two examples of side-by-side comparisons of the standard and near real-time products are shown here. The first shows a Land Surface Reflectance granule over the Midwest. There appears to be no difference between the products. However, under close examination the near real-time view shows slightly more haze West of the Great Lakes.

In contrast, the second side-by-side comparison is for Cloud Top Temperature for the same granule and there are very obvious differences in the region West of the Great Lakes. This is as a result of the sensitivity of this product to the GDAS ancillary data.

Back to top

MOPITT NRT

The MOPITT NRT Product (NRT) differs from the Standard Product (SP) in the following ways:

  • The SP uses NASA's GMAO MERRA-2 reanalysis data to estimate the meteorological conditions, such as water vapor and temperature profiles. The NRT uses MERRA-2 forecasts.
  • The SP uses MODIS cloud masks to filter out cloudy pixels. The NRT uses MOPITT thermal channels compared with the expected radiance from surface in order to discriminate clear from cloudy pixels.
  • SP products are daily files, whereas NRT files are released as soon as they are processed. The granule size varies depending on the input files that are received.
  • SP are inspected for QA to ensure consistency with validated granules. NRT files undergo automatic QA which will alert the processing team if there are any anomalies.
  • Level 2 SP files are further processing to create daily and monthly Level 3 products. NRT files remain at Level 2.
  • SP are regularly delivered to the ASDC for archive and distribution. NRT products are not archived and are distributed from NCAR servers.
OMI NRT

In order to generate data products within 3 hours of observation time, a number of changes have been made to standard processing approach to expedite the availability of input data sets.

The quality of OMI NRT Level 2 products compares favorably to the standard products. There appears to be some variances at high latitudes with high solar zenith angles. In the following table we display the maximum differences for a single day and averaged over a week.

Product Variable Daily Maximum Percentage Difference Weekly Average Maximum Percentage Difference
OMTO3 Total Ozone Column 2.64% 1.40%
OMCLDRR Cloud Fraction 6.02% 1.42%
  Cloud Pressure 2.82% 0.67%
OMAERUV AOD (388 nm) 5.95% 2.31%

Back to top

OMPS NRT

In order to generate data products within 3 hours of observation time, a number of changes have been made to standard processing approach to expedite the availability of input data sets.

The quality of OMPS NRT Level 2 products compares favorably to the standard products. There appears to be some variances at high latitudes with high solar zenith angles. In the following table we display the maximum differences for a single day and averaged over a week.

NMTO3-L2 and NPBUVO3-L2

The science team has not found any large difference in the quality of the products, a more detailed analysis will be provided in the near future.

A comparison between Near Real-Time SNPP/OMPS PCA SO2 product and the Standard SNPP/OMPS PCA SO2 product

In order to generate data products within 3 hours of observation time, a number of changes have been made to standard processing approach to expedite the availability of input data sets for the SNPP/OMPS principal component analysis (PCA) SO2 product (NMSO2-PCA-L2).

In general, the near real-time (NRT) SNPP/OMPS PCA SO2 product is very similar to the standard processing product. On average, the differences between the NRT and standard SO2 products are 0.03 DU (Dobson Units) or less. Note that under most circumstances, the actual SO2 loading in the atmosphere is very small, and the percentage difference would be significant even for a very small absolute difference. There are stripes (or rows) with relative large differences between the NRT and the standard products, especially for high latitudes or the edge of the swath. This is likely due to the setup of the PCA retrieval algorithm, in which principal components (PCs) are extracted from the input radiance data for each cross-track position (row) of the each OMPS orbit. The PCs are then used in spectral fitting for all pixels in the same row. Differences in the input radiance and other input data may lead to different PCs and also different number of PCs that are actually used in the spectral fitting, resulting in relatively large differences for some rows.

The table below summarizes the differences between NRT and standard SNPP/OMPS PCA SO2 products.

Parameter

Max Difference (DU)*

Mean Difference (DU)**

SO2_PBL (total column amount assuming all SO2 near the surface)

1.3

0.027

SO2_TRL (total column amount assuming 3-km SO2 plume height)

1.2

0.022

SO2_TRM (total column amount assuming 8-km SO2 plume height)

0.6

0.013

SO2_TRU (total column amount assuming 13-km SO2 plume height)

0.4

<0.01

SO2_STL (total column amount assuming 18-km SO2 plume height)

0.3

<0.01


*The maximum of the absolute difference between NRT and standard products for one day (10/09/2017).
*The mean of the absolute difference between NRT and standard products for one day (10/09/2017).

VIIRS Land NRT

In order to generate data products within 3 hours of observation time, a number of changes have been made to standard processing approach to expedite the availability of input data sets.

  • NRT processing uses session based L0 files acquired at the end of each downlink session within 10-30 minutes of real-time, delivered by the EOS Data and Operations System (EDOS), whereas standard forward processing acquires 2-hour L0 files (time based) within 7-8 hours of real-time.
  • For geolocation NRT uses definitive orbit information similar to the operational processing. However there could be still minor difference in the geolocation due to difference in certain other ancillary products used by the geolocation processing such as polar wander files. Error introduced by use of such difference ancillaries is minor, but in some situations this position error may be large (several kilometers). In particular, the larger errors may occur after spacecraft maneuvers. Impacted time period may vary from ~30 minutes in case of lunar calibration roll maneuver to ~ 2hrs in case of an inclination maneuver. In these cases users are advised to use the standard product generated using the best available satellite position data and after the quality assurance team has reviewed and removed data degraded by spacecraft maneuver.
  • L1B and most L2 products use the same algorithm and LUTs as used by the operational processing except for difference in some of ancillary. For example Land cloud mask processing might use NISE data from a prior data day if the data is not available for the current day processing resulting in minor difference in land cloud mask product and downstreams. Similarly there could be significant difference in retrieved LST (could be as large as 5 deg K) from use of NCEP at NRT as opposed to the MERRA2 used by the standard processing for water vapor input.
  • For Level 3, rolling products download this PDF to explain the differences between the NRT and standard "n-day" products.
VIIRS Atmosphere NRT

In order to generate near real-time (NRT) VIIRS Atmosphere products within 3 hours of observation time, several changes have been made to the standard processing approach. These differences are outlined as follows.

NRT processing utilizes session-based L0 files ingested by the Atmosphere SIPS from the EOS Data and Operations System (EDOS) available within 1 – 2 hours of real-time. Standard forward stream processing uses time-based L0 files ingested within 5 - 7 hours of real-time.

  • For geolocation, NRT uses definitive orbit information similar to the operational processing. However, there could be still minor difference in the geolocation due to difference in ancillary products used by the geolocation processing such as polar wander files or the geolocation LUT files. Error introduced by use of such difference ancillaries is typically minor, but in some situations this position error may be large (several kilometers). Larger errors may occur after spacecraft maneuvers. Impacted time period may vary from ~30 minutes in case of lunar calibration roll maneuver to ~ 2hrs in case of an inclination maneuver. In these cases users are advised to use the standard product generated using the best available satellite position data and after the quality assurance team has reviewed and removed data degraded by spacecraft maneuver.
     
  • The Standard Forward Stream Processing Mode creates the best possible global VIIRS Level 1B and Atmosphere products using the highest fidelity input data sources, including (but not limited to) retransmitted Level 0 data from EDOS; calibration lookup tables (LUTs) from offline analysis by the NASA VIIRS Characterization Support Team; and final versions of ancillary data from external providers (e.g. LAADS, UCAR, NCDC, etc.). The philosophy for these products is to wait until the best input data are available, and then create the best possible VIIRS Level 1B, VIIRS Atmosphere Level 2, and Level 3 products.
     
  • The Near Real-Time Processing Mode creates low latency global VIIRS Level 1B and Atmosphere Level 2 products, within the constraints imposed by the time delay in receiving the session-based VIIRS L0 files. The philosophy for the NRT products is to provide rapid turnaround to Science Team members and other investigators who require the VIIRS Atmosphere products in near real-time. The VIIRS L1B/L2 products are created using the same or similar algorithms used to create the Standard Forward Stream Mode products, however the calibration LUTs and ancillary data available at the time the Level 0 data are received are used. For NRT production, ancillary input products (e.g. NISE) from an earlier time period, and sometime even several days prior, may be used, while standard mode operational processing may be delayed until the best ancillary data is available.
    • For example, the VIIRS MVCM Cloud Mask NRT product uses the National Oceanic and Atmospheric Administration Global Forecast System (GFS) ancillary product as opposed to the Global Data Assimilation System (GDAS) product used in the standard processing version.
    • The VIIRS Deep Blue Aerosol NRT product utilizes the Goddard Earth Observing System (GEOS-5) model forecasts instead of the reanalysis product. In addition, the Deep Blue product implements a different QA flag (QA flag Ocean = 2) for the NRT product associated with the effects of GEOS-5 wind speed on AOD retrievals over oceans in regions near sun glint.

EOSDIS manages discipline-specific DAACs. Each DAAC processes, archives and distributes EOS science data based on discipline. They serve the user community by providing science quality products from past and present EOS missions. Other services include:

  • Help with obtaining and selecting data
  • Data-handling and visualization tools
  • Notification of data-related news
  • Technical support and referrals
Last Updated