Skip to main content

This tutorial demonstrates how to leverage the Application for Extracting and Exploring Analysis Ready Samples (AppEEARS) Application Programming Interface (API) quality service to decode the quality data associated with NASA's Land Processes Distributed Active Archive Center (LP DAAC) products. Presented as a Jupyter Notebook, users can follow the step-by-step instructions to learn how to access and use the quality web service for decoding Moderate Resolution Imaging Spectroradiometer (MODIS) data products from the LP DAAC data archive.

Background

Quality information for LP DAAC data products is important to consider when determining the usability and usefulness of a data product for a particular science application. However, this information has been notoriously difficult for users to access. Quality information is often stored as an integer value that requires the user to decode into binary strings. In order to interpret the binary string, users must map the unique combinations of bits found in separate subsets of the binary string (i.e. bit-fields) to quality tables that characterize the particular quality attribute associated with each bit-field. This process is difficult for most users and is often passed over.

NASA's LP DAAC has added a quality web service to its AppEEARS service API to alleviate the difficult process of decoding and interpreting quality layers for tiled MODIS and Web-Enabled Landsat Data (WELD) products. The quality web service will decode an integer value for a data product/layer and return the quality attribute information associated with that particular integer value. Additionally, users can determine if a quality layer exists for a particular data product/layer as well as get details about individual quality layers.

Required Packages

Notes

  • Users of this tutorial will need internet access in order to successfully execute the python code.
  • This is a Python-specific tutorial and has been tested in Python 3.6 and 2.7.

Get Started

In [1]:
%autosave 0

Autosave disabled

Let's start by importing the required packages.

In [2]:
import requests
import pandas as pd
 

Next, assign the AppEEARS services API URL to a variable

In [3]:
SERVICES_URL = 'https://lpdaacsvc.cr.usgs.gov/services/appeears-api/'
 

Note: The AppEEARS services API page contains all of the AppEEARS services accompanied by their documentation.

Get All of the Data Layers that Contain Associated Quality Layers

To know what layers have a quality layer associated with them, users can get a print out of the number of layers with associated quality layers.

In [4]:
qualityLayers = requests.get('{}/quality?format=json'.format(SERVICES_URL)).json()
#print(qualityLayers)

'Quality information was found for {0} data layers!'.format(len(qualityLayers))
Out[4]:
'Quality information was found for 1227 data layers!'
 

The request returns a list of dictionaries that can be indexed to filter the information returned.

In [5]:
qualityLayers[218:222]
Out[5]:
[{'Layer': 'Nadir_Reflectance_Band2',
  'ProductAndVersion': 'MCD43A4.006',
  'QualityLayers': ['BRDF_Albedo_Band_Mandatory_Quality_Band2'],
  'QualityProductAndVersion': 'MCD43A4.006'},
 {'Layer': 'Nadir_Reflectance_Band3',
  'ProductAndVersion': 'MCD43A4.006',
  'QualityLayers': ['BRDF_Albedo_Band_Mandatory_Quality_Band3'],
  'QualityProductAndVersion': 'MCD43A4.006'},
 {'Layer': 'Nadir_Reflectance_Band4',
  'ProductAndVersion': 'MCD43A4.006',
  'QualityLayers': ['BRDF_Albedo_Band_Mandatory_Quality_Band4'],
  'QualityProductAndVersion': 'MCD43A4.006'},
 {'Layer': 'Nadir_Reflectance_Band5',
  'ProductAndVersion': 'MCD43A4.006',
  'QualityLayers': ['BRDF_Albedo_Band_Mandatory_Quality_Band5'],
  'QualityProductAndVersion': 'MCD43A4.006'}]

The list of products contained in qualityLayers is long, and trying to find the product/layer of interest via indexing can be inefficient. Instead, let's use a python expression, known as a list comprehension, to get all of the layers in MYD13A2.006 that have quality layers associated with them.

In [6]:
productQuality_A = [q for q in qualityLayers if q['ProductAndVersion'] == 'MYD13A2.006']

productQuality_A
Out[6]:
[{'Layer': '_1_km_16_days_EVI',
  'ProductAndVersion': 'MYD13A2.006',
  'QualityLayers': ['_1_km_16_days_VI_Quality'],
  'QualityProductAndVersion': 'MYD13A2.006'},
 {'Layer': '_1_km_16_days_NDVI',
  'ProductAndVersion': 'MYD13A2.006',
  'QualityLayers': ['_1_km_16_days_VI_Quality'],
  'QualityProductAndVersion': 'MYD13A2.006'},
 {'Layer': '_1_km_16_days_blue_reflectance',
  'ProductAndVersion': 'MYD13A2.006',
  'QualityLayers': ['_1_km_16_days_pixel_reliability'],
  'QualityProductAndVersion': 'MYD13A2.006'},
 {'Layer': '_1_km_16_days_MIR_reflectance',
  'ProductAndVersion': 'MYD13A2.006',
  'QualityLayers': ['_1_km_16_days_pixel_reliability'],
  'QualityProductAndVersion': 'MYD13A2.006'},
 {'Layer': '_1_km_16_days_NIR_reflectance',
  'ProductAndVersion': 'MYD13A2.006',
  'QualityLayers': ['_1_km_16_days_pixel_reliability'],
  'QualityProductAndVersion': 'MYD13A2.006'},
 {'Layer': '_1_km_16_days_red_reflectance',
  'ProductAndVersion': 'MYD13A2.006',
  'QualityLayers': ['_1_km_16_days_pixel_reliability'],
  'QualityProductAndVersion': 'MYD13A2.006'}]

If users do not want to use indexing or list comprehensions (the previous two steps), they may use the product parameter in the quality service URL to get all of the layers in MYD13A2.006 that have associated quality layers.

In [7]:
productQuality_B = requests.get('{}/quality/MYD13A2.006?format=json'.format(SERVICES_URL)).json()

productQuality_B
Out[7]:
[{'Layer': '_1_km_16_days_EVI',
  'ProductAndVersion': 'MYD13A2.006',
  'QualityLayers': ['_1_km_16_days_VI_Quality'],
  'QualityProductAndVersion': 'MYD13A2.006'},
 {'Layer': '_1_km_16_days_NDVI',
  'ProductAndVersion': 'MYD13A2.006',
  'QualityLayers': ['_1_km_16_days_VI_Quality'],
  'QualityProductAndVersion': 'MYD13A2.006'},
 {'Layer': '_1_km_16_days_blue_reflectance',
  'ProductAndVersion': 'MYD13A2.006',
  'QualityLayers': ['_1_km_16_days_pixel_reliability'],
  'QualityProductAndVersion': 'MYD13A2.006'},
 {'Layer': '_1_km_16_days_MIR_reflectance',
  'ProductAndVersion': 'MYD13A2.006',
  'QualityLayers': ['_1_km_16_days_pixel_reliability'],
  'QualityProductAndVersion': 'MYD13A2.006'},
 {'Layer': '_1_km_16_days_NIR_reflectance',
  'ProductAndVersion': 'MYD13A2.006',
  'QualityLayers': ['_1_km_16_days_pixel_reliability'],
  'QualityProductAndVersion': 'MYD13A2.006'},
 {'Layer': '_1_km_16_days_red_reflectance',
  'ProductAndVersion': 'MYD13A2.006',
  'QualityLayers': ['_1_km_16_days_pixel_reliability'],
  'QualityProductAndVersion': 'MYD13A2.006'}]
In [8]:
productQuality_B[0]['QualityLayers'][0]
Out[8]:
'_1_km_16_days_VI_Quality'

Note: In both cases above, EVI (_1_km_16_days_EVI) and NDVI (_1_km_16_days_NDVI) have the same quality layer (_1_km_16_days_VI_Quality). This is the case for many of the MODIS products. However, there are occasions when the layers within products have their own associated quality layers (e.g., MOD/MYD11 products).

Get the Quality Layer Description for MYD13A2.006

Information within quality layers is often stored as bit-packed integers. This means that in order to get the actual quality information, one must convert the integer to binary and parse the binary string into bit-fields. Bit-fields are distinct combinations of bits representing quality categories that give an indication as to the usability and usefulness of the data product. Let's get the quality layer description for MYD13A2.006 by adding the quality layer parameter to the quality service URL.

In [9]:
qualityLayerInfo = requests.get('{}/quality/MYD13A2.006/_1_km_16_days_VI_Quality?format=json'.format(SERVICES_URL)).json()

#qualityLayerInfo
pd.DataFrame(qualityLayerInfo)
Out[9]:
 AcceptableDescriptionNameProductAndVersionQualityLayerValue
0TrueVI produced, good qualityMODLANDMYD13A2.006_1_km_16_days_VI_Quality0
1FalseVI produced, but check other QAMODLANDMYD13A2.006_1_km_16_days_VI_Quality1
2FalsePixel produced, but most probably cloudyMODLANDMYD13A2.006_1_km_16_days_VI_Quality2
3FalsePixel not produced due to other reasons than c...MODLANDMYD13A2.006_1_km_16_days_VI_Quality3
4NoneHighest qualityVI UsefulnessMYD13A2.006_1_km_16_days_VI_Quality0
5NoneLower qualityVI UsefulnessMYD13A2.006_1_km_16_days_VI_Quality1
6NoneDecreasing quality (0010)VI UsefulnessMYD13A2.006_1_km_16_days_VI_Quality2
7NoneDecreasing quality (0011)VI UsefulnessMYD13A2.006_1_km_16_days_VI_Quality3
8NoneDecreasing quality (0100)VI UsefulnessMYD13A2.006_1_km_16_days_VI_Quality4
9NoneDecreasing quality (0101)VI UsefulnessMYD13A2.006_1_km_16_days_VI_Quality5
10NoneDecreasing quality (0110)VI UsefulnessMYD13A2.006_1_km_16_days_VI_Quality6
11NoneDecreasing quality (0111)VI UsefulnessMYD13A2.006_1_km_16_days_VI_Quality7
12NoneDecreasing quality (1000)VI UsefulnessMYD13A2.006_1_km_16_days_VI_Quality8
13NoneDecreasing quality (1001)VI UsefulnessMYD13A2.006_1_km_16_days_VI_Quality9
14NoneDecreasing quality (1010)VI UsefulnessMYD13A2.006_1_km_16_days_VI_Quality10
15NoneDecreasing quality (1011)VI UsefulnessMYD13A2.006_1_km_16_days_VI_Quality11
16NoneLowest qualityVI UsefulnessMYD13A2.006_1_km_16_days_VI_Quality12
17NoneQuality so low that it is not usefulVI UsefulnessMYD13A2.006_1_km_16_days_VI_Quality13
18NoneL1B data faultyVI UsefulnessMYD13A2.006_1_km_16_days_VI_Quality14
19NoneNot useful for any other reason/not processedVI UsefulnessMYD13A2.006_1_km_16_days_VI_Quality15
20NoneClimatologyAerosol QuantityMYD13A2.006_1_km_16_days_VI_Quality0
21NoneLowAerosol QuantityMYD13A2.006_1_km_16_days_VI_Quality1
22NoneAverageAerosol QuantityMYD13A2.006_1_km_16_days_VI_Quality2
23NoneHighAerosol QuantityMYD13A2.006_1_km_16_days_VI_Quality3
24NoneNoAdjacent cloud detectedMYD13A2.006_1_km_16_days_VI_Quality0
25NoneYesAdjacent cloud detectedMYD13A2.006_1_km_16_days_VI_Quality1
26NoneNoAtmosphere BRDF CorrectionMYD13A2.006_1_km_16_days_VI_Quality0
27NoneYesAtmosphere BRDF CorrectionMYD13A2.006_1_km_16_days_VI_Quality1
28NoneNoMixed CloudsMYD13A2.006_1_km_16_days_VI_Quality0
29NoneYesMixed CloudsMYD13A2.006_1_km_16_days_VI_Quality1
30NoneShallow oceanLand/Water MaskMYD13A2.006_1_km_16_days_VI_Quality0
31NoneLand (Nothing else but land)Land/Water MaskMYD13A2.006_1_km_16_days_VI_Quality1
32NoneOcean coastlines and lake shorelinesLand/Water MaskMYD13A2.006_1_km_16_days_VI_Quality2
33NoneShallow inland waterLand/Water MaskMYD13A2.006_1_km_16_days_VI_Quality3
34NoneEphemeral waterLand/Water MaskMYD13A2.006_1_km_16_days_VI_Quality4
35NoneDeep inland waterLand/Water MaskMYD13A2.006_1_km_16_days_VI_Quality5
36NoneModerate or continental oceanLand/Water MaskMYD13A2.006_1_km_16_days_VI_Quality6
37NoneDeep oceanLand/Water MaskMYD13A2.006_1_km_16_days_VI_Quality7
38NoneNoPossible snow/iceMYD13A2.006_1_km_16_days_VI_Quality0
39NoneYesPossible snow/iceMYD13A2.006_1_km_16_days_VI_Quality1
40NoneNoPossible shadowMYD13A2.006_1_km_16_days_VI_Quality0
41NoneYesPossible shadowMYD13A2.006_1_km_16_days_VI_Quality1

Note: The table above contains the description for each bit-field in MYD13A2.006's quality layer (i.e., _1_km_16_days_VI_Quality). For more information of quality layer descriptions, please consult with the product pages.

Decode Quality Value

Say that we have extracted a pixel value from the EVI layer (i.e., _1_km_16_days_EVI) of product MYD13A2.006. We can use the quality service to decode the associated quality value for the pixel by inserting the pixel value from the quality layer (_1_km_16_days_VI_Quality) in the quality service URL. We see that our pixel value from the quality layer equals 4160 and that our EVI value equals 0.3091.

In [10]:
qualityIntDecoder = requests.get('{}/quality/MYD13A2.006/_1_km_16_days_VI_Quality/4160?format=json'.format(SERVICES_URL)).json()

qualityIntDecoder
Out[10]:
{'Adjacent cloud detected': {'bits': '0b0', 'description': 'No'},
 'Aerosol Quantity': {'bits': '0b01', 'description': 'Low'},
 'Atmosphere BRDF Correction': {'bits': '0b0', 'description': 'No'},
 'Binary Representation': '0b0001000001000000',
 'Land/Water Mask': {'bits': '0b010',
  'description': 'Ocean coastlines and lake shorelines'},
 'MODLAND': {'bits': '0b00', 'description': 'VI produced, good quality'},
 'Mixed Clouds': {'bits': '0b0', 'description': 'No'},
 'Possible shadow': {'bits': '0b0', 'description': 'No'},
 'Possible snow/ice': {'bits': '0b0', 'description': 'No'},
 'VI Usefulness': {'bits': '0b0000', 'description': 'Highest quality'}}
 

Display qualityIntDecoder as Pandas dataframe

In [11]:
pd.DataFrame(qualityIntDecoder)
Out[11]:
 Binary RepresentationMODLANDVI UsefulnessAerosol QuantityAdjacent cloud detectedAtmosphere BRDF CorrectionMixed CloudsLand/Water MaskPossible snow/icePossible shadow
bits0b00010000010000000b000b00000b010b00b00b00b0100b00b0
description0b0001000001000000VI produced, good qualityHighest qualityLowNoNoNoOcean coastlines and lake shorelinesNoNo

The above code snippet returns the decoded quality information for value 4160, however, the order in which the quality categories are displayed does not match the order in which the bit-fields are assembled in the binary string. To correct this, the same service call can be used, but the manner in which we bring the request into python changes.

In [12]:
import json
from collections import OrderedDict
qualityIntDecoder = json.loads(
    requests.get('{}/quality/MYD13A2.006/_1_km_16_days_VI_Quality/4160?format=json'.format(SERVICES_URL)).text, 
    object_pairs_hook=OrderedDict)
 

Now let's print out the decoded quality value in its proper order.

In [13]:
#qualityIntDecoder
pd.DataFrame(qualityIntDecoder)
Out[13]:
 Binary RepresentationMODLANDVI UsefulnessAerosol QuantityAdjacent cloud detectedAtmosphere BRDF CorrectionMixed CloudsLand/Water MaskPossible snow/icePossible shadow
bits0b00010000010000000b000b00000b010b00b00b00b0100b00b0
description0b0001000001000000VI produced, good qualityHighest qualityLowNoNoNoOcean coastlines and lake shorelinesNoNo

We can see in the MODLAND column that this is a good quality pixel and the Land/Water Flag column indicates that this pixel is likely near a lake shoreline (given that the pixel extracted was from tile h10v04).

Finally, we can take the decoded python object (a python dictionary) and filter it based on the bit-field description.

In [14]:
qualityIntDecoder['MODLAND']
Out[14]:
OrderedDict([('bits', '0b00'), ('description', 'VI produced, good quality')])

Relevant Data and Tools

ProductLong Name
MOD13A1.006MODIS/Terra Vegetation Indices 16-Day L3 Global 500 m SIN Grid
MYD13A2.006MODIS/Aqua Vegetation Indices 16-Day L3 Global 1 km SIN Grid
AppEEARSThe Application for Extracting and Exploring Analysis Ready Samples
Jupyter
Python tutorial showing how to work with quality data associated with LP DAAC products (Notebook)
 

Details

Last Updated

Published

Data Archive

Land Processes DAAC (LP DAAC)