Quick Guide to MODLAND QA

Obtaining Data From The Goddard Space Flight Center (GSFC) By The EOS Core System (ECS) Subscription Service

RICHARD H. Buss Jr., A.K. Sharma, and Greg Leptoukh
(GSFC MODIS Support)
NASA GODDARD SPACE FLIGHT CENTER Building 32, Code 902 Greenbelt MD 20771
12 April 1999


This information is provisional and under construction. A link will be placed here to the GSFC DAAC web subscription page when it is available.

See Step 2 (below) for the MODIS products available at GSFC DAAC.


OVERVIEW

To obtain any data from the ECS, you must first become a registered user to the system at GSFC. Then, you need to determine the names of the Earth Science Data Types (ESDTs) that you would like to receive. You can automatically receive new data granules of these data types by subscribing to the data. Note that subscription is mainly for new data coming into the archive; old data will have to be obtained by means other than subscription: search and order. This document describes only the subscription method. You will obtain from the GSFC a request form to subscribe to the desired data type. For each data type, complete a subscription request form and email it to GSFC Distributed Active Archive (DAAC). A properly formatted subscription request that is entered into ECS will result in the data granules being sent to you, either electronically (currently available) or by 8mm tape through postal service (not yet available). To receive data granule files electronically, you must have set the permissions and adequate disk space on your local machine so that your machine can actually save what ECS sends to you. When the data arrives in the ECS system, it will look at your subscription and your user registration and automatically send you the all data granules of the type that you requested. An anonymous FTP account would be preferable so that password security is not an issue. Note that many data product files are several hundred megabytes large in hierarchical data format (HDF). A smaller metadata file (MET) will usually accompany each data file. You will receive an email message for each granule in ECS that satisfies your subscription, and after a successful FTP delivery of your data, you will receive an email message notifying you of the delivery. Example messages from ECS are listed below in this document.

SUMMARY

For now, during the pre-launch testing, forms will be passed back and forth by email plain text files. (It is planned for post-launch operations to have web entry forms available for you to register and to subscribe to ECS.) For now, however, follow the following steps (see details below):

1) Register with ECS DAAC User services by e-mailing a completed request form

2) Select one or more data types (ESDTs) that you would like to receive

3) Create your directory(ies) with proper write permissions and size to hold data

4) For each ESDT, complete and send a Subscription Request Form to the GSFC DAAC

5) If needed, phone GSFC DAAC to give the password of your receiving account (Linda McNeely at 301-614-5290)

Step 1: ECS User Registration form

Complete the template form for subscription and email the text file (NOTE the mandatory subject line) to:

To: mcneely@daac.gsfc.nasa.gov

Subject: ECS Registration Request

(Mandatory entries are preceded by a * and valid options for some fields are listed between angle brackets <>. )

----------

Your contact address:

-------------

Title: <(none)Dr. Mr. Ms.>

*First name:

Initial:

*Last name:

Organization:

*Internet email address:

Address:

*City:

*State/province:

*Zip/postal code:

*Country:

*Phone:

Fax:

----------------

Shipping address (for orders): <Given below... Same as contact address (above)>

Title: <(none)Dr. Mr. Ms.>

First name:

Initial:

Last name:

Organization:

Internet email address:

Address:

City:

State/province:

Zip/postal code:

Country:

Phone:

Fax:

----

Billing address (for orders): <Given below...Same as contact address (above) Same as shipping address (above)>

Title: <(none)Dr.Mr.Ms.>

First name:

Initial:

Last name:

Organization:

Internet email address:

Address:

City:

State/province:

Zip/postal code:

Country:

Phone:

Fax:

----------

Your affiliation:

*Type: <Government K-12 University Commercial Other>

*Category: <USA NON_USA>

Data Access Key: (The password that you want to use for access to ECS - don't type it in - just phone it in to DAAC user services, Linda McNeely at 301-614-5290. Please remember your access key for your future logging on to ECS).

Step 2. ESDTs

Here are the ESDTs for the granules we now have available at GSFC DAAC:

ESDT Data Type, Brief Description, Approximate File Size (Mbytes)

MOD01, L1A Raw Counts, 564

MOD03, Geolocation, 61

MOD021KM, L1B 1-km calibrated radiances, 335

MOD02HKM, L1B 500-m calibrated radiances, 253

MOD02QKM, L1B 250-m calibrated radiances, 264

MOD02OBC, L1B housekeeping data, 57

MOD07_L2, Atmospheric profiles [O3, H20], 28

MOD35_L2, Cloud mask, 48

MODVOLC, Volcano alert, 0.002

[You can subscribe to the Insertion, Deletion, or Updating of these data.]

Step 3. Disk Space Allocation

Please note that the large size of some of the listed data types might require a large allocation of disk space or timely disk management/backup at your site in order to avoid system failures of both your site and of ECS. There are approximately 288 granules per day per ESDT, except for the MODVOLC Volcano alerts, which happen only occasionally. With this production information, you can plan your subscriptions and resources accordingly.

Step 4. ECS Subscription Request form (NOTE required for each ESDT)

Complete the template form for subscription and email the text file (NOTE the mandatory subject line) to:

To: mcneely@daac.gsfc.nasa.gov

Subject: ECS Subscription Request

[BLANK TEMPLATE]

begin SubscriptionRequest

REQUEST_TYPE:

ECS_ACCOUNT_NAME:

RETURN_EMAIL_ADDRESS:

ECS_DATA_PRODUCT:

EVENT:

START_DATE_of_SUBSCRIPTION:

EXPIRATION_DATE_of_SUBSCRIPTION:

MESSAGE_of_SUBSCRIPTION:

DATA_ACQUIRE_METHOD:

local_user_name:

local_password:

local_host:

local_directory:

metadata_qualifier:

ECS_subscription_ID_Number:

end SubscriptionRequest

[FORMAT SPECIFIED TEMPLATE]

begin SubscriptionRequest

REQUEST_TYPE: <ADD or DELETE or REVISE>

ECS_ACCOUNT_NAME: <name>

RETURN_EMAIL_ADDRESS: <user@machine.domain>

ECS_DATA_PRODUCT: <ESDTname.version>

EVENT: <INSERT or UPDATEMETADATA or DELETE>

START_DATE_of_SUBSCRIPTION: <Month Day Year: 01/31/2000 >

EXPIRATION_DATE_of_SUBSCRIPTION: <Month Day Year: 01/31/2001 >

MESSAGE_of_SUBSCRIPTION: <Text on one line, specified by requester>

DATA_ACQUIRE_METHOD: <FTP or 8mm or NO>

local_user_name: <YOUR COMPUTER ACCOUNT NAME WHERE DATA ARRIVES or N/A>

local_password: <YOURS---Anonymous FTP preferred: NONSECURE TRANSMISSION! or N/A >

local_host: <IP ADDRESS or DOMAIN NAMES or N/A>

local_directory: <path/directory or N/A>

metadata_qualifier: <N/A or <TYPE><BOOLEAN RELATION><VALUE>>

ECS_subscription_ID_Number: <N/A or existing subscription NUMBER>

end SubscriptionRequest

Explanation of ECS Subscription Template

(Revised March 9 1999 by Richard H. Buss Jr.at the GSFC )

IMPORTANT: To subscribe to ECS data with the template, you must first be a registered ECS user or become one by contacting DAAC user services at daacuso@daac.gsfc.nasa.gov.

INTRODUCTION

Subscription is an ECS capability that allows you to be notified of changes, additions, and deletions to the ECS data and metadata. Subscription also provides the means for you to receive the data and metadata. To subscribe, you fill out a Request, currently by filling in an email template and then sending the request to the DAAC. Eventually, you will be able to request a subscription by using a web form. You can include multiple Subscription Requests in the same email message. The capitalized FIELDS in the beginning of the template are mandatory for you to fill; the lowercase fields are optional but are SOMETIMES necessary if you want to ACQUIRE the data or if you want to qualify your data by metadata values that are associated with your data, or if you revise/delete an existing subscription with a known ID Number.

EXAMPLES:

The following is an example of a completed Subscription Request email message containing two Subscription Requests: one for adding a new subscription, and one for revising an existing subscription. N/A is for Not Applicable fields.

Subscription example 1:

Sends by FTP your specified data granules, unqualified by metadata, and for each data event, also notifies you by e-mailing two email notices : one for the original INSERT of L1B data into ECS, and one for FTP of the data to your localhost directory.

begin SubscriptionRequest

REQUEST_TYPE: ADD

ECS_ACCOUNT_NAME: rbuss

RETURN_EMAIL_ADDRESS: richard.buss@gsfc.nasa.gov

ECS_DATA_PRODUCT: MOD02HKM.001

EVENT: INSERT

START_DATE_of_SUBSCRIPTION: 12 01 1998

EXPIRATION_DATE_of_SUBSCRIPTION: 01 01 2001

MESSAGE_of_SUBSCRIPTION: This Notice is for an Insertion of L1B product MOD02HKM with FTP

DATA_ACQUIRE_METHOD: FTP

local_user_name: rbuss

local_password: xxxxxx

local_host: gsfcsparc6.gsfcmo.ecs.nasa.gov

local_directory: /home/rbuss/FTP/PGE02_QA

metadata_qualifier: N/A

ECS_subscription_ID_Number: N/A

end SubscriptionRequest

Subscription Example 2:

This revised subscription already existed in ECS, which e-mailed previously to you a notice containing your Subscription ID number. You decided to change the subscription by requesting REVISE. This revised subscription notifies you when daytime geolocation granules have had their metadata UPDATED. Note that this Subscription sends only email notices to you. It will continue to notify you but does NOT send the data itself to you, as you requested it not to.

begin SubscriptionRequest

REQUEST_TYPE: REVISE

ECS_ACCOUNT_NAME: rbuss

RETURN_EMAIL ADDRESS: richard.buss@gsfc.nasa.gov

ECS_DATA_PRODUCT: MOD03.001

EVENT: UPDATEMETADATA

START_DATE_of_SUBSCRIPTION: 12 01 1998

EXPIRATION_DATE_of_SUBSCRIPTION: 01 01 2001

MESSAGE_of_SUBSCRIPTION: THE GEOLOCATION MOD03 QA granule was UPDATED at last!

DATA_ACQUIRE_METHOD: NO

local_user_name: N/A

local_password: N/A

local_host: N/A

local_directory: N/A

metadata_qualifier: DAYNIGHTFLAG==Day

ECS_subscription_ID_Number: 178

end SubscriptionRequest

EXPLANATIONS

The following list is a line by line description of the fields in the Subscription Request:

begin SubscriptionRequest

! This is a mandatory line to separate subscriptions from each other.

REQUEST_TYPE: (MANDATORY)

! This is what you want to request for subscribing to ECS data: ADD a new subscription, DELETE an existing subscription, or REVISE an existing subscription. You will be assigned a new subscription ID number, which will be e-mailed to you when the subscription event is triggered inside ECS. You must know the existing subscription ID number to revise or delete.

ECS_ACCOUNT_NAME: (MANDATORY)

! This is your ECS account user name as assigned when you registered previously with ECS.

RETURN_EMAIL_ADDRESS:

! The address to which ECS will send the subscription notices to you. This address is nominally the one you used to register as a user of ECS but it can be different from the address from which you send the subscription request. If the address is non-existent, you won't receive any subscription notices from ECS.

ECS_DATA_PRODUCT: (MANDATORY)

! This is the ESDT shortname and version number (MODxx.001) of the data to which you are subscribing. For example, MOD03 for PGE01, MOD021KM for PGE02, MOD35_L2 for PGE03, with the version number as a suffix. A list of the available data product ESDTs will be published at a future (TBD) place.

EVENT: (MANDATORY)

! The ECS event that triggers a response to your subscription. The value INSERT will trigger your subscription when the data you specified is inserted into the ECS archive. The value UPDATEMETADATA triggers your subscription when the data you specified has had its metadata changed. The value DELETE causes ECS to notify you when the specified data has been deleted from the archive.

START_DATE_of_SUBSCRIPTION: (MANDATORY)

! The date you want your subscription to be activated. From the start of this day, any data and event that satisfies the criteria specified by your subscription, will cause ECS TO NOTIFY YOU AND PERFORM ANY ACQUIRE action that you specify in DATA ACQUIRE METHOD (FTP delivery, 8mm tape mailed, or NO action).

EXPIRATION_DATE_of_SUBSCRIPTION: (MANDATORY)

! The date you want your subscription to be deactivated. Beyond this day, your subscription will be deleted from ECS and you will no longer be notified of ECS events specific to your subscription nor receive the data specified in your subscription. However, other subscription you have to ECS will still be valid, and you can still receive notices and data according to those remaining subscriptions.

MESSAGE_of_SUBSCRIPTION: (MANDATORY)

! This message that you specify will appear in the email notices that ECS returns to you when the subscription is triggered. The text can be anything you want to let you or your software know anything pertaining to the data or you or ECS or your subscription or whatever else you need to be reminded. Of course writing an extremely long message containing dozens of lines could cause ECS warnings or errors with the subscription.

DATA_ACQUIRE_METHOD: (MANDATORY)

! This is the method, if any, by which you want to acquire the ECS granule data along with its metadata. You can choose FTP delivery, 8mm tape mailed, or NO action.

1) NO action means that you will be notified when the event happens, but you will not automatically receive the data. This is a means of keeping informed about a data type without being swamped by receiving many large data files.
2) You can have the data on a DAAC 8mm tape mailed by U.S. post to your ECS registered user postal address.
3) You can have the data FTPed to you if you fill in the following destination information. Otherwise, use N/A in the following "local" fields.

local_user_name: (OPTIONAL)

! This is needed only if you want the data by FTP. Otherwise N/A. This would be the account name that you use on your local host.

local_password: (OPTIONAL)

! Since this is an insecure transmission, you might consider restricting privileges to your FTP account. However, the local directory receiving the ECS granules needs to have write permission for a successful FTP.

local_host: (OPTIONAL)

! The numerical IP address (ex. 198.118.000.001) or the alphanumerical domain name (ex. eosdatacruncher.research.org) of the machine onto which the data will arrive.

local_directory: (OPTIONAL)

! The full directory path where you want the ECS data to arrive. Example: /home/eosuser/FTP/MODIS/GEOLOCATION/quality_assurance/day_data. The directory needs to have write permission for a successful FTP. The directory should be created or exist BEFORE the subscription request is sent to ECS or the GSFC DAAC. You might need to specify the disk in front of the path if you have multiple disks on your machine. ECS will push the data to you only if adequate disk space exists; some files are hundreds of megabytes each!

metadata_qualifier: (OPTIONAL)

! There are dozens of metadata types and hundreds of values one can use to selectively qualify the data to which you are subscribing. Some are listed below, but a more complete list of them can be found either in ESDT descriptor files, metadata files and documents on the ECS data model.

ECS_subscription_ID_Number: (OPTIONAL)

! A number assigned by ECS to a previously entered subscription. The number appears in notices sent out (triggered) by the subscription. The email notice will have the line: Subscription ID: with the number of your subscription identified for ECS.

end SubscriptionRequest

! This is a mandatory line to separate subscriptions from each other.

SAMPLE ECS NOTICE

Here is a sample ECS return email notice you will receive when your subscription is triggered by ECS data events. (The qualifier list given is only a partial list of the supplied metadata values. However, the first three qualifier values are the important UR ECS name, used for internal ECS data tracking, and the LocalGranuleID, used for external data identification of the same ECS granule, and the observational DayNightFlag == Night.):


Subject: ECS Notification

Date: Mon, 25 Jan 1999 13:24:55 -0500 (EST)

From: CM SHARED (CM Code Delivery) <cmshared@g0ins01u.ecs.nasa.gov>

To: Richard.Buss@gsfc.nasa.gov

This Notice is for an Insertion of L2 product MOD02HKM with FTP:

UR:10:DsShESDTUR:UR:15:DsShSciServerUR:13:[GSF:DSSDSRV]:20:SC:MOD02HKM.001:3580

ESDT Information: MOD02HKM.001:INSERT

User Information: rbuss

EventID: 79

Subscription ID: 248

Qualifier List:

UR:10:DsShESDTUR:UR:15:DsShSciServerUR:13:[GSF:DSSDSRV]:20:SC:MOD02HKM.001:3580

MOD02HKM.A1997212.0435.002.1999025132118.hdf 01/25/99 18:21:19

Night

processed once

no further update anticipated

not being investigated >P>Passed

0

EV_1KM_RefSB


SAMPLE FTP NOTICE

This is a sample email that you receive when ECS has successfully sent your subscribed granules by FTP.


Subject: ECS Notification

Date: Mon, 25 Jan 1999 13:33:16 -0500

From: CM SHARED (CM Code Delivery) <cmshared@g0ins01u.ecs.nasa.gov>

To: Richard.Buss@gsfc.nasa.gov

ORDERID: NONE

REQUESTID: 2413256570645

USERSTRING:

FINISHED: 01/25/1999 13:33:15

MEDIATYPE: FtpPush

FTPHOST: gsfcsparc6.gsfcmo.ecs.nasa.gov

FTPDIR: /home/rbuss/FTP/PGE02_QA

MEDIA 1 of 1

MEDIAID:

GRANULE:

UR:10:DsShESDTUR:UR:15:DsShSciServerUR:13:[GSF:DSSDSRV]:20:SC:MOD02HKM.001:3580

ESDT: MOD02HKM.001

FILENAME: MOD02HKM#001073119970435000000000

FILESIZE: 252988371

FILENAME: SCMOD02HKM.0013580.met

FILESIZE: 49772


QUALIFIERS

The granules that you subscribe to can be selected further by specific metadata fields and their values. You will only receive notification or files that satisfy the criteria you specify. While we can give a partial list of allowed qualifiers, we can not possibly give every allowed value of every metadata because of the size, complexity, and currency of the ECS metadata.

------------------------------

BOOLEAN OPERATORS

==

<=

>=

-------------------------------

A partial list of metadata QUALIFIERS for subscription:

DAYNIGHTFLAG

SCIENCEQUALITYFLAG

ORBITNUMBER

GRINGPOINTLONGITUDE

GRINGPOINTLATITUDE

RANGEBEGININGDATE

RANGEENDINGDATE

LOCALGRANULEID

SIZEMBECSDATAGRANULE

PRODUCTIONDATETIME

(The request should be completed and sent as an ORDINARY TEXT FILE without fonts or other document styles)


Updated 6 May 1999, DPR, text by Richard Buss