SABER Project Information Server Data Flow

Download Data & Project Information Server in pdf format

• Scope
• Referenced Documents
• System Overview
• System Requirements
• Computer Software Configuration Items
• Ingest Project Documents and Miscellaneous Data Files
• Ingest and Catalog Data Product NetCDF Files
• Store Files to SPIS Filesystem
• Query Database for Data Product File URL
• Generate Index / Description and URL File
• Acronym List

1. Scope

This Software Design Document (SDD) describes an overall software design plan for the SABER Project Information Server (SPIS) software. The SPIS software will be responsible for providing a user-friendly interface to the documentation and data produced by the SABER project. Data products from the Levels 1B through 2B processing steps will be automatically ingested into the system. Project documents and other data files will be ingested upon user request. Data and documents will be served to the user community via HTTP and FTP protocols.

2. Referenced Documents

• TIMED Project Data Management Plan, JHU/APL, 7363-9330
• General Instrument Interface Specification (GIIS) [pdf]
• SABER Data Management Plan, NASA Langley Research Center. Not available at this time
• Level 1B format document [pdf] [HTML]
• Level 1B format document [pdf] [HTML]

3. System Overview

The SABER Data and Project Information Server data flow is shown in figure 1. NetCDF data files produced by the Level 1B through 2B processing steps will be ingested and cataloged as it is produced. File and record header information will be examined and recorded in a database to provide rapid searching and access to individual radiance and mixing ratio profile data. The preferred method for submitting project documents and miscellaneous data files to the SPIS is via the HTTP file upload capability, which facilitates the automatic ingestion and cataloging of the files. FTP upload capability will be available as needed.

The database will be searched on a nightly basis to identify any new additions to the SPIS. The results of this search will be used to maintain a set of files to enhance access to the data products and documentation. The first files will be used as index and description files to facilitate users browsing through the directory tree containing the data and documents served by the SPIS via HTTP or FTP. The POC Data Product Status file and the POC Data Products URL file defined by the TIMED GIIS will be updated as necessary.

4. System Requirements

The SABER Project Information Server will be based on the architectural concept suggested by the NASA GFSC CESDIS Beowulf project. This architecture is a scalable machine comprised of mass market common off-the-shelf (M²COTS) PC technology components using a freely available operating system (Linux) and industry-standard software packages. This allows the initial cost for the server to remain low while facilitating subsequent enhancements. The Linux operating system is available from multiple vendors. The Linux vendors under consideration provide complete packages with all tools and utilities needed to set up an Internet server. In particular, SPIS will use the Apache HTTP server, Perl (and associated modules), Postgresql SQL database server, and the GNU C/C++ compiler.

5. Computer Software Configuration Items

5-1. Ingest Project Documents and Miscellaneous Data Files

The Ingest Project Documents and Miscellaneous Data Files CSCI provides a mechanism for placing project documentation, correlative data and other files on the SPIS for access by the user community. A web interface will be provided to allow a contributor to specify properties of the document or data file to facilitate placement of the file and subsequent location of the file via database query. This interface will utilize the HTTP file upload protocol. A mechanism for ingesting document and data files via the FTP file upload protocol will also be provided. The information required for ingest purposes has not yet been specified.

Requirements:

• A Web interface will be developed to facilitate file upload and ingestion into the SPIS. This interface will include (but not limited to):
• File Selection widget
• File type or category
• Description of file contents
• A valid username and password combination will be needed to access the file upload capability.
• User registration capability will be needed to allow the Science Team and other interested parties to sign up for upload access to the SPIS.

Testing:

• Test the upload access security with combinations of valid and invalid usernames and passwords.
• Test the user registration procedure by:
• First attempting access with an invalid username and password,
• Next, register the username and password used previously,
• Finally, after the registration process has been completed, attempt access again.
• Test the file ingest process by comparing the information provided to the web interface with the information recorded in the database.
• Test the file upload process by:
• Comparing the actual SPIS URL for the uploaded file, with the URL stored in the database (i.e. is the file actually stored where the ingest process thinks it is stored and can the SPIS actually write to that location).

5-2. Ingest and Catalog Data Product NetCDF Files

This CSCI provides a common interface to the SPIS for SABER Level 1B through Level 2B netCDF data files. The SPIS will be connected to the LAN hosting the systems producing the Level 1B through Level 2B netCDF files allowing seamless access via NFS and utilizing the file I/O classes developed for the science data processing. For the science data product files, the ingest and catalog process consists of reading the file header for key information as shown in table 1, followed by scanning the file for record header information as shown in table 2. NOTE: Tables 1and 2 represent a preliminary estimate of the information desired and are subject to change.

Requirements:

• The CSCI must be able to read the Level 1B through Level 2B netCDF files and extract the required cataloguing information.
• The required cataloging information must then be recorded in a database.
• The information placed in the database must then be accessible via standard SQL queries.

Testing:

• NetCDF files containing predefined Level 1B through Level 2B data are presented to the CSCI.
• Cataloging information is extracted and inserted in the database and the database insertion command is recorded.
• Compare the database insertion command with the predefined data.

Table 1: Ingested File Information

Table 2 : Ingest Record / Event Information

5-3. Store Files to SPIS Filesystem

The document and data files acquired via CSCI 5.1 will be written to the SPIS filesystem in a location distinct from the netCDF data product files ingested via CSCI 5.2. Because the format of these files is assumed to be unknown at ingest time, no attempt will be made to scan the contents to extract information. Instead the files will be handled as a generic byte stream.

Requirements:

• The HTTP server and anonymous FTP server must be able to write to the location specified by the CSCI 5.1

Testing:

• Perform a byte-by-byte comparison between the source file and the file written to the SPIS filesystem.

5-4. Query Database for Data Product File URL

Database query via the web interface will allow users to determine which data product netCDF file they want based on their specified query parameters.

Requirements:

• The database server will be accessible via the HTTP server for standard queries and file ingests.
• The database server will be accessible via a traditional command line or GUI client for custom queries and administrative tasks.

Testing:

• Use command line client to test the database server for operation and set up appropriate schema.
• Ingest predefined / simulated netCDF files (one for each data product) and use the command line client to query the database. Compare results with actual file and file URL.
• Ingest predefined /simulated contributed data files and use the command line client to query the database. Compare the results with the ingest information and file URL.
• Ingest predefined /simulated document files and use the command line client to query the database. Compare the results with the ingest information and file URL.
• Use the web interface to query the database for the predefined / simulated files mentioned above. Compare the results as defined above.

5-5. Generate Index / Description and URL File

This CSCI provides for the maintenance of the index and description files located in each directory of the SPIS filesystem hierarchy. On a periodic basis, the database is queried to determine what updates are needed to these files. A similar periodic query will also be used to generate the POC Data Products URL file required by the TIMED MDC.

Requirements:

• A database query will be automatically run on a periodic (daily) basis producing or updating:
• A POC Data Products URL file as defined in the TIMED GIIS.
• A POC Data Product Status file as defined in the TIMED GIIS.
• A series of index / description files defining the contents of each directory in the SPIS filesystem. This is to help users browse through the available file via FTP or HTTP.

Testing:

• After ingesting the files as defined in CSCI Test 5.4.2, run the database query to generate the specified files. Compare the generated files with the information in the database (assumed to be correct after successfully completing CSCI Test 5.4.2.

6. Acronym List