Copyright © 1998, 1999, 2000, 2001 The California Institute of Technology
All rights reserved.

FEI Overview

Last updated on February 28, 2001


Send questions and comments to Danika Jensen.

Introduction

The File Exchange Interface, referred to as FEI, is a client/server system for managing and transferring files, especially large image files. It has a basic set of file manipulation functions including get, add, replace, delete and rename; but it's best known for its subscription capability. Subscription allows a client to register with a file type. Whenever a file of that type is added or replaced through an FEI server, a copy of the file is immediately sent to each subscriber.
FEI is supported by the Jet Propulsion Laboratory's Multimission Image Processing Laboratory. The system has been used by over 500 users over the past several years associated with Galileo and Mars Pathfinder missions, and is currently scheduled to support 8 up-coming missions.

The most notable application of FEI involved the processing and delivery of images for the Mars Pathfinder Mission. Using event-driven file transfers, FEI delivered images to the Fight Mission Team at near real-time speeds; about 3 seconds for some applications. It was also used to distribute images and data to Web sites for immediate viewing by the press and the public. The immediacy of file delivery by a small team at JPL was part of the mission's dramatic impact.

This document offers an overview of the product and covers the following topics:

 
Features A list of FEI's capabilities and features.
Architecture A summary of the FEI client/server architecture. This section also discusses the concepts implemented in the FEI design.
An Example Application The description of an applications that incorporates FEI features.
Supported Platforms A list of the operating systems supported by FEI.

Features

Location Transparency

FEI organizes files into file types. To transfer a file, or set of files, the client only supplied the file type and the files of that type to transfer. The location of the target FEI server and file directory are transparent to the user. System Administrators can change the FEI server configuration and file directory organization without impacting client applications.

Basic File Transfer Functions

FEI file manipulation functions include get, add, replace, delete, and rename. File locking is incorporated into the system so a client can't get a file while it's being added, replaced or deleted and a file can't be replaced or deleted while another client is getting the file.
Files can be transferred by supplying individual file names or by using regular expressions in file names to transfer a set of files. For example, to get all the files whose names begin with "file", the regular expression "file*" is supplied. A date and time can also be supplied to get files since a particular time. This is useful to transfer only newly arrived files.

Subscription

Subscribing to a file type registers an application to receive newly added or replaced files of that type. For example, if we were subscribing to the "Mars Pathfinder Rover Image" file type, each time an image was received from the Rover, all subscribers immediately get a copy of it. Client applications can automatically store, process or display the files as they arrive.
If a subscription session fails because of client, server or network problems, the "restart" parameter can be included when the processes is restarted. The subscription session then synchronizes the subscription stream so all files in the stream of delivered to the client, include the files entered into the system while the client was unable to receive new files.

Security

FEI uses Kerberos security servers to authenticate users when connecting to server. Access control lists are used to define a user's ability to access files of a particular type.
Optionally, a release date can get specified for a file type. Users can then only get files of a particular type that entered the system before the release date unless they have also been granted the capability. This limits access to newly arrived files. The release date can be updated by the FEI administrator to make newer files available over time.

Database Support

>Each FEI domain is supported by a database management system. FEI configuration parameters, access control lists and file information are stored in the database. Management of the system is handled by standard SQL statements. Also, meta-data about files in the system can be related to information about files stored in the FEI tables. In this way, a client can query for a set of files based on complex query criteria, not just file name.

Data Integrity

Actions on files are atomic; they execute completely or not at all. They are also isolated; the transaction executes as if it we running alone, with no other transactions. This prevents a client from reading a file while it's still being written, and prevents an update or deletion of a file that is being read.

Performance & Scalability

Both FEI servers and clients are multithreaded. This means that a program can do other things while files are being transferred. File actions are queued to the system. The result of a transaction can be examined after other useful work is performed by a client application. Also, since threads are used in the server, new processes are not started to service each client. This speeds transactions.
FEI servers can take advantage of multiple CPU within a machine to speed processing.

Load Balancing

Because files are organized by file types that are location-transparent, different file types can be placed on different machines at any time to balance the systems load. Client applications are unaware of the physical location of processing for any particular type of file.

Processing Streams

Processing steps carried out by different client applications can be linked into processing streams. Each client subscribes to the type of file added to the system by the last client process. At any step, users can subscribe to a generated product in the stream. Using subscriptions "restart" capability, components of the stream can be stopped and restarted without effecting the processing streams state. FEI comes with a framework of classes to implement this type of processing. Application programmers use the framework to create particular processing steps.

Documentation

FEI is documented in a set of HTML pages covering such topics as system overview, installation, using command line utilities, and building applications with the C++ API - including working example code 

Architecture

FEI uses a client/server architecture. The server environment is called an FEI domain, and it includes three types of servers:
Client applications communicate with FEI servers over a TCP/IP network through the FEI Application Program Interface or API written in C++. We use the API to create a set of standard command line applications that communicate with FEI servers. The API comes with a set of documentation and example programs so that it can be used to integrate FEI services into other applications.

A Java interfaces is also planned. The focus here is building GUI applications that have FEI services as a component. We expect this interface to be available by the end of 1998.

The diagram below shows the components of an FEI server domain with a client application connected to it. FEI components are red, client components blue, database management system components orange and security components green.

For simplicity, the diagram shows a single FEI server. However, an FEI domain can have several servers, each managing a different set of file types. Since client access to servers is address-transparent, the domain has a unified logical appearance.

The domain's database contains FEI configuration information, user access control lists, and file information. The database can also be used directly by client applications to access meta-data about files. Project meta-data and FEI file information can be related within the database by joining tables.

Here is an outline of the typical set of events that comprise an FEI session.

  1. A Kerberos ticket is obtained from the Kerberos server using an independent utility called feiInit. (For those people who have worked in a Kerberized environment before, feiInit is actually a renamed copy of the Kerberos utility kinit. We use our own copy because systems often contain multiple copies of this utility and we want to make sure the correct version is used with FEI.)
    2. Once you have a Kerberos ticket, you can make any number of connections to FEI without further authentication efforts. (Kerberos tickets have an expiration time. After that time you need to get a new ticket. However there are options that can be used by applications that must be started automatically over a long time period.)
  2. Once you have a Kerberos ticket, a client application makes a connection to an FEI file type.
  3. Both the FEI client API and the target FEI server access the Kerberos server to authenticate the connection.
  4. The FEI Domain File local to the client application is used to make the connection. The file contains the information that maps file type names to server connection information.
  5. Once connected, the FEI server used the authenticated name and the supplied file type to determine the clients access capabilities and the location of files of that type.

    6. All of the steps above occur very quickly and are accomplished with a single connection command in the API.

  6. The client is able to issue file transfer or file manipulation commands. FEI connections are often long-term, lasting weeks or months. Once authenticated and connected, the client continues to access the file type until a disconnect is issued or until a system failure. (After a system failure, a client can automatically restart and continue. This is true for subscriptions as well. The "restart" options supplied to the subscription command resynchronized the event driven stream that delivers files of a given type.)
  7. The client can use a direct database connection to query for file names based user specified search criteria. The list of files returned is then supplied to FEI to get the files.
  8. Both the client and the FEI server are multithreaded. Thread management complexity is hidden on the client side, so developers to not have to concern themselves with it.

An Example Application

An Automated Remote Sending Example

This example shows two cooperating sites using FEI. In the following diagram, the "blue" site collects and processes remote sensing data of the earth. Images are created and stored in FEI. Image data is deposited in a database management system. At the "pink" site, located in Japan, images are selected based on an area of interest, represented by a database query over a geographic region. The image processing at this site entails comparing newly arrived images with a historical archive. This hypothetical application produces a spectral histogram from the new image an compares it with information from the historical database. If significant changes are found, the new image is sent to a government agency for review. The change may indicate a potential change in the agricultural output for a region. Since acting on this type of information may have economic consequences, the images are transferred in an encrypted state.

Here's how FEI is used in such a system.

  1. Packets received from a satellite are added to FEI, the gray server in the upper diagram.
  2. A demon process associated with the file type used to add the file is executed automatically when an image arrives. This produces the image that is then saved as a different file type in FEI. As part of the image processing procedure, a record is added to a database table. The record contains the name of the image file along with coordinates of it's covering geographic area.
  3. The Batch Exchange processing the the lower diagram, subscribes to the image file type using a special form of subscription called live list. This mode does not send the image, but only the file name for the image. The Batch Exchange process uses the name to query the "blue" site's database. If the image falls within an area of interest, the Batch Exchange process gets the image and adds it to its own FEI server.
  4. The demon process at the "pink" site, automatically makes the image comparison described above using its historical database.
  5. If it finds significant changes in the area of coverage, it sends the image on to a subscribing government agency where the image is given a closer look. The image is encrypted automatically encrypted during this transfer. (The request to encrypt a file is a subscriber option.) Sending the image on to the government agency does require that the image be stored in FEI again, so a special type of subscription called conference is used. This routes the image without saving a copy of it. (It's called conference because it was originally intended for sharing images among users who where in a distributed conference setting.)
This example shows several types of subscription in use to fully automate a flow of data between two sites. The flow of data includes data processing, data storage and the use of coordinated database information to solve a problem.

A Space Flight Mission Example

This example shows a distributed image processing example typical of the configurations used by Space Flight Missions, although simplified to make the example manageable.
Here we have three sites, "blue" in the United States, "pink" in Japan, and "green" somewhere in the world.

The "blue" site performs the following tasks:

  1. Data packets are received from NASA's Deep Space Network and added to an FEI server. These raw files are immediately shared with a subscribing site in Japan, and are also acted upon by a demon process associated with the file type used by the adder.
  2. The demon produces a Level 1 product and adds it back into FEI using a different server. Two servers are used to balance the load between near real-time processing and science team access.
  3. A copy of of the Level 1 file is sent to the FEI server in Japan. (Peer-to-peer exchange if a feature of FEI Version 3 due late in 1998.)
The Japanese site is responsible for products for a particular instrument. (The diagram shows only one processing stream, but we could imagine that several would be in place, each being delivered to its own remote location using subscription.
The Japanese site processes instrument data and makes it available to science team members through direct subscription and to the world in general though further processing  adds meta-data to a database and to a Web server, or a collection of Web servers.

The rest of the world accesses data from the Web sites.

Looking at the over all process flow we can make another few points:

  1. All of the flow uses event driver subscription connections. Initially adding a file, will cause that file to flow through the entire system.
  2. Since a single mechanism, subscription, is used, we have a sense of confidence in the make-up of the system. If subscription works, then the system as a whole should work.
  3. Upon failure of a component, subscription with restart, will resynchronize the flow of files through the system. While a component is down, the FEI file systems act as a buffering queue of work that will again begin to flow once the component is brought back on line.
  4. While it's not shown in the diagram, content error conditions can be handled by this system. That is, if a file is rejected for some reason, a process can send it to another file type and a demon can be used to send an alert message. An analyst can then "get" the file from FEI, determine the cause of the failure, fix it, and "replace" it in FEI at the proper position using a file type name. The fixed file will then flow through the system like any other file.
  5. This system could be extended to include an archive site. Nothing special is required. The archive could either subscribe to a file type for finished products or it could query the instrument database in Japan for the names of files ready for archiving. This is one example of how an event-driven, component based system can be extended.

Supported Platforms