Contents
3 Porting IBIS to Unix
3.1 IBIS-1 File Format
3.1.1 Platform Dependence
3.1.2 Intelligibility
3.1.3 File Format Rigidity.
3.2 IBISFIL Subroutine Library - Issues
3.2.1 Robustness for Large Files
3.2.2 Subroutine Interface
3.3 New IBIS-2 File Format
3.3.1 IBIS-2 System Label
3.3.2 IBIS-2 Property Label
3.3.3 The New IBIS-2 Subroutine Library
3.4 Porting IBIS Programs
3.4.1 Efficiency Considerations
3.4.2 Obsolete Assumptions
3.4.3 Stages of Porting an IBIS Program
3.4.4 Making code "GRAPHICS-1 Friendly"
In addition, through the use of EQUIVALENCE statements, a lot of programs try to manipulate the same column space as INTEGER*4 data. At the very least, there needs to be some information in the label telling the subroutine library how to convert the binary contents of each column into the format of the local host.
GRAPHICS-1 files have the additional problem that the dimension (NC) of the file is not stored in the file, so even the structure of the data is ambiguous. Theoretically, the "DIM" system label could have been used for this purpose, but this was never implemented.
The expansion of the data format capability would contribute to better data representation in applications. If a particular column is to represent only byte pixel values (for example, in a color lookup table), it makes no sense to store the values as REAL*4, when BYTE columns would work just as well, and be more efficient (both in I/O and disk space) besides.
A number of IBIS applications work with rows more often than columns. There are currently users who have expressed interest in the ability to continuously append more and more rows onto the end of a pre-existing IBIS-format archive. It would be useful if the file format could be extended so that, for example, data was organized with "row-contiguous" data, allowing for a fast way to concatenate two IBIS files by row without taking a large I/O hit.
There is also no current ability for an IBIS program to operate on a GRAPHICS-1 file, even though both file formats represent two-dimensional rectangular arrays of data, and both share the "record" concept. The old VICAR programs PCOPIN and PCOPOUT are often called numerous times within an IBIS procedure to address this limitation, and their only function is to convert between "row-oriented" GRAPHICS-1 format and the analogous "column-oriented" IBIS-1 file format.
This is because the IBIS subroutines which access columns will currently only read in whole columns at once. Users that have their own VAX platforms have been known to log on as SYSTEM, give themselves huge page file quotas and allocate all available virtual memory, just to be able to run a simple procedure on a single enormous IBIS file. This is not an intrinsic limitation of the IBIS file format, and should not be necessary.
The ported subroutines will also require separate entry points for the C and FORTRAN calling sequences, and so since the new P2 subroutine libraries are distinct from the old P2SUB, we might as well clean up the interfaces while we're at it, to reflect the abstract model of the new IBIS-2 file format.
An IBIS-2 file consists of one or more columns of various data types. The file itself may have a subtype, such as TABULAR, GRAPHICS, TIEPOINT, LUT, BIDR, or other types TBD.
The information describing each column is stored in the file's "IBIS" Property label. There is no limit on number of rows, except by the amount of addressable disk space available. The number of columns is constrained only by the limitations imposed by VICAR multi-valued label-lengths. For current purposes, the number of columns is restricted to be no more than 1024, but programs should not assume this limitation.
The column data format may be BYTE, HALF, FULL, REAL, DOUB, COMP, or an ASCII format specified by the strings "A1" through "A256". These last refer to CHARACTER*N data, where N is between 1 and 256. The ASCII data is stored in null-delimited form in the file (ie, so that A1 requires two bytes, etc). The data is stored in the VICAR binary header, allowing for the possibility of appending image data to the file.
In addition to having a format, a column may belong to several other defined column GROUPS, including one indicating what UNITs of measurement the column values are measured in. If the format of a column is not specified by a FMT_xxx label, it will be in a format specified by FMT_DEFAULT parameter.
The data may be stored by contiguous column values, or by contiguous row values. The location of the column data is indicated indirectly using byte-offsets.
The host format used for the column entries will be determined by the BHOST, BINTFMT and BREALFMT label entries of the VICAR SYSTEM label. In addition, the BLTYPE of the file will be set to "IBIS" to indicate the type of binary header data.
The IBIS file subtype indicates the presence or absence of various properties, formats or groups, which will evolve as new sub-types become defined. For example, a possible definition of the features common to the current GRAPHICS-1 file would be the following:
GRAPHICS file: An IBIS-2 file which possess (possibly named) columns containing line-sample coordinate values, commonly though not necessarily with contiguous row organization.
Notice that by this definition, many TABULAR files may be used as GRAPHICS files without conversion (i.e. the VICAR programs PCOPIN and PCOPOUT will be unnecessary).
TABULAR files need no longer be internally organized solely by contiguous column values, so that, for example, if a data table will primarily require row operations, it may be created with initially contiguous row values for more efficient disk IO. In addition, the physical order of the columns may not coincide with the logical order; this information is stored in the label, but should not cause too many difficulties, as the subroutine libraries will make logical column access transparent to the program and the user.
Note that GRAPHICS files, as special cases of TABULAR, may now be manipulated with the usual IBIS routines, such as MF, SORT, etc.
Remark: When the time comes to integrate the GRAPHICS-1 file format into the IBIS-2 configuration, consideration should be given to the manner in which polygons are delimited. Currently this is indicated by a set of zero coordinate values within the list, but if non-line-sample coordinate spaces are also desired (ie, zero-zero based), it may be necessary to replace the (0,0) delimiter in the new format with an additional "CONTROL_CODE" column, which would indicate whether a given set of coordinates is a VERTEX or simply an END-OF-DATA delimiter (other extensible codes may also be defined). In the process, this would automatically yield the capability of merging in the GRAPHICS-2 format, which contains more object-based information (e.g, there could be a CONTROL_CODE for ELLIPSE, with the next values being the major-minor axes and orientation. This is all TBD.
For an example of a generic new-version IBIS file, refer to figure 3.3.1.
Col# 1 Col#2 Col#3 Col#4 ... Col # 1022 REAL*4 BYTE A10 DOUB COMP_____________ 3.0000 012 'image1.red' 2.345978454447 4.56 + i * 1.94 2.5800 128 'image1.grn' 6.392209944e+2 0.00 + i * 0.00 1.e-10 256 '@@@@@@@@@@' 0.000000000e+0 0.00 - i * 3.45 0.0000 000 'image3.blu' 0.0033040e-123 33.0 + i * 2.71 3.0000 012 'image1.red' 2.345978454447 4.56 + i * 1.94 2.5800 128 'image1.grn' 6.392209944e+2 0.00 + i * 0.00 1.e-10 256 '@@@@@@@@@@' 0.000000000e+0 0.00 - i * 3.45 0.0000 000 'image3.blu' 0.0033040e-123 33.0 + i * 2.71 Figure 3.3.1: A typical new IBIS file
In addition, the BLTYPE of the file will be set to "IBIS" to indicate the format of the binary header.
Since the IBIS data is stored in the binary header, this opens the possibility of creating VICAR images with arbitrary image data formats and sizes, which also contain IBIS-2 information in the binary header. This may be useful for installing IBIS-format lookup tables into an image, creating GRAPHICS overlays for the image, etc. However, once an image has been appended to an IBIS file, the IBIS data may not be expanded (by adding columns, etc) beyond its current boundaries.
Here are the REQUIRED IBIS Properties for TABULAR and GRAPHICS1 type.
Property Values Descriptions NC Integer Number of Active Columns NR Integer Number of Active Rows ORG "COLUMN","ROW" Contiguous Data Organization FMT_DEFAULT 'BYTE','HALF'... Default Column Data Format SEGMENT Integer Smallest complete data group BLOCKSIZE Integer # Usable bytes of each line COFFSET (OFF1,...OFFN) Offset to Logical Column
Property NC: Number of Active Columns
The number of columns in which data is currently defined. There may be more unused space in the file, perhaps from a deleted column left in place.
Property NR: Number of Active rows
The number of rows in which valid data is currently defined. Note that there may be space available in the file for additional rows, which is currently unused.
Property ORG: Physical organization of contiguous data
In old IBIS files, the data was organized so that the elements of a column were contiguous, whereas in GRAPHICS-1 files the row (coordinate) elements were contiguous. This label item allows the specification of either organization for an IBIS file. In some applications where most operations involve rows, ORG='ROW' files may be more efficient than ORG='COLUMN'
Property FMT_DEFAULT: Default Column Format
This is an ASCII label item, indicating the format of all columns whose data formats are not explicitly set by a FMT_XXX label item. Its values may be either BYTE, HALF, ... or a string value of the form "An", where <n> is an integer between 1 and 256, indicating that any defaulted column will be ASCII, of size <n>.
Property COFFSET: This is an internal property used by the subroutine library, and usually won't be needed by the user. For more information on the internal representation of the IBIS-2 file format, see Appendix B.
Property BLOCKSIZE: This is an internal property used by the subroutine library, and usually won't be needed by the user. For more information on the internal representation of the IBIS-2 file format, see Appendix B.
Property SEGMENT: This is an internal property used by the subroutine library, and usually won't be needed by the user. For more information on the internal representation of the IBIS-2 file format, see Appendix B.
The following values are CONDITIONAL, and may or may not appear, depending on the data contents:
Property Values Descriptions FMT_BYTE Integer Array List of Columns using BYTE data. FMT_HALF Integer Array List of Columns using HALF data. FMT_FULL Integer Array List of Columns using FULL data. FMT_REAL Integer Array List of Columns using REAL data. FMT_DOUB Integer Array List of Columns using DOUB data. FMT_COMP Integer Array List of Columns using COMP data. FMT_ASCII Integer Array List of Columns using ASCII data. ASCII_LEN Integer Array Length of ASCII columns
Property FMT_BYTE,...FMT_ASCII: Column Formatting Declaration
These Optional labels specify an array of integers, consisting of the column numbers using the corresponding format. If a column is not explicitly set by one of these label items, it is in in the format defined by FMT_DEFAULT, above.
Property ASCII_LEN: ASCII character-length declaration
For the set of columns specified in the FMT_ASCII array, if any, this integer label item specifies the character length of the column; that is, it is the "CHARACTER*N" value of the column. If the FMT_DEFAULT is a particular ASCII format (e.g., "FMT_DEFAULT='A200'), then the defaulted ASCII columns do not appear in this list. Note that the file size of an N-character string is N+1, to allow for the null-delimiter.
User-definable IBIS Properties for TABULAR and GRAPHICS1 files: these are OPTIONAL and do not affect the file structure or data formats.
Propert Values Descriptions TYPE 'LUT','TIEPOINT'.. IBIS subtype UNITS (String1,...StringN) degrees, acres, kg*m/sec^2, etc... GROUPS (String1,...StringN) Line, Samp, Average,... UNIT_<N> (Col1, ...ColM) Columns using unit UNITS[N] GROUP_<N> (Col1, ...ColM) Columns belonging to GROUPS[N]
Property TYPE: IBIS Subtype Declaration
This is an ascii value indicating the subtype of the IBIS file, and may be found to be useful in setting up subclasses of IBIS files. For example, a lookup table may be defined as having certain named columns and formats, whose existence will be indicated by the TYPE property being set to "LUT".
Property GROUPS and UNITS: These properties allow client-defined subsets of columns, collectively known as groups. For the <N>th name in the GROUPS (or UNITS) list there is a corresponding integer-valued list called GROUP_<N>, (respectively UNIT_<N>). For example, a client program may wish to define column #1 as "LINE" and #2 as "SAMPLE", and then put both into a group "POSITION". To do this, the following IBIS properties may be added:
GROUPS=("LINE", "SAMPLE", "POSITION")
GROUP_1=1
GROUP_2=2
GROUP_3=(1,2)Similarly, to define the units of some columns, the properties added could be:
UNITS=("METERS", "MILES_PER_HOUR")
UNIT_1=(3,4,5)
UNIT_2=6Units differ from groups only in that a column is only allowed to have at most one unit, but may belong to many groups.
As will be seen, these group names may be used in place of explicit column numbers to access data within an IBIS-2 file. For example, instead of having to remember that the line coordinate is in, say, column number 23 of a file, the client program can simply query the file to see which column is in the group "LINE", and then read data from that column.
The name given to a group or a unit must begin with an alphanumeric character, but otherwise the characters following the first character may be ANY printable character except colons, up to 32 characters total. The complete list of characters that may be used are:
[a-z][A-Z][0-9] ~!@ # $ % ^ & * _ + - = (){}[]<> | ? ; , . " ' ` \/ Here are some valid examples:
kg*m/sec^2
2
a_long_Long_LOng_LONg_LONG_name!
has_[brackets]_and_{braces}(etc)
!@#$%^&*}\/?=
A name with spacesHere are some invalid examples:
%starts_with_NON_alphanumeric has:colons:in:its:name a_long_Long_LOng_LONg_LONG_Whoops!_TOO_long_name!
mode 1: Modifying file architecture - ncols, nrows, or col. formats using IBISRow, IBISColumn Delete/New routines
mode 2: Modifying file values using the IBISRow,IBISColumn Read/Write routines
mode 3: Modifying file values using the IBISRecord routines
Operations performed within each mode will not cause substantial degradation of performance; however, switching back and forth between modes will likely force the subroutine library to perform file-I/O intensive operations to keep the record buffers in sync. This is the price for not storing the whole file in memory at once.
If speed is a premium, there are several things one can do to improve performance. First, if you are creating a brand new IBIS file, by default the IBISFileUnitOpen routine will initialize all of the column values of the file to zero; if you intend to set all the values in your program, you may turn off this feature by changing the AUTO_INIT attribute of the file to 'OFF'.
Also, if this is a special-purpose application, involving the development of an IBIS subtype, consideration should be given to whether the file is to organized by rows or columns:
If the application involves intense manipulation between a relatively small (less than 500, say) number of separate sets of data, and the data is all numeric, it is probably better to put each set of data in a separate column, and organize the file by COLUMNS. It is not expensive in I/O to create or delete additional columns for column-oriented files, and you have the ability to access each column by name, by putting them in separate GROUPs. The weakness of this approach is that appending additional rows will often be an I/O intensive operation. If for some reason you anticipate extending the length of a column at a later date, you can allow for this when your application creates the file by using IBISFileSet to set the NR attribute to a large number, open the file, and then using IBISFileSet to decrease the NR attribute back to the current value. This has the effect of allocating a much larger padded space in the file for each column, so that appending new rows later may be done "in file". For column-oriented files the IBISColumn Read/Write routines are the most efficient, although the IBISRecord routines are also very efficient.
If the application is intended to be a static index database, or for some other reason consists of a very large (more than 2000, say) number of data sets, the elements of the data sets are of different types (mixed ASCII, real, and integer), and are rarely going to be numerically combined with each other, then you should organize the file by ROWS. It is not expensive in I/O to append additional rows to a file of this type, and if you need to have a 'GROUP' capability, you can accomplish this by making on of the columns of the file an ASCII column, containing the names to be attached to each row. The weakness of this approach is that creating new columns will often be an I/O intensive operation. As with the column-oriented files, if you anticipate having to change the number of columns, you can expand the file with a larger 'NC' value first, and then decrease it after opening the file. However, it is not advisable to do this if your database file will contain a huge (more than 500,000, say) number of rows, as the file will be wasting disk space for each row. For row-oriented files the IBISRecord and IBISRow routines are the most efficient.
The column data types will be INTEGER*4, REAL*4, or CHARACTER*4, whose format is not specified in the file, but imposed by the user or the program.
The data is stored by listing all values of column 1, padded out to 512 -byte records, followed by all values of column 2, etc.
The data is organized into 512 byte records, and is stored in the VICAR image data portion of the file.
The number of columns is no more than 40, and is stored in the first VICAR line of the file.
Initially, the IBISFIL front-end will not create IBIS-2 format files, but will be able to read them. This is because several IBIS programs currently try to get around the limitations of the old IBISFIL routines by directly manipulating the IBIS files using VICAR xvread, xvwrit calls, and so will have to be re-written before they will be able to handle the new formats. Until these programs have been made forward-compatible, the IBISFIL front-end will not create IBIS-2 files by default.
The second stage of porting IBIS program will involve actually re-writing IBIS programs to use the IBIS2 calls directly -- there will be no IBISFIL routines in the p2sub library. At the very least, the new program should take advantage of the IBIS2 buffering routines which free the program from having to store an entire IBIS file in memory, and should exit gracefully if one of the extended type IBIS files is encountered (for example, IBIS files with non-4 byte columns). This step will, of course also include the usual steps for porting any VICAR program to UNIX. Note that this means that EQUIVALENCE statements for converting between real and integer columns is not permitted (they may be used only to conserve memory).
The final stage of porting an IBIS file should include the extension of the program to handle the full range of new IBIS files, such as DOUBle formats and A256 columns. Some examples of such programs will be provided to get the programmer started.
To make IBIS programs "GRAPHICS-1 Friendly" it is suggested that during the process of porting IBIS-1 code written for interface files only, or while writing brand-new code, that a "G1DIM" parameter be provided in the PDF file, so that the user may specify the dimension of graphics files they are processing. Old GRAPHICS-1 programs already have a "DIM" parameter for this purpose, and so they may retain this parameter name. This parameter value may then be passed in to IBISFileUnit or IBISFileOpen to tell the library the NC size of the file.