hrptin [ parameter=value ... ] [ outputfile ... ] hrptin [ parameter=value ... ] [ directory ]
Parameters are hrpt_types, hrpt_format, start_conseq, max_hrpt_frames, on_pass_disk, tape_device, pass_number, files_per_pass, pass_year, channels, fix_missing, max_fix_miss, sharpen, solar_corr, min_sun_cos, calibrate, temp_units, byte_output, base_temp, temp_step, full_report, geo_correct, delta_line, delta_sample, use_lat_range, min_lat, max_lat, use_master, master_file, start_time, num_tip_lines, num_lines, num_samples, shared_mem, sort_by_var, dim_counts, pipe_avhrr.
hrptin creates TeraScan AVHRR (Advanced Very High Resolution Radiometer), DCS (Data Collection System), and TOVS (TIROS Operational Vertical Sounder) datasets from HRPT telemetry data. Input telemetry data can be read from the pass disk, a tape, or standard UNIX disk files. Multiple passes on tape or the pass disk can be processed at once.
The user can specify one or more of the five AVHRR channels to extract from the telemetry data. The data can also be spatially subsampled. The overall spatial subset of data to be extracted can be specified in one of three ways:
The smallest rectangular area required to cover a region defined by a master dataset.
A rectangular area of specific size, beginning at a user-specified time and scan sample.
A rectangular area of specific width, beginning at a user-specified scan sample, and intersecting a user-specified latitude range.
Output DCS datasets contain the following variables:
date - The date the message was received.
time - The time the message was received.
platform_id - The identification number of the message sender
dcs_doppler - The unprocessed DCS doppler shift numbers. It is used to compute
position.
dcs_quality - The message quality. 0 indicates a good value. 1 indicates
questionable data.
dcs_count - The number of 4-byte quantities in dcs_data. It ranges between 0 and 7.
dcs_data - The unprocessed DCS message.
Platform latitude, longitude, and elevation variables are added by the dcsloc function.
Output TIP files contain raw 520-byte TIP records. The set of records extracted can be specified in one of three ways:
TIP records corresponding to AHVRR scan lines intersecting a master dataset.
A specific number of TIP records starting at a given time and scan sample.
TIP records corresponding to AVHRR scan lines intersecting a user-specified latitude range.
NOTE: there are 3 AVHRR lines for each TIP record.
There are no input TeraScan datasets. The sources for input telemetry data are described above. Output is either to a user-supplied directory or a user-supplied list of dataset names. If a directory name is entered, output names will be automatically generated and the files will reside in the specified directory. Names will be generated that look like n#.yyddd.hhmm.type, where # is the NOAA satellite number, yyddd is the year and julian date, hhmm is the start hour and minute of the output dataset, and type is the type of data produced (avhrr, dcs, or tip). If a directory name is not entered, the name of the file will be used with the type of dataset appended to the end of the file.
The option of ingesting data based on latitude range is intended for reading GAC data from DOMSAT. In this one case, there can be multiple output AVHRR, TIP, and DCS datasets per input satellite pass. That is because GAC data is subsampled (delta_line=3, delta_sample=5) and the typical GAC pass contains data for one or more complete satellite orbits.
In all other cases, there will be at most one AVHRR, TIP, and DCS output dataset per input satellite pass.
This specifies the type(s) of data to be extracted from the raw HRPT telemetry.
Valid responses are [avhrr, dcs, tip]. The default is avhrr.
OPTIONAL. This specifies the type of the input HRPT data format. If hrpt_format=auto, the format type is detected automatically which works in most cases, including the one when the data is captured with the TeraScan system. See NOTES for a description of the formats listed below.
Valid responses are [auto scripps dlr toh jwa aes dartcom sharp 14800 14848 15360 global dundee bug lasham lac gac user jcook]. The default is auto.
OPTIONAL. This specifies the number of consecutive HRPT frames with respect to time needed to recognize the true start of useful pass data. Consecutive lines must be 1/6 of a second apart, and must also have good sync and matching satellite ids. Note that all but the last two HRPT frames from this initial sequence are discarded.
Valid responses are [> 2]. The default is 2.
OPTIONAL. Maximum number of HRPT frames per input satellite pass. The default is 6000 frames, which translates to over 16 minutes of live HRPT data. This parameter is needed only to process exceptionally long passes taken from satellite recorders. Setting the maximum number of frames greater than 15000 (about 40 minutes of data, a little less than half an orbit) may cause problems when using masters.
Answer yes if the input is from the pass disk, or no if the input is from tape or standard UNIX disk file.
Valid responses are [yes or no]. The default is yes.
If on_pass_disk=no, this is the name of the input tape device or UNIX disk file name. Tape device names are machine specific. For example, the following devices are commonly used on the Sun SPARC station:
/dev/nwd0 - DAT, fixed blocking, APUNIX driver /dev/nrwd0 - DAT, variable blocking, APUNIX driver /dev/nsx0 - 8mm, fixed blocking, APUNIX driver /dev/nrsx0 - 8mm, variable blocking, APUNIX driver /dev/nrmt - 8mm or CCT, variable blocking, SUN driverAll of the above device names imply no rewind on close.
The default is extracted from the UNIX environment variable TAPE. Only tape devices listed in the file $PASSDIR/devtable, or simple UNIX disk file names, are accepted.
This specifies the number(s) of the pass(es) to process, when input is from tape or the pass disk. Passes are numbered starting with 1. Input tape is always assumed to be positioned at the first pass.
Valid responses for input from the pass disk are [1 to maximum number of passes that can be stored on your pass disk (see lspass(1))]. The default response is the number of the last acquired pass.
Valid responses for input from tape are [1 to 100]. The default response is 1.
If input is from tape, this is the number of tape files per pass. If files_per_pass=2, then each pass is assumed to consist of a header file, followed by a data file. If files_per_pass=1, then the tape is assumed not to have any header files.
Valid responses are [1 or 2]. The default response is 2.
If on_pass_disk=no, this is the year that the pass was collected. All passes processed in one execution of hrptin must have the same year.
The valid range is [>=1970]. The default is the current year.
If hrpt_type=avhrr, this is a list of the AVHRR channels to extract from the HRPT pass. The five channels correspond to 0.6-0.7, 0.7-1.1, 3.5-3.9, 10.3-11.3, and 11.5-12.5 micrometers.
Valid responses are [1,2,3,4,5]. The default is 1,2,3,4,5.
If hrpt_type=avhrr, answer yes if the dataset is to have bad or missing lines of AVHRR data filled in by duplicating nearby lines. Performs the same function as avfix(1). Answer no if the dataset is to contain all bad or missing lines.
Valid responses are [yes or no]. The default is yes.
OPTIONAL. Only if fix_missing=y. The maximum number of sequential missing lines that can be fixed.
Valid response is any positive number. The default is 1.
If hrpt_type=avhrr, answer yes if the channel is to be filtered using a deconvolution algorithm defined in 5th AVHRR Data User's Proceedings (see avsharp). The algorithm will "sharpen" an AVHRR image by subtracting from each pixel any contamination caused by sensor over-sampling into adjacent pixels. Performs the same function as avsharp. Answer no if channel is not to be sharpened. sharpen (y/n) has to be specified for each selected channel. For example, for channels=1 2 4 5, to sharpen channels 1 4 only, the selection sharpen=y n y n has to be made.
Valid responses are [yes or no]. The default is no.
Determines whether or not to perform the solar zenith angle correction.
Valid responses are [yes or no]. The default is no.
OPTIONAL. Only if solar_corr=y. The minimum value for cosine of sun zenith angle.
Valid response is a real value in the range 0.0 .. 1.0. The default is 0.05.
if hrpt_type=avhrr, answer yes if the AVHRR dataset is to be converted to values with engineering units. For channels 1 and 2 the units are percent albedo. For channels 3, 4, and 5 the units are brightness temperature. This process is often referred to as radiometrically calibrating the data. New calibration coefficients are computed for each 100 input lines.
The calibration algorithm is based on information contained in Techniques for Data Extraction and Calibration of TIROS-N/NOAA Series Satellite Radiometers for Direct Readout Users by Levin Lauritson and Gary Nelson (NOAA-NESS Publication #107, July 1979). Performs similar operations as avcal(1). Answer no if no calibration is to take place.
Valid responses are [yes or no]. The default is yes.
If calibrate=yes, converts the raw AVHRR data to brightness temperatures with units of degrees Celsius, Kelvin, or Fahrenheit. Degrees Kelvin or Celsius should be used if the data will be used to estimate sea surface temperature using nitpix because nitpix does not currently support input data with units of Fahrenheit.
Valid responses are [celsius, kelvin, fahrenheit]. The default is celsius.
If calibrate=yes, answer yes if the output dataset is to be saved as 8-bit integer data. Radiance percent albedo is converted to 8-bit values by multiplying by 4 and rounding to the nearest integer. Temperature output is converted to 8-bit values using the base_temp and the temp_step parameters.
Answer no if the output dataset is to be written as 16-bit integer data. Both radiance percent albedo and temperature degrees are converted to 16-bit values by multiplying by 100 and rounding to the nearest integer.
Valid responses are [yes or no]. The default is no.
If calibrate=yes and byte_output=yes, these two parameters are used to convert temperature output to 8-bit values as follows:
8-bit value = nearest int( ( temp - base)/step )temp_step must be in the range [.1, 2.]. There are no restrictions on base_temp. The default values for base_temp and temp_step depend on temp_units. For Kelvin or Celsius, the default temp_step is .5. For Fahrenheit, the default temp_step is 1. The default values for temp_base are -80, 193, and -120 for Celsius, Kelvin, and Fahrenheit respectively.
If calibrate=yes, answer no if only the calibration report for the first 100 lines is printed. Answer yes if a calibration report is printed for every 100 lines of input data.
Valid responses are [yes or no]. The default is no.
If hrpt_type=avhrr, answer yes if the dataset is to be geometrically corrected to compensate for the curvature of the earth. Because the AVHRR sensor scans at a constant rate, pixels near the ends of scan lines are up to 5 times wider than pixels at the center of scan lines. This option redistributes the pixels along each scan line so each pixel is approximately square, about 1.1 km on a side. This involves replicating pixels near the ends of a given scan line, while discarding a few pixels at the center of the scan line. This option performs the same function as avgc.
Geometrically correcting full 2048-sample AVHRR scan lines results in output lines containing 2560 samples.
Valid responses are [yes or no]. The default is no.
If hrpt_type=avhrr. this parameter specifies the rate of subsampling in the direction of subtrack. If delta_line=3 every third line will be used. If delta_line=1, all lines will be used.
Valid responses are [1 through 8]. The default is 1.
If hrpt_type=avhrr, this parameter specifies the rate of subsampling in the scan direction. If delta_sample=3 every third sample will be used. If delta_sample=1, all samples will be used.
Valid responses are [1 through 8]. The default is 1.
Indicates that only AVHRR and TIP data corresponding to AVHRR lines that intersect a user specified latitude range will be ingested. This option is intended for use with GAC input. This parameter is OPTIONAL unless hrpt_format=gac.
Valid reponses are [yes, no]. The default is yes when hrpt_format=gac, otherwise the default is no.
Minimum and maximum latitudes, used only when use_lat_range=yes. If min_lat is greater than max_lat, they are interchanged invisible to the user.
Latitudes must be specified in standard format, e.g., 34.5 n, 34 30 n, or 34 30 00 n. See formats. There is not default for either parameter.
Used only if use_lat_range=no. Answer yes if the data is to be selected from the intersection of a master dataset (number 1 in the description above, also see master). Answer no if subsets are to be specified by start time and sample (number 2 in description above).
Valid responses are [yes, no]. The default is yes.
If use_master=yes, this is the name of the master dataset used to specify a region for data extraction (see master).
Valid responses are any TeraScan dataset that contains a map projection-based earth transform. The default is Master.
If use_master=no and use_lat_range=no, this is the start time for data extraction. The first line with a time greater than start_time is the first output line. A response of 00:00:00 instructs hrptin to begin processing at the first scan line.
Any valid time is allowed (see formats). The default is 00:00:00.
If use_master=no and use_lat_range=no and hrpt_type=tip, this is the number of records written to the output TOVS dataset. Note that these do not correspond to scan lines. If the end of the pass is detected before the output dataset is complete, the output dataset is truncated.
The valid range is [>=1]. The default is 2000.
If use_master=no and use_lat_range=no and hrpt_type=avhrr, this is the number of lines written to the output AVHRR dataset. delta_line * num_lines input lines will be processed to produce an output dataset containing num_lines. If the end of pass is detected before the output dataset is complete, the output dataset is truncated.
The valid range is [>=1]. The default is 6000/delta_line.
If use_master=no, this is the first sample to extract from the pass. Samples are numbered starting with 1.
The valid range is [1 to 2048]. The default is 1.
This parameter controls the number of samples written to the output dataset. If geo_correct=no, num_samples is the exact number of samples per line in the output dataset. If geo_correct=yes, then delta_sample * num_samples input samples will be processed to produce geometrically corrected lines that are then subsampled according to the delta_sample parameter.
The valid range is determined by [start_sample -1 + num_samples * delta_sample <= 2048]. The default is 1 + (2048 - start_sample)/delta_sample.
OPTIONAL. If ingest coincides with pass reception (e.g., the input pass is live), this parameter enables or disables reading directly from shared memory. It may be wise not to read from shared memory if writing output data is so slow that input telemetry is cycled out of shared memory before it can be read (e.g., when the output dataset is remote or when using pipe_avhrr=yes). Valid responses are [yes, no]. The default is yes.
OPTIONAL. Determines whether or not data for each AVHRR variable is contiguous in the output datasets. Valid responses are [yes, no]. The default is no, which means the output variable data is line interleaved.
OPTIONAL. Determines whether or not a progress variable called dim_counts is written to the output datasets. A progress variable has an entry for each of the five AVHRR channels; the entries are updated to reflect the number of lines written for each channel. This parameter is intended for use by scrolling applications; it must only be used when sort_by_var=yes.
Valid responses are [yes, no]. The default is no.
OPTIONAL. If UNIX stdout is not a terminal, this parameter enables writing of AVHRR data to stdout in addition to the normal output dataset. Piped data is serialized, and is generally used in conjunction with deflate and inflate functions. Valid responses are [yes, no]. The default is no.
Create two TeraScan datasets named dataset1.avhrr and dataset2.avhrr from two consecutive passes archived on tape. No processing of the data will take place. The area for data selection is specified by a master dataset.
[1] % hrptin dataset1 dataset2 hrpt_types : char( 17) ? [avhrr] on_pass_disk : char( 3) ? [yes] n tape_device : char(255) ? [/dev/nwd0] pass_number : int ( 2) ? [1] 1 2 files_per_pass : int ? [2] pass_year : int ? [1993] channels : int ( 5) ? [1 2 3 4 5] fix_missing : char( 3) ? [yes] n sharpen : char( 3) ? [yes] n calibrate : char( 3) ? [yes] n geo_correct : char( 3) ? [no] delta_line : int ? [1] delta_sample : int ? [1] use_master : char( 3) ? [yes] master_file : char(255) ? [Master] Processing pass # 1. /dev/nwd0: noaa-12 93/07/20 16:16:43.59 dataset1.avhrr: Creating ... dataset1.avhrr: Actual size is 944 lines by 511 samples. Processing pass # 2. /dev/nwd0: noaa-11 93/07/20 13:08:10.43 dataset2.avhrr: Creating ... dataset2.avhrr: Actual size is 938 lines by 569 samples.
Create two TeraScan datasets named dataset1.avhrr and dataset1.dcs from a pass archived on tape. Datasets hold avhrr and dcs data respectively. Avhrr data is sharpened and calibrated using byte output. The area of interest is selected from the beginning of the pass by specifying the time and start sample.
[2] % hrptin dataset1
hrpt_types : char( 17) ? [avhrr] avhrr dcs
on_pass_disk : char( 3) ? [yes] n
tape_device : char(255) ? [/dev/nwd0]
pass_number : int ( 2) ? [1] 1
files_per_pass : int ? [2]
pass_year : int ? [1993]
channels : int ( 5) ? [1 2 3 4 5]
fix_missing : char( 3) ? [yes]
sharpen : char( 3) ? [yes]
calibrate : char( 3) ? [yes]
byte_output : char( 3) ? [no] y
full_report : char( 3) ? [no]
geo_correct : char( 3) ? [no]
delta_line : int ? [1]
delta_sample : int ? [1]
use_master : char( 3) ? [yes] n
start_time :
Processing pass # 1.
/dev/nwd0: noaa-12 93/07/20 16:16:43.59
dataset1.dcs: Creating ...
dataset1.avhrr: Creating ...
line 100 ch slope intcpt space target prt average
1 0.10422 -4.45 42.3 1 397.59
2 0.10144 -3.99 40.6 2 404.41
3 -0.00167 1.65 990.9 620.6 3 395.00
4 -0.16336 162.02 991.8 324.3 4 398.00
5 -0.18218 181.41 995.8 317.3
dataset1.avhrr: Actual size is 3045 lines by 2048 samples.
dataset1.dcs: 923 DCS records written.
avhrr, dcs, tovs, datasets, formats, etx, avin, dcsin, tipin, avcal, avfilt3, avfix, avgc, avsharp, nitpix, satvis, gac, lac, fixline, master.
Sharpening algorithm taken from Manstein & Gesell, Proceedings of 5th AVHRR Data User's Meeting, pg 53.
The TeraScan HRPT archive format uses one 22528 byte block per HRPT minor frame, and no header blocks. The TeraScan online HRPT format has the same block structure, but the first 5 HRPT words (i.e., 10 bytes) may or may not be missing from each data block. Also, the online format may or may not be byte-swapped.
hrpt_format can take on any of the following values. Bytes refer to 8-bit integers, words refer to 16-bit integers, and long words refer to 32-bit integers. Words are numbered starting at zero. With the exception of the packed formats, 10-bit HRPT minor frame words are generally stored in 16-bit words. In the packed formats, 3 10-bit HRPT frame words are stored in one 32-bit long word (i.e., 4 bytes).
num_blocks - number of data blocks per HRPT minor frame
header_size - basic header block size
num_headers - number of header blocks
(In reality, the parameters block_size, num_blocks, header_size and num_headers can be overridden on the command line for any of the above HRPT formats; this should only be done as a last resort.)
In the above packed formats, 14800, 14848, and 15360, three 10-bit HRPT words are packed into four 8-bit bytes (really one 32-bit long word). hrptin detects whether the packing scheme is left-justified or right-justified.
If input is from tape, records of length of 22180 or less must be written as variable-length records (i.e., not as a stream of 1024-byte blocks).
If input is from tape, data files may or may not be preceded by header files. Header files are assumed to contain no useful information. If a tape has per-pass header files, the tape must be positioned at any per-pass header file prior to running hrptin. If a tape has no per-pass header files, the tape must be positioned at the beginning of any pass file prior to running hrptin.