Analysis

From OPS
Jump to: navigation, search

Back to CReSIS Toolbox Guide Calibration and In Flight Monitoring section.

Contents

Overview

The analysis.m function is designed to analyze the data to produce intermediate data products for understanding the performance of the radar for use in other processing steps. Each data segment is broken into processing blocks (the size is determined by analysis.block_size). Each processing block is analyzed by analysis_task.m which loads the raw data for all the wf,adc pairs for each image (one image at a time). Therefore it is possible to write analyze commands which compare various wf-adc pairs, but it is not currently possible to compare different images. The loaded wf-adcs are then processed with various commands. Multiple commands may be specified at a time. Commands are passed in as a cell array structure: analysis.cmd{1,...,num_cmds}. The temporary outputs for each command are stored in a directory CSARP_analysis_tmp/YYYYMMSS_DD under the filenames with this format: cmd_wf_W_adc_A_BLOCK_RANGE.mat.

The analysis processing guide is one part of the CReSIS Toolbox Guide.

analysis.m can be run from run_analysis.m and from run_master.m using the generic column.

Analysis Commands

Available commands are:

Command
cmd.method
Temp Files Description
burst_noise burst_noise_wf_W_adc_A_start_rec_stop_rec.mat Detects burst noise. Used by run_collate_burst_noise.m.
coh_noise coh_noise_wf_W_adc_A_start_rec_stop_rec.mat Estimates coherent noise. Used by run_collate_coh_noise.m.
qlook qlook_img_II_start_rec_stop_rec.mat Create quick look data product including surface tracking. NOT IMPLEMENTED.
saturation saturation_wf_W_adc_A_start_rec_stop_rec.mat Finds waveforms with the maximum value in a specified time gate. NOT IMPLEMENTED.
specular specular_wf_W_adc_A_start_rec_stop_rec.mat Finds specular waveforms. Used by run_collate_deconv.m
statistics statistics_wf_W_adc_A_start_rec_stop_rec.mat Extracts statistics. Statistics functions are passed in as a cell array of inline functions. Used by run_collate_stat_noise.m, run_collate_stat_noisePSD.m.
waveform cmd_CC/waveform_wf_W_adc_A_start_rec_stop_rec.mat Extracts waveforms. Should be used with layer restriction to prevent too much data from being loaded. Used by run_collate_equal.m, run_collate_svLUT.m, and run_collate_radiation_pattern.m.

W Waveform; A ADC

Common Arguments

List of parameters param.analysis.* which apply to all commands:

Field Description
bit_mask Scalar that specifies the bit mask to use for marking records as bad. This field provides the mask of which bits in records.bit_mask that will cause a record to be bad. Default is 1 which includes records with header and data stream errors. Set this bit_mask to 3 to mask out stationary and bad records (useful for coherent noise estimation on ground based data that may have stationary records). This works because 3 is b11 in binary: the first bit is for bad records and the second bit is for stationary data marked with gpr_find_bad_records.m.
block_size Specifies the block size in range lines (records). This is the number of range lines that will be processed per task. Usually this is set based on desired memory usage for each task.
cmd Specifies the cell array of commands. See below for list of fields.
imgs Specifies the images and the wf-adc pairs in each image to load.
out_path Specifies the output directory, CSARP_$out_path. Default is "analysis".
presums Default is 1 which is no presumming on load. Defines the number of presums to apply during loading. Generally this should be 1 unless the data rate is greater than the Nyquist criterion for along-track sampling and the loading needs to be sped up.

Common Arguments to All Commands

List of arguments that are valid for all commands in param.analysis.cmd{CC}.* struct:

Field Description
B_filter Default is [1] when dec = 1, otherwise the default is hanning(dec*2+1). Defines the feed-forward coefficients to use for the along-track averaging FIR filter which will be applied to coherent data (equivalent to "B" argument to Matlab's filter.m function). Feedback coefficients (the "A" argument to Matlab's filter.m function) are not used. The B_filter coefficients are always normalized to sum to one so that they are an estimator of the mean.
dec Default is 1 which is no decimation. Integer scalar indicating the amount of decimation to be applied after the B_filter is run.
en Logical indicating whether or not to run the command. Default is true.
surf_layer Specifies the surface layer. Should be a struct formatted as an input argument to opsLoadLayers, E.g. struct('name','surface','source','layerdata')
method String describing the method. See cmd.method column in the table above for valid methods.
Nt Positive scalar integer that specifies the number of samples from each range line to load starting at the range bin specified by the "layer" field. May be a positive integer or infinity. If infinity, then all time bins to the end of the record will be returned. Padding with NaN will be necessary if the records are of unequal length. Default is infinity.
num_sam_hint If a cell array, it indicates the number of fast-time samples for each image after the analysis operation is done. If a numeric scalar, then each image is assumed to have this number of fast-time samples. Default is empty and the number of samples is assumed to equal the number of range bins in the record that was determined during preprocess.m. Especially for statistics commands, the number may be much smaller. For example, if a max operation is done on each range line, there will be only one sample output per range line. For another example, if 128 range bins are extracted out by the start/stop time parameters for each range line, there will only be 128 samples output per range line. In these cases, the user should specify the num_sam_hint field otherwise the cpu_time and mem estimates will be incorrect. The num_sam_hint would be something like {1} or {128}. THE FOLLOWING IS NOT IMPLEMENTED YET SINCE THE NUMBER OF SAMPLES IS HARD CODED, but for radars where the number of range bins per range line is not known ahead of time, the num_sam_hint field must be set or the analysis script will fail because the estimated number of samples will be zero.
out_path String specifying the output directory, CSARP_$out_path, to use. Default is usually 'analysis', but depends on the command
wf_adcs Cell array the same size as imgs that specifies a subset of wf-adc pair indexes for each image to use. The default is to use all wf-adc pairs in each img.

analysis burst_noise

This creates outputs: CSARP_analysis/burst_noise_YYYYMMDD_SS_wf_W_adc_A.mat

This is used by run_collate_burst_noise.m. This applies a constant false alarm rate (CFAR) detector on the data looking for burst noise. The range bins and range lines of the detections are stored and the full range line is stored for up to a specified number of burst noise waveforms that are detected.

waveform specific options:

Field Description
threshold Scalar numeric. Burst noise detection constant false alarm rate threshold in dB. Default is 17. This is the amount above the filtered noise floor that the filtered signal needs to be to cause a detection.
noise_filt 2-element vector with positive odd integers which indicate the noise filter averaging window in fast-time (first element) and slow-time (second element). Default is [11 101]. Even numbers are not allowed. The window is centered on the pixel that is being tested for burst noise.
signal_filt 2-element vector with positive odd integers which indicate the signal filter averaging window in fast-time (first element) and slow-time (second element). Default is [11 1]. Even numbers are not allowed. The window is centered on the pixel that is being tested for burst noise.
valid_bins Cell array of valid bin ranges for each image. The number of elements in the cell array should equal the number of images being processed. Each entry in the cell array should be a 2-element vector with positive integers which indicate the range bins in fast time that will be searched for burst noise. To indicate all range bins use "inf" in the second position. The default is [501 inf] which uses range bins 501 to the end of the record.
max_bad_waveforms Scalar nonnegative integer. Default is 100. This indicates the maximum number of burst noise waveforms that will be stored. The entire range line is stored so this field prevents too large output files. Set to inf to not have a limit.

analysis coh_noise

This function is used to estimate the coherent noise. The output files are read in by the function run_collate_coh_noise.m which creates the coherent noise estimates that data_pulse_compress.m uses to remove the coherent noise. Because the signal energy can be very large and may contaminate the coherent noise estimate, there is a threshold mechanism to remove samples that are too large. A two-way travel time dependent threshold is usually required for depth sounder radars and the threshold is also created by run_collate_coh_noise.m and requires repeating the coherent noise analysis twice (once to determine the threshold and once to determine the final coherent noise estimate).

This function averages "cmd.block_ave" range lines and decimates by the same number. Therefore, if the segment has with 5,000,000 records and cmd.block_ave is 1,000, there will be 5,000 averaged records output from this function.

This coherent noise estimates for each specified wf-adc pair and stored in outputs of this form:

CSARP_analysis/coh_noise_YYYYMMDD_SS_wf_W_adc_A.mat
  • YYYYMMDD_SS: segment name
  • W: waveform
  • A: adc

The temporary files created by analysis_task are (analysis_combine_task combines them into the final output):

CSARP_analysis_tmp/YYYYMMDD_SS/coh_noise_wf_W_adc_A_START_STOP.mat
  • YYYYMMDD_SS: segment name
  • W: waveform
  • A: adc
  • START: start record
  • STOP: stop record

This is used by run_collate_coh_noise.m to produce threshold files used by a second run of analysis coh_noise and estimates of the coherent noise used by data_pulse_compress.m:

CSARP_analysis/coh_noise_simp_YYYYMMDD_SS_wf_W_adc_A.mat

The coh_noise specific options are:

Field Description
block_ave Default is 2000. Scalar integer indicating the number of averages to do before storing. This must be a factor of the block_size, i.e. block_size = N*block_ave where N is some integer. In the final block in a segment, the size may not be block_size since the segment length is generally not a multiple of the block_size. In this case, the incomplete block_ave is dropped/ignored.
threshold Can be an empty numeric, scalar numeric or a string. If threshold is an empty numeric, then no thresholding is done. If it is a scalar numeric, it indicates the signal power threshold over which bins will not be included in the coherent noise average under the assumption that they are contaminated by too much non-coherent noise energy. If threshold is a string, it indicates the path to a threshold file using ct_filename_out.m. If the string is empty, then the default is 'analysis' and the path generated by ct_filename_out.m is: CSARP_analysis/coh_noise_simp_YYYYMMDD_SS_wf_W_adc_A.mat. The "threshold" variable will be loaded from this file and be used to provide a range bin dependent threshold. The threshold file is created by run_collate_coh_noise.m.

analysis qlook

Creates standard quick look data products: CSARP_qlook/Data_img_II_YYYYMMDD_SS_FFF.mat CSARP_qlook/Data_YYYYMMDD_SS_FFF.mat

qlook specific options:

Field Description
inc_ave Same as param.qlook.inc_ave
surf Same as param.qlook.surf

analysis saturation

This creates outputs: CSARP_analysis/saturation_wf_W_adc_A_YYYYMMDD_SS.mat

waveform specific options:

Field Description
 ?  ?

analysis specular

Analyzes the along-track peak values for specular coherence by using an along-track DFT. The command outputs a coherence peakiness (our specularity score) for all range lines. Waveforms with a peakiness (specularity score) which exceeds the threshold, will be individually extracted and returned.

This creates outputs:

  • CSARP_analysis/specular_wf_W_adc_A_YYYYMMDD_SS.mat

The specular waveforms are used by run_collate_deconv.m to create waveforms for deconvolution. These deconvolution waveforms are used by data_pulse_compress.m to deconvolve the system impulse response.

  • CSARP_analysis/deconv_tmp_wf_W_adc_A_YYYYMMDD_SS.mat
  • CSARP_analysis/deconv_wf_W_adc_A_YYYYMMDD_SS.mat

Specular command specific options:

Field Description
gps_times N_good element vector. The DFT with a GPS time closest to each GPS time in the vector will be included regardless of its peakiness score. Default is an empty list [].
max_rlines Scalar integer specifying the maximum number of specular waveforms to return. If more waveforms pass the peakiness threshold test than max_rlines, then the max_rlines highest values are returned. Default is 10. If max_rlines is empty or inf, then all specular waveforms will be returned. The reason for setting this number to a small value is to prevent too many waveforms from being found and slowing down that particular task too much since each waveform must be processed individually.
min_bin Scalar double specifying the minimum time of the peak value. The default is 0 for radar_type='deramp' and the pulse duration, Tpd, for other radars.
noise_doppler_bins Specifies which along-track DFT bins to use for estimating incoherent power. Default is [12:end-11]. There are [1:specular.ave] bins that can be used with DC being bin 1 so for example: [12:117] is the default for rlines = 128.
rlines Numeric scalar specifying the number of bins to use in the along-track DFT. Default is 128.
signal_doppler_bins Specifies which along-track DFT bins to use for estimating coherent signal power. Default is [1:4 end-3:end]. There are [1:specular.ave] bins that can be used with DC being bin 1 so for example: [1:4 125:128] is the default for rlines = 128.
threshold Numeric scalar specifying the minimum ratio (in dB) of coherent power (signal_doppler_bins) to incoherent power (noise_doppler_bins) that is required for a waveform to be extracted. This ratio is called the peakiness in the source code. Default is 40.

analysis statistics

This creates outputs: CSARP_analysis_NAME/stats_YYYYMMDD_SS_wf_W_adc_A.mat

For conventions on standard statistics data products that are generated from the data, see the Signal statistics section.

Statistics specific options:

Field Description
block_ave If '1', entire data is processed with @fh at once. Otherwise, data is divided into sub blocks and @fh is run on each block for all the range lines.
start_time Default is -inf. Determines the first range bin to use in the analysis. If numeric scalar, units are seconds and the first range bin greater or equal to this in fast-time will be the start bin. If a structure, then it specifies a layer_param structure that loads a layer with opsLoadLayers. This structure will have eval.Tpd, eval.dt, eval.Tstart, and eval.Tend added to it for pulse duration, time step, start time of record, and end time of record respectively. If a string, then it specifies a command to run with "eval" and the variables es.Tpd, es.dt, es.Tstart, and es.Tend are available with the same meaning as the layer_param structure eval variables.
stop_time Default is inf. Determines the last range bin to use in the analysis. Identical to start_time, except that it specifies the last bin that will be included.
stats Cell array of inline functions that will be run on the data. The inline functions will be concatenated on the column/2nd dimension for each block of data.

analysis waveform

Creates waveform files: CSARP_analysis/NAME_YYYYMMDD_SS_img_II.mat

This is used by run_collate_equalization.m, run_generate_complex_svLUT.m and run_plot_rad_patterns.m.

waveform specific options:

Field Description
chan_eq Not used, but reserved for follow on processing.
delay Not used, but reserved for follow on processing.
motion_comp Not used, but reserved for follow on processing.
noise_bin Not used, but reserved for follow on processing.
out_fn String specifying the "NAME" portion of the waveform files. Default is 'surf'. If multiple waveform commands are to be run, then this field must be set or the output files will overwrite each other.
ref_wf_adc Not used, but reserved for follow on processing.
retrack Not used, but reserved for follow on processing.
start_time See analysis statistics for explanation.

Examples using analysis

Typical commands for pulsed radar systems in the field:

Command Description
qlook Create quick look data product including surface tracking. Functionality close to qlook.m.
waveform Extract surface returns from image 1 for receiver equalization, radiation pattern, complex steering vector generation.
statistics Extract mean power and max power statistics for image 1 surface return. {@(x) nanmean(x.*conj(x),[],1),@(x) nanmax(x.*conj(x),[],1)}
waveform Extract internal layers from image 2 for receiver equalization.
statistics Extract mean and max power statistics for image 2 internal layers. {@(x) nanmean(x.*conj(x),[],1),@(x) nanmax(x.*conj(x),[],1)}
waveform Extract internal layers from image 3 for receiver equalization.
statistics Extract mean and max power statistics for image 3 internal layers. {@(x) nanmean(x.*conj(x),[],1),@(x) nanmax(x.*conj(x),[],1)}
statistics Extract statistics (mean/max power, PSD) for image 3 noise:
{@(x) nanmean(x.*conj(x),[],1),@(x) nanmax(x.*conj(x),[],1),@(x) mean(abs(fft(x)).^2,[],2)}
specular Extract specular returns from image 1 for deconvolution.
saturation Extract saturated returns from image 1 for monitoring saturation. Limit max search to surface time bins.
coh_noise Extract coherent noise from each image.

Typical commands for single channel and single waveform FMCW radar system in the field:

cmd{1}.type = 'layer'

Command Description
qlook Create quick look data product including surface tracking. Functionality close to qlook.m.
specular Extract specular returns from image 1 for deconvolution.
statistics Extract mean power and max power statistics for image 1 surface return. {@(x) nanmean(x.*conj(x),[],1),@(x) nanmax(x.*conj(x),[],1)}
statistics Extract statistics (mean/max power, PSD) for image 1 noise:
{@(x) nanmean(x.*conj(x),[],1),@(x) nanmax(x.*conj(x),[],1),@(x) mean(abs(fft(x)).^2,[],2)}
saturation Extract saturated returns from image 1 for monitoring saturation.
coh_noise Extract coherent noise from image 1.
Personal tools
Namespaces

Variants
Actions
Navigation
Tools