# Analysis

Back to CReSIS Toolbox Guide Calibration and In Flight Monitoring section.

## Contents |

# Overview

The analysis.m function is designed to analyze the data to produce intermediate data products for understanding the performance of the radar for use in other processing steps. Each data segment is broken into processing blocks (the size is determined by analysis.block_size). Each processing block is analyzed by analysis_task.m which loads the raw data for all the wf,adc pairs for each image (one image at a time). Therefore it is possible to write analyze commands which compare various wf-adc pairs, but it is not currently possible to compare different images. The loaded wf-adcs are then processed with various commands. Multiple commands may be specified at a time. Commands are passed in as a cell array structure: analysis.cmd{1,...,num_cmds}. The temporary outputs for each command are stored in a directory CSARP_analysis_tmp/YYYYMMSS_DD under the filenames with this format: cmd_wf_W_adc_A_BLOCK_RANGE.mat.

The analysis processing guide is one part of the CReSIS Toolbox Guide.

analysis.m can be run from run_analysis.m and from run_master.m using the generic column.

## Analysis Commands

Available commands are:

Command cmd.method |
Temp Files | Description |
---|---|---|

burst_noise | burst_noise_wf_W_adc_A_start_rec_stop_rec.mat | Detects burst noise. Used by run_collate_burst_noise.m. |

coh_noise | coh_noise_wf_W_adc_A_start_rec_stop_rec.mat | Estimates coherent noise. Used by run_collate_coh_noise.m. |

qlook | qlook_img_II_start_rec_stop_rec.mat | Create quick look data product including surface tracking. NOT IMPLEMENTED. |

saturation | saturation_wf_W_adc_A_start_rec_stop_rec.mat | Finds waveforms with the maximum value in a specified time gate. NOT IMPLEMENTED. |

specular | specular_wf_W_adc_A_start_rec_stop_rec.mat | Finds specular waveforms. Used by run_collate_deconv.m |

statistics | statistics_wf_W_adc_A_start_rec_stop_rec.mat | Extracts statistics. Statistics functions are passed in as a cell array of inline functions. Used by run_collate_stat_noise.m, run_collate_stat_noisePSD.m. |

waveform | cmd_CC/waveform_wf_W_adc_A_start_rec_stop_rec.mat | Extracts waveforms. Should be used with layer restriction to prevent too much data from being loaded. Used by run_collate_equal.m, run_collate_svLUT.m, and run_collate_radiation_pattern.m. |

**W** Waveform; **A** ADC

### Common Arguments

List of parameters param.analysis.* which apply to all commands:

Field | Description |
---|---|

bit_mask | Scalar that specifies the bit mask to use for marking records as bad. This field provides the mask of which bits in records.bit_mask that will cause a record to be bad. Default is 1 which includes records with header and data stream errors. Set this bit_mask to 3 to mask out stationary and bad records (useful for coherent noise estimation on ground based data that may have stationary records). This works because 3 is b11 in binary: the first bit is for bad records and the second bit is for stationary data marked with gpr_find_bad_records.m. |

block_size | Specifies the block size in range lines (records). This is the number of range lines that will be processed per task. Usually this is set based on desired memory usage for each task. |

cmd | Specifies the cell array of commands. See below for list of fields. |

imgs | Specifies the images and the wf-adc pairs in each image to load. |

out_path | Specifies the output directory, CSARP_$out_path. Default is "analysis". |

presums | Default is 1 which is no presumming on load. Defines the number of presums to apply during loading. Generally this should be 1 unless the data rate is greater than the Nyquist criterion for along-track sampling and the loading needs to be sped up. |

### Common Arguments to All Commands

List of arguments that are valid for all commands in param.analysis.cmd{CC}.* struct:

Field | Description |
---|---|

B_filter | Default is [1] when dec = 1, otherwise the default is hanning(dec*2+1). Defines the feed-forward coefficients to use for the along-track averaging FIR filter which will be applied to coherent data (equivalent to "B" argument to Matlab's filter.m function). Feedback coefficients (the "A" argument to Matlab's filter.m function) are not used. The B_filter coefficients are always normalized to sum to one so that they are an estimator of the mean. |

dec | Default is 1 which is no decimation. Integer scalar indicating the amount of decimation to be applied after the B_filter is run. |

en | Logical indicating whether or not to run the command. Default is true. |

surf_layer | Specifies the surface layer. Should be a struct formatted as an input argument to opsLoadLayers, E.g. struct('name','surface','source','layerdata') |

method | String describing the method. See cmd.method column in the table above for valid methods. |

Nt | Positive scalar integer that specifies the number of samples from each range line to load starting at the range bin specified by the "layer" field. May be a positive integer or infinity. If infinity, then all time bins to the end of the record will be returned. Padding with NaN will be necessary if the records are of unequal length. Default is infinity. |

num_sam_hint | If a cell array, it indicates the number of fast-time samples for each image after the analysis operation is done. If a numeric scalar, then each image is assumed to have this number of fast-time samples. Default is empty and the number of samples is assumed to equal the number of range bins in the record that was determined during preprocess.m. Especially for statistics commands, the number may be much smaller. For example, if a max operation is done on each range line, there will be only one sample output per range line. For another example, if 128 range bins are extracted out by the start/stop time parameters for each range line, there will only be 128 samples output per range line. In these cases, the user should specify the num_sam_hint field otherwise the cpu_time and mem estimates will be incorrect. The num_sam_hint would be something like {1} or {128}. THE FOLLOWING IS NOT IMPLEMENTED YET SINCE THE NUMBER OF SAMPLES IS HARD CODED, but for radars where the number of range bins per range line is not known ahead of time, the num_sam_hint field must be set or the analysis script will fail because the estimated number of samples will be zero. |

out_path | String specifying the output directory, CSARP_$out_path, to use. Default is usually 'analysis', but depends on the command |

wf_adcs | Cell array the same size as imgs that specifies a subset of wf-adc pair indexes for each image to use. The default is to use all wf-adc pairs in each img. |

### analysis burst_noise

This creates outputs: CSARP_analysis/burst_noise_YYYYMMDD_SS_wf_W_adc_A.mat

This is used by run_collate_burst_noise.m. This applies a constant false alarm rate (CFAR) detector on the data looking for burst noise. The range bins and range lines of the detections are stored and the full range line is stored for up to a specified number of burst noise waveforms that are detected.

waveform specific options:

Field | Description |
---|---|

threshold | Scalar numeric. Burst noise detection constant false alarm rate threshold in dB. Default is 17. This is the amount above the filtered noise floor that the filtered signal needs to be to cause a detection. |

noise_filt | 2-element vector with positive odd integers which indicate the noise filter averaging window in fast-time (first element) and slow-time (second element). Default is [11 101]. Even numbers are not allowed. The window is centered on the pixel that is being tested for burst noise. |

signal_filt | 2-element vector with positive odd integers which indicate the signal filter averaging window in fast-time (first element) and slow-time (second element). Default is [11 1]. Even numbers are not allowed. The window is centered on the pixel that is being tested for burst noise. |

valid_bins | Cell array of valid bin ranges for each image. The number of elements in the cell array should equal the number of images being processed. Each entry in the cell array should be a 2-element vector with positive integers which indicate the range bins in fast time that will be searched for burst noise. To indicate all range bins use "inf" in the second position. The default is [501 inf] which uses range bins 501 to the end of the record. |

max_bad_waveforms | Scalar nonnegative integer. Default is 100. This indicates the maximum number of burst noise waveforms that will be stored. The entire range line is stored so this field prevents too large output files. Set to inf to not have a limit. |

### analysis coh_noise

This function is used to estimate the coherent noise. The output files are read in by the function run_collate_coh_noise.m which creates the coherent noise estimates that data_pulse_compress.m uses to remove the coherent noise. Because the signal energy can be very large and may contaminate the coherent noise estimate, there is a threshold mechanism to remove samples that are too large. A two-way travel time dependent threshold is usually required for depth sounder radars and the threshold is also created by run_collate_coh_noise.m and requires repeating the coherent noise analysis twice (once to determine the threshold and once to determine the final coherent noise estimate).

This function averages "cmd.block_ave" range lines and decimates by the same number. Therefore, if the segment has with 5,000,000 records and cmd.block_ave is 1,000, there will be 5,000 averaged records output from this function.

This coherent noise estimates for each specified wf-adc pair and stored in outputs of this form:

- CSARP_analysis/coh_noise_YYYYMMDD_SS_wf_W_adc_A.mat
- YYYYMMDD_SS: segment name
- W: waveform
- A: adc

The temporary files created by analysis_task are (analysis_combine_task combines them into the final output):

- CSARP_analysis_tmp/YYYYMMDD_SS/coh_noise_wf_W_adc_A_START_STOP.mat
- YYYYMMDD_SS: segment name
- W: waveform
- A: adc
- START: start record
- STOP: stop record

This is used by run_collate_coh_noise.m to produce threshold files used by a second run of analysis coh_noise and estimates of the coherent noise used by data_pulse_compress.m:

- CSARP_analysis/coh_noise_simp_YYYYMMDD_SS_wf_W_adc_A.mat

The coh_noise specific options are:

Field | Description |
---|---|

block_ave | Default is 2000. Scalar integer indicating the number of averages to do before storing. This must be a factor of the block_size, i.e. block_size = N*block_ave where N is some integer. In the final block in a segment, the size may not be block_size since the segment length is generally not a multiple of the block_size. In this case, the incomplete block_ave is dropped/ignored. |

threshold | Can be an empty numeric, scalar numeric or a string. If threshold is an empty numeric, then no thresholding is done. If it is a scalar numeric, it indicates the signal power threshold over which bins will not be included in the coherent noise average under the assumption that they are contaminated by too much non-coherent noise energy. If threshold is a string, it indicates the path to a threshold file using ct_filename_out.m. If the string is empty, then the default is 'analysis' and the path generated by ct_filename_out.m is: CSARP_analysis/coh_noise_simp_YYYYMMDD_SS_wf_W_adc_A.mat. The "threshold" variable will be loaded from this file and be used to provide a range bin dependent threshold. The threshold file is created by run_collate_coh_noise.m. |

### analysis qlook

Creates standard quick look data products: CSARP_qlook/Data_img_II_YYYYMMDD_SS_FFF.mat CSARP_qlook/Data_YYYYMMDD_SS_FFF.mat

qlook specific options:

Field | Description |
---|---|

inc_ave | Same as param.qlook.inc_ave |

surf | Same as param.qlook.surf |

### analysis saturation

This creates outputs: CSARP_analysis/saturation_wf_W_adc_A_YYYYMMDD_SS.mat

waveform specific options:

Field | Description |
---|---|

? | ? |

### analysis specular

Analyzes the along-track peak values for specular coherence by using an along-track DFT. The command outputs a coherence peakiness (our specularity score) for all range lines. Waveforms with a peakiness (specularity score) which exceeds the threshold, will be individually extracted and returned.

This creates outputs:

- CSARP_analysis/specular_wf_W_adc_A_YYYYMMDD_SS.mat

The specular waveforms are used by run_collate_deconv.m to create waveforms for deconvolution. These deconvolution waveforms are used by data_pulse_compress.m to deconvolve the system impulse response.

- CSARP_analysis/deconv_tmp_wf_W_adc_A_YYYYMMDD_SS.mat
- CSARP_analysis/deconv_wf_W_adc_A_YYYYMMDD_SS.mat

Specular command specific options:

Field | Description |
---|---|

gps_times | N_good element vector. The DFT with a GPS time closest to each GPS time in the vector will be included regardless of its peakiness score. Default is an empty list []. |

max_rlines | Scalar integer specifying the maximum number of specular waveforms to return. If more waveforms pass the peakiness threshold test than max_rlines, then the max_rlines highest values are returned. Default is 10. If max_rlines is empty or inf, then all specular waveforms will be returned. The reason for setting this number to a small value is to prevent too many waveforms from being found and slowing down that particular task too much since each waveform must be processed individually. |

min_bin | Scalar double specifying the minimum time of the peak value. The default is 0 for radar_type='deramp' and the pulse duration, Tpd, for other radars. |

noise_doppler_bins | Specifies which along-track DFT bins to use for estimating incoherent power. Default is [12:end-11]. There are [1:specular.ave] bins that can be used with DC being bin 1 so for example: [12:117] is the default for rlines = 128. |

rlines | Numeric scalar specifying the number of bins to use in the along-track DFT. Default is 128. |

signal_doppler_bins | Specifies which along-track DFT bins to use for estimating coherent signal power. Default is [1:4 end-3:end]. There are [1:specular.ave] bins that can be used with DC being bin 1 so for example: [1:4 125:128] is the default for rlines = 128. |

threshold | Numeric scalar specifying the minimum ratio (in dB) of coherent power (signal_doppler_bins) to incoherent power (noise_doppler_bins) that is required for a waveform to be extracted. This ratio is called the peakiness in the source code. Default is 40. |

### analysis statistics

This creates outputs: CSARP_analysis_NAME/stats_YYYYMMDD_SS_wf_W_adc_A.mat

For conventions on standard statistics data products that are generated from the data, see the Signal statistics section.

Statistics specific options:

Field | Description |
---|---|

block_ave | If '1', entire data is processed with @fh at once. Otherwise, data is divided into sub blocks and @fh is run on each block for all the range lines. |

start_time | Default is -inf. Determines the first range bin to use in the analysis. If numeric scalar, units are seconds and the first range bin greater or equal to this in fast-time will be the start bin. If a structure, then it specifies a layer_param structure that loads a layer with opsLoadLayers. This structure will have eval.Tpd, eval.dt, eval.Tstart, and eval.Tend added to it for pulse duration, time step, start time of record, and end time of record respectively. If a string, then it specifies a command to run with "eval" and the variables es.Tpd, es.dt, es.Tstart, and es.Tend are available with the same meaning as the layer_param structure eval variables. |

stop_time | Default is inf. Determines the last range bin to use in the analysis. Identical to start_time, except that it specifies the last bin that will be included. |

stats | Cell array of inline functions that will be run on the data. The inline functions will be concatenated on the column/2nd dimension for each block of data. |

### analysis waveform

Creates waveform files: CSARP_analysis/NAME_YYYYMMDD_SS_img_II.mat

This is used by run_collate_equalization.m, run_generate_complex_svLUT.m and run_plot_rad_patterns.m.

waveform specific options:

Field | Description |
---|---|

chan_eq | Not used, but reserved for follow on processing. |

delay | Not used, but reserved for follow on processing. |

motion_comp | Not used, but reserved for follow on processing. |

noise_bin | Not used, but reserved for follow on processing. |

out_fn | String specifying the "NAME" portion of the waveform files. Default is 'surf'. If multiple waveform commands are to be run, then this field must be set or the output files will overwrite each other. |

ref_wf_adc | Not used, but reserved for follow on processing. |

retrack | Not used, but reserved for follow on processing. |

start_time | See analysis statistics for explanation. |

### Examples using analysis

Typical commands for pulsed radar systems in the field:

Command | Description |
---|---|

qlook | Create quick look data product including surface tracking. Functionality close to qlook.m. |

waveform | Extract surface returns from image 1 for receiver equalization, radiation pattern, complex steering vector generation. |

statistics | Extract mean power and max power statistics for image 1 surface return. {@(x) nanmean(x.*conj(x),[],1),@(x) nanmax(x.*conj(x),[],1)} |

waveform | Extract internal layers from image 2 for receiver equalization. |

statistics | Extract mean and max power statistics for image 2 internal layers. {@(x) nanmean(x.*conj(x),[],1),@(x) nanmax(x.*conj(x),[],1)} |

waveform | Extract internal layers from image 3 for receiver equalization. |

statistics | Extract mean and max power statistics for image 3 internal layers. {@(x) nanmean(x.*conj(x),[],1),@(x) nanmax(x.*conj(x),[],1)} |

statistics | Extract statistics (mean/max power, PSD) for image 3 noise: {@(x) nanmean(x.*conj(x),[],1),@(x) nanmax(x.*conj(x),[],1),@(x) mean(abs(fft(x)).^2,[],2)} |

specular | Extract specular returns from image 1 for deconvolution. |

saturation | Extract saturated returns from image 1 for monitoring saturation. Limit max search to surface time bins. |

coh_noise | Extract coherent noise from each image. |

Typical commands for single channel and single waveform FMCW radar system in the field:

cmd{1}.type = 'layer'

Command | Description |
---|---|

qlook | Create quick look data product including surface tracking. Functionality close to qlook.m. |

specular | Extract specular returns from image 1 for deconvolution. |

statistics | Extract mean power and max power statistics for image 1 surface return. {@(x) nanmean(x.*conj(x),[],1),@(x) nanmax(x.*conj(x),[],1)} |

statistics | Extract statistics (mean/max power, PSD) for image 1 noise: {@(x) nanmean(x.*conj(x),[],1),@(x) nanmax(x.*conj(x),[],1),@(x) mean(abs(fft(x)).^2,[],2)} |

saturation | Extract saturated returns from image 1 for monitoring saturation. |

coh_noise | Extract coherent noise from image 1. |