Old FMCW Radar Processing Guide

From OPS
Jump to: navigation, search

Contents

Processing Steps

The steps for processing are:

  1. Create your parameter spreadsheet. Set the “Season Name” and “Radar Name” fields.
    1. Many of the functions use the “Season Name” and “Radar Name” to refer to the data.
  2. Make sure the Matlab GPS file is created (use a script like make_gps_2011_greenland_P3.m as a template where you have entered GPS file information for each day)
    1. The same GPS files are used for all radar systems
    2. See make_gps.m for details on the file_type, params, and gps_source fields.
    3. More details are given in the GPS section.
  3. Update param spreadsheet
    1. Make sure segments are ordered according to when they were collected. You can do this by reading the header information from the raw files using basic_load_fmcw.m or from the time stamp in the filename if it exists. If you forget to do this, the output files will need to be renamed (or deleted and recreated correctly).
    2. The filename is RADARNAME_param_SEASON.xls (e.g. snow_param_2009_Antarctica_DC8.xls)
  4. Update your personal copy of run_master (e.g. run_master_snow.m or run_master_kuband.m)
    1. Set your scheduler type
    2. Set the cluster size if necessary
  5. Run “create vectors” with run_master.m
  6. Run plot_vectors to check vector files
  7. Run “qlook” with run_master.m
  8. Make layer files with run_make_layer_files.m
  9. Run the picker or image browser with run_picker.m to check files
  10. Run “post” with run_master
  11. Create KML and CSV files from vector files with post_vectors_to_csv_kml (uses generic column in spreadsheet)
  12. Verification and GPS synchronization check with run_load_data_by_gps_time. See document on GPS Synchronization and Verification.
  13. Make sure surface data is good, correct Nyquist zone was chosen, and flag bad frames with create_frames_fmcw_accum.m
    1. If a frame was bad (i.e. no surface can be seen or the surface crossed Nyquist zones) then set processing mode to 1 (press "1") instead of 0. No further work is required.
    2. If the Surface was not tracked, then run run_make_layer_files.m to create CSARP_layerData for the bad frame (uses generic column in spreadsheet). Then correct the Surface using run_picker.m. Read the surface in from the layerData file and save it to the "Surface" variable in the echogram file. Then repost the frame.
    3. If the Nyquist zone was wrong, then set the Nyquist zone (press "n#" where # is the correct Nyquist zone). Reprocess and repost the frame.
  14. Upload KML, CSV, and JPG files
  15. At the end of the season:
    1. Run compress_echograms.m to create CSARP_qlook in CSARP_post directory (uses generic column in spreadsheet)
    2. Verify that all segments are there and the correct number of files are in each directory using check_data_products.m (uses generic column in spreadsheet)
    3. NSIDC posting guide

Command Worksheet

An example command worksheet is shown below:

Fmcw command worksheet.jpg

  1. The top three rows are for the version (e.g. "1.0"), radar name (snow, kuband, snow2, or kuband2), and the mission name (e.g. 2009_Antarctica_DC8)
  2. Each segment is listed in the first two columns in the order that it was collected. The segment ID is formed as YYYYMMDD_SS where YYYY is the year, MM is the month, DD is the day, and SS is the segment number.
  3. The "frames to process" column effects which data frames the qlook and post processes operate on.
    1. Leaving this empty causes all frames to be processed
    2. A single frame or range can be indicated, but a range should be done as [10:15] rather than 10:15 so that Excel does not interpret it as a time
  4. The "create vectors", "create records", "qlook", and "post" are binary entries (empty means zero or false) that control which processes are run on which segments. Processes are run one segment at a time, from left to right. Create vectors and create records always operate on every frame.
    1. Create vectors usually creates outputs in the csarp_support/vectors/RADARNAME/SEASONNAME/ directory.
    2. Create records usually creates outputs in the csarp_support/records/RADARNAME/SEASONNAME/ directory.
    3. Qlook usually creates outputs in the gRadar.out_path / RADARNAME / SEASONNAME / CSARP_qlook directory.
    4. Post usually creates outputs in the gRadar.out_path / RADARNAME / SEASONNAME / CSARP_post directory.
  5. The "generic" column is used by external functions to determine which segments to operate on. External functions include run_make_layer_files.m, compress_echograms.m, update_data_files.m, run_update_records.m.
  6. The "mission name" column is used for the mission name. If there is no official mission name, use one that describes the area that the measurements were taken.
  7. The "notes" column should include any issues with that segment. The phrase "do not process" has special meaning and should be included if the segment should not be posted.
  8. OTHER: Use red fill to indicate segments that should not be processed and make sure "Do not process" is included somewhere in the notes entry

Vectors Worksheet

An example vectors worksheet is shown below:

Fmcw vectors worksheet.jpg

  1. Each segment is listed in the first two columns in the order that it was collected. The segment ID is formed as YYYYMMDD_SS where YYYY is the year, MM is the month, DD is the day, and SS is the segment number. The ordering must match the command worksheet.
  2. The vectors worksheet's main purpose is to define which data files are to be included in this segment. This is the purpose of the first five columns.
    1. "file base_dirs" contains a string which should go to the base of the dataset; this string should be identical for every segment
    2. "file adc_folder_name" contains a string which gives the rest of the path (i.e. it is appended to "file base_dirs")
    3. "file prefix" contains the data filename prefix
    4. A search is performed and all files matching the above three fields is returned using get_filenames. The files to be used in the segment are then selected with the "file start_idx" and "file stop_idx" which are both 1-indexed. Generally every file is used starting with file 0000. Occasionally it is known that some files are bad (e.g. the last couple files will occasionally contain FIFO overflow garbage) and the start/stop indicies should reflect this.
  3. The "out fn" should always be left blank so the default vectors file is created (gRadar.support_path / vectors / RADARNAME / SEASONNAME / vectors_YYYYMMDD_SS.mat
  4. The "gps fn" should always be left blank so the default GPS file is used (gRadar.support_path / gps / SEASONNAME / gps_YYYYMMDD_SS.mat
  5. "utc time correction (sec)" contains the offset that will be added to the UTC time header fields in the raw data files (for a perfect radar system this field would be zero)
  6. "utc time halved" tells whether or not the UTC time in the raw data files should be halved (for a perfect radar system this field would be zero)

Qlook Worksheet

An example qlook worksheet is shown below:

Fmcw qlook worksheet.jpg

  1. Each segment is listed in the first two columns in the order that it was collected. The segment ID is formed as YYYYMMDD_SS where YYYY is the year, MM is the month, DD is the day, and SS is the segment number. The ordering must match the command worksheet.
  2. The qlook worksheet's main purpose is to control the qlook processing and the surface tracking.
  3. The "qlook output" should be left empty so the default path is used. If special processing is being done, enter a keyword like "qlook_special". These files will then end up in the CSARP_qlook_special directory.
  4. The "qlook out enable" determines whether or not files will be stored (useful for debugging).
  5. The "gps enable" flag determines whether or not GPS data will be synchronized to the data (useful for lab data where no GPS is available).
  6. The "records enable" flag determines whether or not records will be used with this file. Generally records should always be used since there is greater fidelity, but during field processing there may not be enough time to do this.
  7. "fast-time window" is a function handle to a single argument function that will provide the fftshifted window coefficients to be applied before the FFT in quick look processing (usually @hanning)
  8. "coh ave" is the number of additional post processing coherent averages (i.e. presumming or unfocussed SAR processing) that will be performed.
  9. "incoh ave" is the number of [range_bins range_lines] that will be used for multilooking (filter2 is used with boxcar window)
  10. "decimate" is the along-track decimation rate
  11. "plot mode" has 4 valid values:
    1. 0: do not plot (this is the usual mode when post processing)
    2. 1: plot echogram with y-limits around surface
    3. 2: plot full echogram
    4. 3: post echogram
  12. "good rbins" should be set to empty. However, you can hard code it. It dictates which samples from the raw data will be used in the FFT (usually the first few and last few bins are truncated because there is no valid signal there)
  13. "Nyquist zone" dictates which Nyquist zone will be used and is zero-indexed. For example, with a 125 MHz sampling frequency:
    1. 0: 0-62.5 MHz
    2. 1: 62.5-125 MHz
    3. 2: 125-187.5 MHz
    4. etc.
  14. "coh noise method" controls the coherent noise method used:
    1. 0: no coherent noise removal
    2. 1: slow-time mean is removed on a file by file basis
    3. 2: slow-time median is removed on a file by file basis
    4. 3: slow-time high pass filter using window centered on DC, argument is the window, e.g. “hanning(21)”
    5. 4: slow-time filter, argument is the filter coefficients function, e.g. “@() fir1(64,0.2,'high')”
    6. 5: multi-file mean removal where argument 1 is the number of files to consider on each side of the current file, saturated/high-received-power range lines are ignored and argument 2 specifies the threshold for saturation, a hanning window is applied to the mean from each file so that the files closest to the one being processed are weighted more heavily. The two arguments are passed in a cell array, e.g. {5,30}.
  15. "detrend poly-order" should probably be set to 0 (it can be used to remove fast-time noise floor variations, but is probably not good to use)
  16. "surf min time" the smallest time the surface tracker should be allowed to track the surface to (often just set to 0 seconds)
  17. "surf range" the jump range allowed for surface tracking (often set to [-120:120])
  18. "leverarm function" is a function handle to a valid lever arm function usually of the form lever_arm_RADARNAME_SEASONNAME_GPSSOURCE.m
    1. The lever is only applied when records files are used ("records enable" set to true)
    2. Leave this field blank if you do not want the lever arm applied

Post Worksheet

An example post worksheet is shown below:

Fmcw post worksheet.jpg

  1. Each segment is listed in the first two columns in the order that it was collected. The segment ID is formed as YYYYMMDD_SS where YYYY is the year, MM is the month, DD is the day, and SS is the segment number. The ordering must match the command worksheet.
  2. The post worksheet's main purpose is to control the posting process (creation of map and echogram image files)
  3. "in path" should match "qlook output" in the qlook worksheet
  4. "out path" should be left empty so the default directory "CSARP_post" is used. If you are doing a special posting, you can type in a keyword here such as "post_special" and the output will be stored in "CSARP_post_special"
  5. "gps en" specifies whether or not to use the GPS information in the qlook file
  6. "num-frm combine" specifies the number of frames to combine together in a single image
  7. "depth rng" is the depth range to show in the echogram. This is relative to the surface (which is automatically found during the qlook processing). For sea ice we use "[-3 4]" and for land ice where the surface may vary a lot around the average surface we access the special "Surface_Depth" variable and use "[min(Surface_Depth)-10 max(Surface_Depth)+40]"
  8. "map en" is a flag to create a map or not
  9. "map type" is the type of the map to use. Options are:
    1. none: no map is generated (same as map en is false)
    2. contour: plots without geotiff
    3. combined: plots with geotiff
    4. combined,geotiff_fn: plots with geotiff defined by fullfile(gRadar.gis_path,geotiff_fn)
    5. ascat: plots with ASCAT geotiff (use ascat_to_geotif.m script first)
  10. "vectors fn" specifies that a vectors file should be used rather than the default method which is to just look at the contents of the qlook directory
  11. "location" specifies the map location ("Arctic", "Antarctica", "Greenland", "Norway", and "Canada")
    1. Norway support is limited to Austfonna
  12. "er_ice" specifies the effective permittivity to use for the depth axis
  13. "image type" controls the print.m commands image output type
  14. "image dpi" controls the print.m commands image resolution in dots per inch
  15. "plot params" controls the images output parameters (all of these parameters which are passed in as a cell array will be sent to the figures before printing to an image file)

Radar Worksheet

An example radar worksheet is shown below:

Fmcw radar worksheet.jpg

  1. Each segment is listed in the first two columns in the order that it was collected. The segment ID is formed as YYYYMMDD_SS where YYYY is the year, MM is the month, DD is the day, and SS is the segment number. The ordering must match the command worksheet.
  2. The radar worksheet's main purpose is to contain radar parameters that are not stored in the files.
  3. "PRF" is used to check the EPRI measured in the files (discrepancies can help determine if the utc time halved field in the vectors worksheet should be set
  4. "fs" is the sampling frequency
  5. "f0" is the start frequency of the DDS
  6. "f1" is the stop frequency of the DDS
  7. "fmult" is the PLL multiplication factor
  8. "fLO" is the frequency offset from the PLL output
  9. "Tpd" is the pulse duration
  10. "ADC bits" is the number of ADC bits in the ADC
  11. "Vpp Scale" is the full scale voltage of the ADC
  12. "Time Delay" is the system time delay
  13. "Record start index" is the sample index that the data acquisition system starts recording on. Usually always 281.
  14. "presum override" leave empty to use the presum amount in the file headers. Occasionally this is in error and can be overwritten here (e.g. 2011_Austfonna_GPR taken By Kirsty Langley at University of Oslo).

Common Record Generation Problems

See Common_FMCW_radar_header_errors guide.

Personal tools
Namespaces

Variants
Actions
Navigation
Tools