Old Accumulation Radar Processing Guide

From OPS
Jump to: navigation, search

Contents

Processing Steps

The steps for processing are:

  1. Create your parameter spreadsheet. Set the “Season Name” and “Radar Name” fields.
    1. Many of the functions use the “Season Name” and “Radar Name” to refer to the data.
  2. Make sure the Matlab GPS file is created (use a script like make_gps_2011_greenland_P3.m as a template where you have entered GPS file information for each day)
    1. The same GPS files are used for all radar systems
    2. See make_gps.m for details on the file_type, params, and gps_source fields.
    3. More details are given in the GPS section.
  3. Update param spreadsheet
    1. Make sure segments are ordered according to when they were collected. You can do this by reading the header information from the raw files using basic_load_fmcw.m or from the time stamp in the filename if it exists. If you forget to do this, the output files will need to be renamed (or deleted and recreated correctly).
    2. The filename is RADARNAME_param_SEASON.xls (e.g. snow_param_2009_Antarctica_DC8.xls)
  4. Update your personal copy of run_master (e.g. run_master_snow.m or run_master_kuband.m)
    1. Set your scheduler type
    2. Set the cluster size if necessary
  5. Run “create vectors” with run_master.m
  6. Run plot_vectors to check vector files
  7. Run “qlook” with run_master.m
  8. Make layer files with run_make_layer_files.m
  9. Run the picker or image browser with run_picker_snow or run_picker_kuband to check files
  10. Run “post” with run_master
  11. Create KML and CSV files from vector files with post_vectors_to_csv_kml
  12. Verification and GPS synchronization check with run_load_data_by_gps_time. See document on GPS Synchronization and Verification.
  13. Upload KML, CSV, and JPG files
  14. At the end of the season:
    1. Run compress_echograms.m to create CSARP_qlook in CSARP_post directory
    2. Run run_make_layer_files.m to create CSARP_layerData in CSARP_post directory
    3. Verify that all segments are there and the correct number of files are in each directory using check_data_products.m
    4. NSIDC posting guide

Command Worksheet

An example command worksheet is shown below:

Accum command worksheet.jpg

  1. The top three rows are for the version (e.g. "1.0"), radar name (snow, kuband, snow2, or kuband2), and the mission name (e.g. 2009_Antarctica_DC8)
  2. Each segment is listed in the first two columns in the order that it was collected. The segment ID is formed as YYYYMMDD_SS where YYYY is the year, MM is the month, DD is the day, and SS is the segment number.
  3. The "frames to process" column effects which data frames the qlook and post processes operate on.
    1. Leaving this empty causes all frames to be processed
    2. A single frame or range can be indicated, but a range should be done as [10:15] rather than 10:15 so that Excel does not interpret it as a time
  4. The "create vectors", "create records", "qlook", and "post" are binary entries (empty means zero or false) that control which processes are run on which segments. Processes are run one segment at a time, from left to right. Create vectors and create records always operate on every frame.
    1. Create vectors usually creates outputs in the csarp_support/vectors/RADARNAME/SEASONNAME/ directory.
    2. Create records usually creates outputs in the csarp_support/records/RADARNAME/SEASONNAME/ directory.
    3. Qlook usually creates outputs in the gRadar.out_path / RADARNAME / SEASONNAME / CSARP_qlook directory.
    4. Post usually creates outputs in the gRadar.out_path / RADARNAME / SEASONNAME / CSARP_post directory.
  5. The "generic" column is used by external functions to determine which segments to operate on. External functions include run_make_layer_files.m, compress_echograms.m, update_data_files.m, run_update_records.m.
  6. The "mission name" column is used for the mission name. If there is no official mission name, use one that describes the area that the measurements were taken.
  7. The "notes" column should include any issues with that segment. The phrase "do not process" has special meaning and should be included if the segment should not be posted.
  8. OTHER: Use red fill to indicate segments that should not be processed and make sure "Do not process" is included somewhere in the notes entry

Vectors Worksheet

An example vectors worksheet is shown below:

Accum vectors worksheet.jpg

  1. Each segment is listed in the first two columns in the order that it was collected. The segment ID is formed as YYYYMMDD_SS where YYYY is the year, MM is the month, DD is the day, and SS is the segment number. The ordering must match the command worksheet.
  2. The vectors worksheet's main purpose is to define which data files are to be included in this segment. This is the purpose of the first five columns.
    1. "file base_dirs" contains a string which should go to the base of the dataset; this string should be identical for every segment
    2. "file adc_folder_name" contains a string which gives the rest of the path (i.e. it is appended to "file base_dirs")
    3. "file prefix" contains the data filename prefix
    4. A search is performed and all files matching the above three fields is returned using get_filenames. The files to be used in the segment are then selected with the "file start_idx" and "file stop_idx" which are both 1-indexed. Generally every file is used starting with file 0000. Occasionally it is known that some files are bad (e.g. the last couple files will occasionally contain FIFO overflow garbage) and the start/stop indicies should reflect this.
  3. The "out fn" should always be left blank so the default vectors file is created (gRadar.support_path / vectors / RADARNAME / SEASONNAME / vectors_YYYYMMDD_SS.mat
  4. The "gps fn" should always be left blank so the default GPS file is used (gRadar.support_path / gps / SEASONNAME / gps_YYYYMMDD_SS.mat
  5. "utc time correction (sec)" contains the offset that will be added to the UTC time header fields in the raw data files (for a perfect radar system this field would be zero)
  6. "utc time halved" tells whether or not the UTC time in the raw data files should be halved (for a perfect radar system this field would be zero)

Qlook Worksheet

An example qlook worksheet is shown below:

Accum qlook worksheet.jpg

  1. Each segment is listed in the first two columns in the order that it was collected. The segment ID is formed as YYYYMMDD_SS where YYYY is the year, MM is the month, DD is the day, and SS is the segment number. The ordering must match the command worksheet.
  2. The qlook worksheet's main purpose is to control the qlook processing and the surface tracking.
  3. The "qlook output" should be left empty so the default path is used. If special processing is being done, enter a keyword like "qlook_special". These files will then end up in the CSARP_qlook_special directory.
  4. The "qlook out enable" determines whether or not files will be stored (useful for debugging).
  5. The "gps enable" flag determines whether or not GPS data will be synchronized to the data (useful for lab data where no GPS is available).
  6. The "records enable" flag determines whether or not records will be used with this file. Generally records should always be used since there is greater fidelity, but during field processing there may not be enough time to do this.
  7. "time tukey" is the time-domain tukey window that was applied on transmit (this field should be in the radar worksheet)
  8. "band window" is a function handle to a single argument function that will provide the fftshifted window coefficients to be applied before pulse compression in quick look processing (usually @hanning)
  9. "time window" leave blank
  10. "freq window" frequency domain to apply to the whole bandwidth
  11. "coh ave" is the number of additional post processing coherent averages (i.e. presumming or unfocussed SAR processing) that will be performed.
  12. "incoh ave" is the number of [range_bins range_lines] that will be used for multilooking (filter2 is used with boxcar window)
  13. "decimate" is the along-track decimation rate
  14. "plot mode" has 4 valid values:
    1. 0: do not plot (this is the usual mode when post processing)
    2. 1: plot echogram with y-limits around surface
    3. 2: plot full echogram
    4. 3: post echogram
  15. "elev comp": ???
  16. "detrend poly-order" should probably be set to 0 (it can be used to remove fast-time noise floor variations, but is probably not good to use)
  17. "surf min time" the smallest time the surface tracker should be allowed to track the surface to (often just set to 0 seconds)
  18. "surf range" the jump range allowed for surface tracking (often set to [-120:120])
  19. "leverarm function" is a function handle to a valid lever arm function usually of the form lever_arm_RADARNAME_SEASONNAME_GPSSOURCE.m
    1. The lever is only applied when records files are used ("records enable" set to true)
    2. Leave this field blank if you do not want the lever arm applied

Post Worksheet

An example post worksheet is shown below:

Accum post worksheet.jpg

  1. Each segment is listed in the first two columns in the order that it was collected. The segment ID is formed as YYYYMMDD_SS where YYYY is the year, MM is the month, DD is the day, and SS is the segment number. The ordering must match the command worksheet.
  2. The post worksheet's main purpose is to control the posting process (creation of map and echogram image files)
  3. "in path" should match "qlook output" in the qlook worksheet
  4. "out path" should be left empty so the default directory "CSARP_post" is used. If you are doing a special posting, you can type in a keyword here such as "post_special" and the output will be stored in "CSARP_post_special"
  5. "gps en" specifies whether or not to use the GPS information in the qlook file
  6. "num-frm combine" specifies the number of frames to combine together in a single image
  7. "depth rng" is the depth range to show in the echogram. This is relative to the surface (which is automatically found during the qlook processing). For sea ice we use "[-3 4]" and for land ice where the surface may vary a lot around the average surface we access the special "Surface_Depth" variable and use "[min(Surface_Depth)-10 max(Surface_Depth)+40]"
  8. "map en" is a flag to create a map or not
  9. "map type" is the type of the map to use. Options are:
    1. none: no map is generated (same as map en is false)
    2. contour: plots without geotiff
    3. combined: plots with geotiff
    4. combined,geotiff_fn: plots with geotiff defined by fullfile(gRadar.gis_path,geotiff_fn)
    5. ascat: plots with ASCAT geotiff (use ascat_to_geotif.m script first)
  10. "vectors fn" specifies that a vectors file should be used rather than the default method which is to just look at the contents of the qlook directory
  11. "location" specifies the map location ("Arctic", "Antarctica", "Greenland", "Norway", and "Canada")
    1. Norway support is limited to Austfonna
  12. "er_ice" specifies the effective permittivity to use for the depth axis
  13. "image type" controls the print.m commands image output type
  14. "image dpi" controls the print.m commands image resolution in dots per inch
  15. "plot params" controls the images output parameters (all of these parameters which are passed in as a cell array will be sent to the figures before printing to an image file)

Radar Worksheet

An example radar worksheet is shown below:

Accum radar worksheet.jpg

  1. Each segment is listed in the first two columns in the order that it was collected. The segment ID is formed as YYYYMMDD_SS where YYYY is the year, MM is the month, DD is the day, and SS is the segment number. The ordering must match the command worksheet.
  2. The radar worksheet's main purpose is to contain radar parameters that are not stored in the files.
  3. "proc wfs" List of waveforms to process, usually all 16
  4. "PRF" is used to check the EPRI measured in the files (discrepancies can help determine if the utc time halved field in the vectors worksheet should be set
  5. "fs" is the sampling frequency
  6. "f0" is the start frequency of the DDS
  7. "fLO" is the frequency offset from the PLL output
  8. "f_step" is the step between each band
  9. "BW" is the transmit bandwidth of each chirp/band
  10. "Tpd" is the pulse duration for each chirp/band
  11. "ADC bits" is the number of ADC bits in the ADC
  12. "Vpp Scale" is the full scale voltage of the ADC
  13. "Time Delay" is the system time delay
  14. Channel weights are the weights applied to each band during processing. Note that these are in dB power and the data will be divided by this number. For example, if the value is 0.9 for band 2, then band2_data = band2_data ./ 10^(0.9/20).

Common Record Generation Problems

See FMCW Radar Processing Guide#Common Record Generation Problems

Personal tools
Namespaces

Variants
Actions
Navigation
Tools