Parameter Spreadsheet Guide

From OPS
Jump to: navigation, search

All CReSIS Toolbox File Guides | All CReSIS Toolbox Development Guides

Keywords:
param, ct_params, params, spreadsheet, xls, param spreadsheet

Contents

Overview

The parameter spreadsheet controls the processing pipelines for the CReSIS toolbox. It contains radar and processing parameters for each segment from a season. Each sensor for each season has its own parameter spreadsheet. The spreadsheet provides a way to store, manipulate, and document parameter structure arrays. Technically the processing parameters could just be stored in a .m file that generates the parameters or a .mat file which stores the parameter structures, but the spreadsheet format allows for convenient manipulation and documentation of the fields. Currently the ability to write parameter spreadsheets using the toolbox is limited and only reading is fully supported.

Filename

The standard parameter spreadsheet filename is:
GENERIC_SENSOR_NAME_param_SEASON_NAME.xls where

GENERIC_SENSOR_NAME
is one of 'accum, 'kaband', 'kuband', 'rds', or 'snow'
SEASON_NAME
is the season name in YYYY_LOCATION_PLATFORM format where YYYY is the year, LOCATION indicates the approximate location (usually Greenland, Arctic, Alaska, or Antarctica), and PLATFORM is the aircraft or similar name (e.g. P3 for NASA P3B, C130 for NASA C130, DC8 for NASA DC8, TO for Kenn Borek Twin Otter, Polar6 for AWI's Polar6 Basler, TOnrl or NRL's Twin Otter from Twin Otter International Incorporated, etc).

Format

The cmd worksheet is the only nonstandard format for a worksheet. All of the other worksheets use the top row to specify the variable name that will be used in Matlab and the second row specifies the type. Variable names can be structures (e.g. echo.elev_comp creates a variable echo with field elev_comp). Variable types are "r" for Matlab command, "t" for text, and "b" for boolean. To create a structure array that is defined over multiple columns, use the format ar:STRUCTNAME for the type. To create a cell array that is defined over multiple columns, use the ac:CELLNAME for the type. Since the first two rows specify the field information for the corresponding column, the order of the columns in the worksheet is unrestricted except that the first column is for the Date and the second column is for the Segment. Also, columns can be omitted from a sheet as long as there is a default defined in the processing functions that use the worksheet or the missing fields are set manually before the processing functions are called.

The standard worksheets are:

Field Description
cmd The command worksheet.
records The records worksheet.
qlook The records worksheet.
sar The records worksheet.
array The records worksheet.
radar The records worksheet.
post The records worksheet.
analysis_noise The analysis worksheet for coherent noise estimation.
analysis_equal The analysis worksheet for multichannel equalization (transmit and receive).
analysis_spec The analysis worksheet for specular target detection (used for deconvolution).
analysis_stat The analysis worksheet for extracting statistics about the data.

The parameter spreadsheet types are described below. The default is used when the value of the field is empty for a particular segment. Although "r" could be used for logical fields, it is better documentation to use the boolean/logical field. Also, it is often convenient to have empty fields be turned into false (e.g. when setting the command worksheet) which is how type "b" works and is why boolean is used on the command worksheet.

Type Description Default
r Any Matlab command; contents passed to eval.m. Empty matrix
t Any string (should not include single quotes). Empty string
b Binary or logical Matlab command; contents passed to eval.m and then converted to logical. false

Struct Array

If the struct array has length greater than one, then one column with name "size" and type "ar:STRUCTNAME" must be set before any columns for the array. The contents of the size field should be a numeric giving the length of the struct array. For example, a name of "size" and type ar:wfs creates a structure named "wfs" and if the contents for a segment are set to 2, then a two dimension struct array called wfs is created. If size is not declared, the default is length 1. All fields after "size" use the type field set to aT:NAME(INDEX) to fill in the struct at index INDEX with type "T" and the variable name field specifies the fieldname within that structure. The type fields are the same as regular columns (type T can be "r", "t", or "b"). The radar worksheet has an example where the struct array field "wfs" is created. For example, row 1 name "Tpd" and row 2 type "ar:wfs(2)" would set the field wfs(2).Tpd.

Cell Array

Behaves in the same was as a struct array except "c" is used instead of "a". So "ac:CELLNAME" is the format. The primary difference between a cell array and a struct array is that a cell array does not have to have the same fields at every index whereas a struct array requires every index to have the same fields and unset fields are left as empty matrices.

Field Names

Field names must be a valid Matlab command that can be executed. Typical examples are "adc_bits", "Tpd". However, more complicated fieldnames are possible to: "frames.geotiff_fn", "data_map{1}", "data_map{2}", "wfs(1).Tpd", "('my field name with spaces').field" and so on. Although struct and cell arrays can be defined this way, if the length of the struct array or cell array varied across segments, then the methods defined in Struct Array and Cell Array should be used because they allow a size field to be set.

Matlab Reader

The Matlab reader for the parameter spreadsheets is read_param_xls.m. It returns a structure array where each element of the structure array corresponds to a segment.

To read the parameter spreadsheet, use read_param_xls.m. For example:

params = read_param_xls(ct_filename_param('snow_param_2017_Arctic_Polar5.xls'));

To load specific segments, use the second argument as a regular expression:

% Uses the regular expression regexpi function in Matlab to select all the segments from 20170406 and 20170408.
params = read_param_xls(ct_filename_param('snow_param_2017_Arctic_Polar5.xls','2017040[68]'));

% Uses the regular expression regexpi function in Matlab to select all the segments from 20150913.
params = read_param_xls(param_fn,'20150913');

% Uses the regular expression regexpi function in Matlab to select only the segment 20150913_02.
params = read_param_xls(param_fn,'20150913_02');

To load custom worksheets, use the third argument. Each row in the third argument cell array specifies a worksheet to load. The first column is the name of the worksheet and the second column is the field name in the param structure. For example, this would load analysis_equal from the parameter spreadsheet and it would be accessed by param.analysis.*:

params = read_param_xls(ct_filename_param('rds_param_2011_Greenland_P3.xls'),'',{'analysis_equal' 'analysis'});

Manipulating the Parameters from Matlab

Functions are run and controlled by setting the parameter worksheet or by adjusting the parameter structures in matlab. There are several ways to override or modify the parameters stored in the parameter spreadsheets. Most functions support an argument called param_override that is a structure whose contents will the param structure contents. The params variable loaded with read_param_xls.m can be modified directly or the ct_set_params.m function can be used. The ct_set_params.m function is useful for setting whole sets of segments at once. Some common examples are:

Turns on cmd.generic for all segments which do not have "do not process" in the cmd.notes.

params = ct_set_params(params,'cmd.generic',1,'cmd.notes','^((?!do not process).)*$');

Turns on cmd.generic for all segments and then turns off those which do not have "do not process" in the cmd.notes.

params = ct_set_params(params,'cmd.generic',1);
params = ct_set_params(params,'cmd.generic',0,'cmd.notes','do not process');

Turns on cmd.generic for all segments with "2-18" in the cmd.notes and then turns off those which do not have "do not process" in the cmd.notes.

params = ct_set_params(params,'cmd.generic',1,'cmd.notes','2-18');
params = ct_set_params(params,'cmd.generic',0,'cmd.notes','do not process');

Matlab Reader Standards Guide

For boolean fields, a 0 is interpreted the same as a cell left blank

You may have to use the tic mark to proceed certain fields to force Excel to treat them like text. A common example is when a directory path is a number that needs to be interpretted as a string.

When inputting a vector like [10:14] be sure to include brackets because 10:14 will be interpretted as a time by Excel.

General reminder: use Linux file separators for all paths.

Color Coding

Please avoid putting any other color coding schemes or notes in the parameter spreadsheets when checking them in (i.e. try to make sure final revisions of spreadsheets are clean).

Mandatory Color Coding

Please do not use orange, yellow, or red fill unless it is for one of the following purposes.

ORANGE FILL on the command sheet for "do not process" segments. Fill in the frms to generic column. Use the standard Excel orange which is Red: 255, Green: 204, Blue: 0. (newer Excel uses 255,192,0 for orange)

YELLOW FILL for parameters that are out of sync with all the other parameters or to draw attention to issues.

DO NOT USE RED FILL anywhere because this is the color Excel uses for showing differences between parameter spreadsheets and the RED FILL from the differencing will be confused with any RED FILL that is in the sheet.

Please follow this example:

Fmcw command worksheet.jpg

Conversion from older formats to 4.0

all worksheets

  1. Remove debug_level
  2. Header rows standardized
    • top row is variable name
    • bottom row is variable type
    • Ensure "1" is in the top row (C1 on the cmd sheet and A1 for the rest of the sheets)
    • Ensure other header fields Date, YYYYMMDD, Segment are present

cmd

  1. Worksheet name changed from command to cmd
  2. Ensure Version, Radar, and Season fields are present
  3. Change the version string to '4.0 (the tic in front ensures Excel treats it as a string)
  4. Make sure column names are these names/types and in this order:
    • frms type r
    • records type b
    • qlook type b
    • sar type b
    • array type b
    • generic type r
    • mission_names type t
    • notes type t
  5. Ensure that all bad segments are turned into orange fills from frms to generic and "do not process" is in notes field
    • If row is red filled or has "do not post" or "do not process" in the notes, then assume it is "do not process" and change it
    • If yellow, check for the existence of files on dataproducts/public/data.
      • If not there, then assume "do not process" and change it
      • If it is there, remove yellow fill
  6. Clear regular text from command area

general (no longer used)

If worksheet has "general" or is missing one of the standard worksheets, email cresis_data@cresis.ku.edu to get help to remove "general" worksheet and convert to "radar" worksheet.

vectors (no longer used)

Remove the "vectors" worksheet, but copy columns to the "records" worksheet first.

records

  1. file_version: Rename to file.version and make sure it is set for all segments. Consult Raw_File_Guide to see what version to use.
  2. Remove records_fn, frames_fn
  3. Remove force_all ONLY if it is empty for every segment. Email cresis_data@cresis.ku.edu if not for help.
  4. Remove tmp_fn_uses_adc_folder_name
  5. Remove use_ideal_epri if it is empty or false for every segment. Email cresis_data@cresis.ku.edu if not for help.
  6. Add file.boards field. This is {'board0','board1','board2','board3'} for MCoRDS P3 missions. Single channel systems are usually {}. Multichannel systems may often be of the form {'chan1','chan2', ... ,'chan8'}. This should be a cell array specifying the file directories that are written to by the radar.
  7. Ensure file.start_idx, file.stop_idx, file.base_dir are all defined. No changes should be required.
  8. Change file.adc_folder_name to file.board_folder_name. Replace "board%b" with "%b" and "chan%d" with "%d".
  9. Replace file.file_prefix with file.prefix
  10. Add file.clk field if the data files have fields that use clock counts. For example, most Arena RSS systems require 10e6 for file.clk. Most NI systems require some factor of the sampling clock. For example, MCoRDS on the P3 is 1e9/9.
  11. Remove gps_fn, out_fn, gps_verification
  12. Remove gps.utc_time_halved ONLY if it is empty OR zero for every segment (generally only old kuband/snow will have this set to true). Leave this field as is otherwise.
  13. Remove file.radar_num
  14. Remove file.adcs
  15. Change frame_mode to frames.mode.
  16. Change geotiff_fn to frames.geotiff_fn.

get_heights (now qlook)

  1. Rename worksheet from get_heights to qlook
  2. Remove frm_types
  3. Remove coh_noise_method
  4. Remove coh_noise_arg
  5. Remove ft_wind
  6. Remove ft_wind_time
  7. Remove pulse_rfi.*
  8. Remove roll_correction
  9. Remove lever_arm_fh
  10. Remove elev_correction
  11. Remove surf.*
  12. Remove qlook.en
  13. Clear qlook.out_path and change to out_path
  14. Rename all qlook.* fields so that the qlook struct part is removed. For example qlook.out_path becomes out_path and qlook.img_comb becomes img_comb.
  15. If img_comb is empty for all segments, delete the column. For example, this column is always empty for kuband radar systems.
  16. Change img_comb to 3N format (so field should be 0, 3, 6, etc. elements in length where N is the number of imgs defined).
    • For three images defined in imgs field, a typical case would be: [3e-06 -Inf 1e-06 1e-05 -Inf 3e-06]
    • E.g. [3e-6 1e-6 4e6- 10e-6] should become [3e-6 -inf 1e-6 4e-6 -inf 10e-6]
    • E.g. [0.00001]: This is the really old format. Copy the "imgs" field and the img_comb" fields from the combine worksheet in this case.
  17. Remove B_filter
  18. Rename decimate_factor with dec
  19. Rename inc_ave with inc_dec
  20. Add surf.profile. Choose one the profiles listed in Surface tracker.

csarp (now sar)

  1. Rename worksheet from csarp to sar
  2. Remove frm_types
  3. Remove chunk_overlap
  4. Remove frm_overlap
  5. Remove coh_noise_removal
  6. Remove time_of_full_support
  7. Remove combine_rx if a single channel system or if field season is over
  8. Remove pulse_rfi.*
  9. Remove trim_vals
  10. Remove pulse_comp
  11. Remove ft_dec
  12. Remove ft_wind
  13. Remove ft_wind_time
  14. Remove lever_arm_fh
  15. Remove mocomp.type if it is 2 for all rows
  16. Remove mocomp.filter if it is {@butter,2,0.1} for all rows
  17. Remove mocomp.uniform.en if it is true for all rows
  18. Remove st_wind
  19. Remove refraction_method
  20. Remove skip_surf
  21. Remove start_range_bin_above_surf
  22. Remove start_range_bin end_range_bin
  23. Clear out_path
  24. Change sar_type from 'f-k' to 'fk'

combine (now array)

  1. Rename worksheet from combine to array
  2. Remove array_path
  3. Remove frm_types
  4. Clear in_path and out_path
  5. Change img_comb from 2N format to 3N format (so 0, 3, 6, etc.) [3e-06 -Inf 1e-06 1e-05 -Inf 3e-06]
    • E.g. [3e-6 1e-6 4e6- 10e-6] should become [3e-6 -inf 1e-6 4e-6 -inf 10e-6]
  6. Change rline_rng to line_rng
  7. Change three_dim.en to tomo_en
  8. Change Nsig to Nsrc
  9. Remove Nsrc if a single channel system
  10. Remove Nsv if a single channel system
  11. Remove tomo_en if a single channel system

radar

This worksheet may require help from cresis_data@cresis.ku.edu.

  1. Add lever_arm_fh column of type r. Set to "@lever_arm" for every segment.
  2. If wfs(wf).BW is defined and has two elements, remove the second element of the wfs(wf).BW field and add a new wfs(wf).ft_dec field and set it to "[N,D] = rat(BW(2) / fs); ft_dec = [N,D];"
    • For example, if BW = [21e6 30e6] and fs = 120e6, then BW should be [21e6] and ft_dec should be "[N,D] = rat(30e6/120e6)" which gives ft_dec = [1 4].
  3. Ensure that channel equalization fields are named wfs(wf).chan_equal_dB, wfs(wf).chan_equal_deg, wfs(wf).Tsys. For example, this means row 1 is "chan_equal_dB" and row 2 is "ar:wfs(WF)" where WF is a number representing the struct array index.
  4. Change adc_gains to adc_gains_dB. Change the fields to reflect this. For example "10.^((72-25*ones(1,16))/20);" becomes "72-25*ones(1,16)"
  5. Change DDC_mode to DDC_dec. For MCoRDS5/1600 MHz sampling frequency system: DDC_mode of 1 means set DDC_dec to 4. DDC_mode of 2 means set DDC_dec to 8. DDC_mode of 0 means set DDC_dec to 1.

post

  1. Remove in_path
  2. Remove img_type
  3. Remove img_dpi
  4. Remove echo.colorbar
  5. Remove echo.plot_params
  6. Remove echo.surf_filt_en, echo.detrend, echo.filter
  7. Rename echo.depth_axis to echo.depth
  8. Clear out_path
  9. For rds, set echo.elev_comp to 3
  10. For rds, set echo.depth to this format:
    • From "[min(Surface_Elev)-3500 max(Surface_Elev)+100]" to "[publish_echogram_switch(Bbad,0.25,Surface_Elev,-3500,DBottom,-100),max(Surface_Elev+100)]
    • For values different than -3500 as in "min(Surface_Elev)-3500", consider just changing all to -3500 unless you are certain the particular segment has a maximum ice thickness that is different than -3500 meters.
  11. Add ops.en, type "r" and set to true for rds assuming the layer data is stored in the OPS database, false otherwise.
  12. Add ops.location, type "t" and set to arctic or antarctic
  13. For rds, add ops.layers, type "r" and set to {'bottom','surface'}
  14. For rds, add ops.gaps_dist, type "r" and set to [300 60]

Matlab Excel Spreadsheet Reader Errors

Do not use "false" or "true" since Matlab cannot read fields with these keywords. You can use them only if you put a tic ' in front of them so they are treated as text.

For some reason the first rows/columns can get truncated by the Matlab reader. The Matlab reader returns a num matrix and a text matrix (each corresponding to the worksheet's cells). Occasionally the num and text matrix will be smaller than they should be because the first few rows or columns were not included. The behavior is sporadic, but only occurs when there are no numbers or no text in the first rows/columns of the worksheet. To avoid this problem, make sure the first row and first column of every worksheet contain a cell with a number and a cell with text.

Do not use special punctuation anywhere (e.g. try to restrict yourself to hyphens, commas, colons, and periods).

Always save in Windows 95 format.

Do not use the Open Office software since this seems to corrupt the sheet in a way that prevents Matlab from reading it.

The more content you put into a worksheet, the more likely it is to have reading problems. Avoid lenghly notes or fields in the main sheet and put them somewhere else if possible.

Even with all these precautions, you may occasionally lose the ability to load a worksheet in an spreadsheet. If that happens, make a new sheet a copy and past the contents of the old sheet into that one.


Personal tools
Namespaces

Variants
Actions
Navigation
Tools