Code for Multiple Segments

From OPS
Jump to: navigation, search

All CReSIS Toolbox Development Guides

To run FUNCTION on multiple segments, there should be:

  1. A run script called "run_FUNCTION" which can be used to run the function on multiple segments within the same season
  2. A function called "FUNCTION" with a single input parameter structure with the function specific parameters formatted like param.FUNCTION.parameters. For example, the function "collate_deconv" should have all of its function-specific parameters stored in the param struct param.collate_deconv.PARAMETERS.
  3. Cluster Only: A function called "FUNCTION_task" that takes a single parameter structure and outputs a binary success field that can be matlab compiled to run on a cluster node

To run FUNCTION on multiple seasons, there should also be:

  1. A run script called "run_all_FUNCTION" which can be used to run the function on multiple seasons

Contents

Run Script

The run script should have a format like:

% script run_FUNCTION
%
% Script for running FUNCTION
%
% Authors: FILL_IN_AS_NEEDED
%
% See also: run_FUNCTION.m, FUNCTION.m, FUNCTION_task.m, FILL_IN_AS_NEEDED

%% User Setup
% =====================================================================
param_override = [];

params = read_param_xls(ct_filename_param('rds_param_2009_Antarctica_TO.xls'));

% Syntax for running a specific segment and frame by overriding parameter spreadsheet values
%params = read_param_xls(ct_filename_param('rds_param_2009_Antarctica_TO.xls'),'20091228_01');
%params = ct_set_params(params,'cmd.generic',0);
%params = ct_set_params(params,'cmd.generic',1,'day_seg','20091224_01|20091228|20091229|20100101_02|20100103|20100104_01|20100112');
%params = ct_set_params(params,'cmd.frms',[]);

% Set override parameters using ct_set_params, by setting param_override, or by setting FUNCTION_params variable which eventually is copied to param_override.FUNCTION).
% These two examples for setting input parameters are equivalent.
% Example 1
params = ct_set_params(params,'FUNCTION.param1','first parameters');
params = ct_set_params(params,'FUNCTION.param2',[2]);
% Example 2
param_override.FUNCTION.param1 = 'first parameters';
param_override.FUNCTION.param2 = [2];

%param_override.sched.type = 'no scheduler'; % Example to override default cluster settings

%% Automated Section
% =====================================================================

% Input checking
global gRadar;
if exist('param_override','var')
  param_override = merge_structs(gRadar,param_override);
else
  param_override = gRadar;
end


% Process each of the segments
for param_idx = 1:length(params)
  param = params(param_idx);
  if ~isfield(param.cmd,'generic') || iscell(param.cmd.generic) || ischar(param.cmd.generic) || ~param.cmd.generic
    continue;
  end
  FUNCTION(param,param_override);
end

% Post process code (only include if necessary)

Function

The function should have an interface like this:

function FUNCTION(param,param_override)
% FUNCTION(param,param_override)
%
% FUNCTION DESCRIPTION
%
% param: struct with processing parameters
%         -- OR --
%         function handle to script with processing parameters
% param_override: parameters in this struct will override parameters
%         in param.  This struct must also contain the gRadar fields.
%         Typically global gRadar; param_override = gRadar;
%
% Example:
%  See run_FUNCTION.m for how to run this function directly.
%  This function may be called from the run_master.m script using the
%  param spreadsheet and the cmd.generic column.
%
% Authors: FILL_IN_AS_NEEDED
%
% See also: run_FUNCTION.m, FUNCTION.m, FUNCTION_task.m, FILL_IN_AS_NEEDED

%% General Setup
% =====================================================================

param = merge_structs(param, param_override);

fprintf('=====================================================================\n');
fprintf('%s: %s (%s)\n', mfilename, param.day_seg, datestr(now));
fprintf('=====================================================================\n');

%% Input checks
% =====================================================================

% Load frames file
frames = frames_load(param);

param.cmd.frms = frames_param_cmd_frms(param,frames);

%% Setup processing
% =====================================================================

...

THE CODE BELOW HERE IS ONLY REQUIRED FOR RUNNING ON THE CLUSTER:

%% Create and setup the cluster batch
% =====================================================================
ctrl = cluster_new_batch(param);
cluster_compile({'YOUR_TASK_FUNCTIONS.m','YOUR_TASK_FUNCTIONS.m'},ctrl.cluster.hidden_depend_funs,ctrl.cluster.force_compile,ctrl);

%% Create task
% =====================================================================

% Set the task's static parameters
sparam.argsin{1} = param; % Static parameters
sparam.task_function = 'YOUR_TASK_FUNCTION';
sparam.num_args_out = YOUR_TASK_NARGOUT;

% Set the task's dynamic parameters
dparam = [];
    
% Create success condition
dparam.success = '';

% CPU Time and Memory estimates:
dparam.cpu_time = 0;
dparam.mem = 0;
dparam.notes = sprintf('%s:%s:%s %s_%03d (%d of %d)/%d of %d USER_STUFF', ...
      mfilename, param.radar_name, param.season_name, param.day_seg, frm, frm_idx, length(param.cmd.frms), ...
      USER_STUFF);
   
ctrl = cluster_new_task(ctrl,sparam,dparam);

... AFTER ADDING ALL TASKS, THEN CREATE THE BATCH CHAIN

ctrl_chain = {{ctrl}}

Function Task

This function is only required for running on the cluster. The FUNCTION_task function should look like this:

function [success] = FUNCTION_task(param)
% [success] = FUNCTION_task(param)
%
% FUNCTION DESCRIPTION
%
% param: struct with processing parameters
%  FILL_IN_AS_NEEDED
%
% success: logical scalar (1 for success, function will throw an
%   error with an error message on failure)
%
% Example:
%  Normally this function is called from run_FUNCTION.m using the param spreadsheet.
%
% Authors: FILL_IN_AS_NEEDED
%
% See also: run_FUNCTION.m, FUNCTION.m, FUNCTION_task.m, FILL_IN_AS_NEEDED

return 1 for success and run "error('MESSAGE')" on failure

Run All Script

The run all (or multiple) seasons script should have a format like:

% script run_all_FUNCTION

% script run_all_FUNCTION
% run_all_FUNCTION
%
% DESCRIPTION
%
% OUTPUTS
%   
% Author: AUTHORS
%
% See also: LIST RELATED MATLAB M-FILES

%% User Settings
% =========================================================================

run_all;

% Set parameters for the function
param_override = [];
param_override.FUNCTION.FIELDS = VALUES;

%% Automated Section
% =========================================================================
% Input checking
global gRadar;
if exist('param_override','var')
  param_override = merge_structs(gRadar,param_override);
else
  param_override = gRadar;
end

% INITIALIZATION HERE

%% Loop to process each season
for param_idx = 1:length(param_fns)
  
  % INITIALIZATION HERE
  
  % Read in parameter spreadsheet
  param_fn = ct_filename_param(param_fns{param_idx});
  fprintf('Reading %s\n', param_fn);
  params = read_param_xls(param_fn,'');
  
  if isempty(params)
    continue;
  end

  % Run all segments (except "do not process")
  params = ct_set_params(params,'cmd.generic',1);
  params = ct_set_params(params,'cmd.generic',0,'cmd.notes','do not process');
  
  %% Load each segment
  for param_idx = 1:length(params)
    param = params(param_idx);
    
    if ~isfield(param.cmd,'generic') || iscell(param.cmd.generic) || ischar(param.cmd.generic) || ~param.cmd.generic
      % Possibly include message about not processing: fprintf('%s: Do not process\n', param.day_seg);
      continue;
    end
    
    % INITIALIZATION HERE

    % RUN THE FUNCTION
    [outputs] = FUNCTION(param,param_override);

    % POST PROCESSING
  end
  % MORE POST PROCESSING
end
% MORE POST PROCESSING

Personal tools
Namespaces

Variants
Actions
Navigation
Tools