CReSIS Toolbox Setup

From OPS
Jump to: navigation, search

Return to Main page or CReSIS Toolbox Guide.

The CReSIS toolbox is a set of Matlab, C/C++, python, and bash codes used to process, analyze and distribute radar data (especially Cryospheric radar sounder data). The version control system is Git and the main version repository is hosted at https://github.com/CReSIS/.

Please direct questions, help requests, suggestions, and error notifications to cresis_data@cresis.ku.edu.

Contents

Setup

  1. Git (All Machines): Ensure you have git installed on your machine. Please go through a tutorial on Git and review our Git page before continuing if you have not used Git before.
  2. Matlab (All Machines): Ensure you have Matlab. R2015b is the preferred version and all code should be written to be compatible with this version.
    Matlab Toolboxes that are required:
  3. Modify .bashrc file (Linux Server only): Only required for Linux if software modules need to be loaded. You will need to rerun the .bashrc setup file once you change it to have the changes take affect. One way is to logout and then login since the files are run at login. You can also run the files from the bash command line. The bash command to run it is "source ~/.bashrc".
    • Under Linux the .bashrc file is located at ~/.bashrc. From linux terminal "gedit ~/.bashrc". At KU, the Linux home directory is mounted on H: and so you also have the option to edit from windows. We recommend notepad++ for Windows. The file path from windows should be H:\.bashrc.
    • Add the following lines to the .bashrc file:
      
      # For University of Kansas CReSIS use these lines:
      module rm gcc
      module rm python
      module rm matlab
      module load matlab/2015b
      
      # For Indiana University Karst/Carbonate use these lines:
      module rm core
      module rm matlab
      module rm curl
      module load core
      module load matlab/2015a
      module load curl
      
      # For University of Kansas/Indiana University Field Processing setup use these lines:
      # NONE REQUIRED
      
      # For Alfred Wegener Institute Ollie use these lines:
      module rm matlab
      module load matlab/2015b
      module load git
      
      # Recommended lines for everywhere:
      umask 002 # Ensures that files generated will have the correct user/group permissions so that all users in our group can work with the files.
      
      alias cp="cp -i"
      alias rm="rm -i"
      alias mv="mv -i"
      alias ssh="ssh -Y"
      
    • KU and IU use modules. Modules are used to set environmental variables required by the software that we use. For more information: Modules
    • To view all available packages from the command line, run "module avail"
    • To view all packages you currently have loaded from the command line, run "module list"
    Example of setting HOME environment variable
  4. Change your home directory (Windows 7 Only at Kansas University): Open "Control Panel". Search for "User Accounts". Click on "User Accounts". In the left panel, choose "Change my environment variables". In the window that pops up, add or edit your HOME user variable to be your local home directory (e.g. C:\Users\USERNAME). See the example thumbnail on the right. Note that adding a backslash at the end of the path seems to cause git to fail; in other words "C:\Users\USERNAME" works, but "C:\Users\USERNAME\" does not.
  5. Create and/or modify .gitconfig (All Machines). The path for Linux/Mac is "~/.gitconfig". Under Windows it is usually "C:\Users\USERNAME\.gitconfig". We recommend "notepad++" to edit if using Windows. The file may already exist. We recommend adding the following contents in the git configuration file. Comment/uncomment the lines that do not apply to you and fill in the correct name and email address.
    [user]
      name = YOUR_NAME
      email = WORK_EMAIL_ADDRESS
    [core]
      autocrlf = false
      eol = lf
    [alias]
      lgb = log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset%n' --abbrev-commit --date=relative --branches
    [color]
      ui = auto
    [receive]
      denyCurrentBranch = updateInstead
    [push]
      default = simple # USE THIS LINE FOR NEW VERSIONS OF GIT >1.7.11 (try this first, if git fails then switch to tracking)
    #  default = tracking # USE THIS LINE FOR OLD VERSIONS OF GIT <1.7.11 (e.g. on KU and IU servers)
    
    # Only include these lines if using Windows
    [credential]
      helper = wincred
    
    # Only consider including these lines if using Linux and https fails
    # If certificates are not in the usual system location (may be necessary for enterprise setups: KU, IU, AWI, etc.)
    #[http]
    #  sslcainfo = /etc/pki/tls/certs/ca-bundle.crt # For KU
    #  sslcainfo = /etc/ssl/certs/ca-bundle.crt # For AWI and IU
    
    # Only include for Linux and if you like the gedit or VIM editor better than the default editor
    #[core]
    #  editor = gedit
    #  editor = vim
    
    # NOTE: Only enable/include one diff/merge tool. Here are typical choices:
    # ========================================================================
    
    # 1. Only include these lines if using TortoiseGit for diff/merge (nothing to include)
    
    # 2. Only include these lines if using meld for diff/merge
    # If using Git for Windows, but using meld instead of TortoiseGit:
    #   1. Install Meld-3.18.3-win32.msi
    #   2. Create C:\Windows\system32\meld text file with two lines:
    #     #!/bin/bash
    #     /c/Program\ Files\ \(x86\)/Meld/Meld.exe $@
    #[diff]
    #  tool = meld
    #[difftool]
    #  prompt = false
    #[merge]
    #  tool = meld
    #[mergetool]
    #  prompt = false
    
    # 3. Only include these lines if using gvimdiff for diff/merge
    #[diff]
    #  tool = gvimdiff
    #[difftool]
    #  prompt = false
    
    # 4. Only include these line if using vimdiff for diff/merge
    #[diff]
    #  tool = vimdiff
    #[difftool]
    #  prompt = false
    # ========================================================================
    
    Tortoise Git Clone Example
  6. Clone cresis-toolbox (All Machines): This copies the software repository from https://github.com/CReSIS/ to your computer and sets up a working directory where you can edit and use the software. The default branch is "stable". Developers should branch from "master" and do pull requests to master.
    Linux/Mac
    1. Log in using ssh (e.g. PUTTY or Nxclient or Thinlinc)
    2. mkdir ~/scripts/
    3. cd ~/scripts/
    4. git clone https://GITHUB_USERNAME@github.com/CReSIS/cresis-toolbox

    Windows

    1. In your Documents (sometimes called "My Documents") folder, create a directory called "scripts". This is usually located at: C:\Users\USERNAME\Documents\scripts\cresis-toolbox
    2. Navigate to this "scripts" folder
    3. git clone the repository into this directory. This can be done several ways and depends on which git client interface you use.
      TortoiseGit: Right click in the folder and choose "Git clone..." from the context menu. See Tortoise Git Clone example to the right. Replace username with your actual username and replace URL with https://GITHUB_USERNAME@github.com/CReSIS/cresis-toolbox
      git for windows: Right click in the folder and choose "Git bash here...". Then run "git clone https://GITHUB_USERNAME@github.com/CReSIS/cresis-toolbox". Replace username with your github username.
  7. Create startup.m file (All Machines): Inside the cresis-toolbox repository, copy (do not move) the file HOME/scripts/cresis-toolbox/example_startup.m to the directory returned when you run "userpath" in Matlab. Rename the file in the userpath directory to "startup.m". The startup.m file adds the cresis-toolbox to the Matlab path so you can run all the cresis-toolbox functions and it also sets the gRadar variable (which holds your default cresis toolbox settings).
  8. Set the current profile in startup.m(All Machines): Inside the "startup.m" file, set the current profile to the profile that best matches your setup. Instructions on how to do this are at the top of the startup.m file. The recommended profile for Windows and Linux/Mac are:
    • See profile list at the top of startup.m to choose the correct profile. Some institutions already have profiles defined, but may still require path changes in the profile.
    • Windows and can map the KU CReSIS network drives: Set cur_profile to 5. No path changes.
    • Linux and can mount KU CReSIS network drives: Set cur_profile to 3. No path changes. FYI: The userpath is usually "~/Documents/MATLAB/". At KU, the windows mount location for this is "H:\Documents\MATLAB\".
    • Windows: Set cur_profile to 1 and adjust paths as necessary.
    • Linux/Mac: Set cur_profile to 2 and adjust paths as necessary.

    Modify the corresponding profile settings in the startup.m if necessary. Usually this is just changing the file paths to ensure they point to the folders where you have downloaded the git repositories, the data, and the metadata.

  9. Create "matlab" folder (All Machines):
    • Under Windows: My Documents\scripts\matlab
    • Under Linux/Mac: ~/scripts/matlab

    This is your personal scratch space. You should keep temporary files which you do not plan to push to the Git repository in this space.

  10. Clone the ct_params and ct_docs repositories (All Machines): This is done the same way as cresis-toolbox, except the URLs are https://GITHUB_USERNAME@github.com/CReSIS/ct_params.git and https://GITHUB_USERNAME@github.com/CReSIS/ct_docs.git. The default branch is "stable". Developers should branch from "master" and do pull requests to master.
    • Under Windows, in Windows Explorer navigate to My Documents\scripts\. Clone in this directory so that you should have ct_params and ct_docs in your scripts directory.
    • Under Linux/Mac, change your working directory (cd) to ~/scripts/. Clone in this directory so that you should have ct_params and ct_docs in your scripts directory.
  11. Recommended Matlab Preferences (Optional):
    1. Set Matlab->Keyboard->Shortcuts: Set "Active settings" to "Windows Default Set"
    2. Set Matlab->Editor/Debugger: Uncheck "Save changes upon clicking away from a file"
    3. Set Matlab->Editor/Debugger: Uncheck "Reload unedited files that have been externally modified"
    4. Set Matlab->Editor/Debugger->Tab: Set "Tab size" and "Indent size" to 2
    5. Set Matlab->Editor/Debugger->Backup Files: Set "Location" to "Single directory" and choose a separate path such as "/tmp" in Linux/Mac or "C:\tmp" in windows.
    6. Set Matlab->General->Java Heap Memory: Set "Java Heap Size" to 2048 MB. This may not be required except for some OPS database operations like opsBulkInsert.
  12. Download the GIS data required by the toolbox (All Machines): Email cresis_data@cresis.ku.edu for URL. This is only required if the GIS data are not already available at your site. The following command may be useful to download (run from the folder where you have or want the "GIS_data" folder to be):
    wget -m -nH --cut-dirs=3 ftp://data.cresis.ku.edu/data/temp/ct/GET_FOLDERNAME_FROM_CRESIS_DATA
  13. OPS user account (OPS Write Access Only): Request an Open Polar Server account if you are using imb.picker to make edits to layers in the OPS database. This is NOT required for browsing data. At KU, email cresis_data@cresis.ku.edu for an account.

CReSIS Toolbox Development Guides

Code for Multiple Segments
Commenting style example
CReSIS toolbox command reference
Debug output conventions
Filename conventions
Function conventions
Matlab version specific code
Parameter Spreadsheet Guide
Variable naming conventions and Mathematical symbols

Cluster

Cluster Guide
Old Cluster Guide

Mex Compiler Setup

General C/C++ Coding Notes

Linear Algebra Libraries

mex -v -largeArrayDims -lmwlapack -lmwblas wb_compute_cost.cpp

 -v = verbose
 -largeArrayDims required for LAPACK and BLAS libraries

Linear Algebra PACKage (LAPACK) Information (used for matrix inverse, zgetrf and zgetri): http://www.netlib.org/lapack/explore-html/d7/d99/_v_a_r_i_a_n_t_s_2lu_2_c_r_2zgetrf_8f.html edit([matlabroot '/extern/include/lapack.h'])

Cross Platform Coding

For cross platform and 32/64 bit compatibility, use size_t and ptrdiff_t for referencing pointers.

Custom C compiler switches

Some switches must be passed through the options file rather than from the mex command line. The options file location can be found by adding -v option to the mex function call.

Windows minGW C/C++ Compiler

  1. Download and install 64 bit msys2:
    http://www.msys2.org/
    E.g. http://repo.msys2.org/distrib/x86_64/msys2-x86_64-20161025.exe
    
  2. From msys2 terminal run:
    pacman -Syu
    
  3. After quitting msys2 when it asks you to by clicking the “x” to kill the shell, open msys2 again and run this:
    pacman -Su
    
  4. Then install g++ and support tools by running:
    pacman -S --needed base-devel mingw-w64-i686-toolchain mingw-w64-x86_64-toolchain \
                       git subversion mercurial \
                       mingw-w64-i686-cmake mingw-w64-x86_64-cmake
    # https://github.com/orlp/dev-on-windows/wiki/Installing-GCC--&-MSYS2
    
  5. Then add g++ and libraries to your Windows PATH environment variable:
    C:\msys64\mingw64\bin;C:\msys64\mingw64\lib\gcc\x86_64-w64-mingw32\6.2.0
      [CHANGE "x86_64-w64-mingw32\6.2.0" to match the version you have installed… browse to the gcc lib directory to check]
    
  6. Setup Matlab to use mingw by adding this Windows environment variable:
    MW_MINGW64_LOC   C:\msys64\mingw64
    
  7. Test in Matlab and set it up (restart matlab first and then run these two commands):
    mex -setup 
    mex -setup C++
    

Linux C++ Compiler Issues

gcc version mismatch

CReSIS specific: If you get "libstdc++.so.6: version CXXABI_1.3.8' not found" you may need to unload the gcc module so that you are using the OS installed gcc which is an older version that works with Matlab r2015b. The problem is that mex will link with the new version, but when the mex file is run it will only see the matlab environment. There may be a way to get the matlab environment to see the new libraries, but that solution is not documented yet.

C++11 Option Error

To compile C++ programs, run "mex -setup C++". This creates the options file that you may need to fix this file. If you get a C++11 option error, you may be using pre-G++ 4.7. You can check the g++ version with system('g++ --version'); To fix this, add -v option to mex function and look for a line like this:

 Options file: ~/.matlab/R2015b/mex_C++_glnxa64.xml

In the options file, replace -std=c++11 with -std=c++0x (should occur in two places)

Reference: http://stackoverflow.com/questions/14674597/cc1plus-error-unrecognized-command-line-option-std-c11-with-g

Code Review

THIS PROJECT IS NOT ACTIVE.

The code review is an ongoing project to propose clean up of the code and supporting files. One task of the code review is to review all the functions in the toolbox and make sure the comments are up to date. If you help with this project, follow the link below:

docs.google.com/spreadsheet/ccc

Guides

Old Processing Guides

RDS Simulator Guide (only for mcords currently)
Old Accumulation Radar Processing Guide (accum)
Old FMCW Radar Processing Guide (OLD: kuband, kuband2, snow, snow2)

Personal tools
Namespaces

Variants
Actions
Navigation
Tools