CReSIS Toolbox Setup
The CReSIS toolbox is a set of Matlab, C/C++, python, and bash codes used to process, analyze and distribute radar data (especially Cryospheric radar sounder data). The version control system is Git and the main version repository is hosted at https://github.com/CReSIS/.
Please direct questions, help requests, suggestions, and error notifications to firstname.lastname@example.org.
- Git (All Machines): Ensure you have git installed on your machine. Please go through a tutorial on Git and review our Git page before continuing if you have not used Git before.
- Matlab (All Machines): Ensure you have Matlab. R2015b is the preferred version and all code should be written to be compatible with this version.
Matlab Toolboxes that are required:
- The Signal Processing Toolbox is required for processing and many visualization tools (e.g. some operations in imb.picker).
- The Image Processing Toolbox is required for processing and many visualization tools (e.g. some operations in imb.picker).
- The Mapping Toolbox is required for imb.picker, processing, posting and a few other operations (e.g. some operations in imb.picker).
- The Optimization Toolbox is required for parametric estimation in 3D imaging (array_proc.m), but not beam forming based 3D imaging. Most users will not need this toolbox.
- The Matlab Compiler is required for Torque/Slurm clusters only when a code change is made to the code running on the cluster. Most users will not need this toolbox.
- Occasionally code is accidentally committed which uses Other Toolboxes; please email email@example.com if this occurs.
- Other (All Machines): Some operations requires access to "wget" for downloading directories from the web and ghostscript "gs" to concatenate pdf files. To use the commands you will need to have them available in your path.
- Modify .bashrc file (Linux Server only): Only required for Linux if software modules need to be loaded. You will need to rerun the .bashrc setup file once you change it to have the changes take affect. One way is to logout and then login since the files are run at login. You can also run the files from the bash command line. The bash command to run it is "source ~/.bashrc".
- Under Linux the .bashrc file is located at ~/.bashrc. From linux terminal "gedit ~/.bashrc". At KU, the Linux home directory is mounted on H: and so you also have the option to edit from windows. We recommend notepad++ for Windows. The file path from windows should be H:\.bashrc.
- Add the following lines to the .bashrc file:
# For University of Kansas CReSIS use these lines: module rm gcc module rm python module rm matlab module load matlab/2015b # For Indiana University Karst/Carbonate use these lines: module rm core module rm matlab module rm curl module load core module load matlab/2015a module load curl # For University of Kansas/Indiana University Field Processing setup use these lines: # NONE REQUIRED # For Alfred Wegener Institute Ollie use these lines: module rm matlab module load matlab/2015b module load git # Recommended lines for everywhere: umask 002 # Ensures that files generated will have the correct user/group permissions so that all users in our group can work with the files. alias cp="cp -i" alias rm="rm -i" alias mv="mv -i" alias ssh="ssh -Y"
- KU and IU use modules. Modules are used to set environmental variables required by the software that we use. For more information: Modules
- To view all available packages from the command line, run "module avail"
- To view all packages you currently have loaded from the command line, run "module list"
- Change your home directory (Windows 7 Only at Kansas University): Open "Control Panel". Search for "User Accounts". Click on "User Accounts". In the left panel, choose "Change my environment variables". In the window that pops up, add or edit your HOME user variable to be your local home directory (e.g. C:\Users\USERNAME). See the example thumbnail on the right. Note that adding a backslash at the end of the path seems to cause git to fail; in other words "C:\Users\USERNAME" works, but "C:\Users\USERNAME\" does not.
- Create and/or modify .gitconfig (All Machines). The path for Linux/Mac is "~/.gitconfig". Under Windows it is usually "C:\Users\USERNAME\.gitconfig". We recommend "notepad++" to edit if using Windows (some programs like regular Notepad will not allow you to save with the long extension ".gitconfig"). The file may already exist. We recommend adding the following contents in the git configuration file. Comment/uncomment the lines that do not apply to you and fill in the correct name and email address.
[user] name = YOUR_NAME email = WORK_EMAIL_ADDRESS [core] autocrlf = false eol = lf [alias] lgb = log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset%n' --abbrev-commit --date=relative --branches [color] ui = auto [receive] denyCurrentBranch = updateInstead [push] default = simple # USE THIS LINE FOR NEW VERSIONS OF GIT >1.7.11 (try this first, if git fails then switch to tracking) # default = tracking # USE THIS LINE FOR OLD VERSIONS OF GIT <1.7.11 (e.g. on KU and IU servers) # Only include these lines if using Windows [credential] helper = wincred # Only consider including these lines if using Linux and https fails # If certificates are not in the usual system location (may be necessary for enterprise setups: KU, IU, AWI, etc.) #[http] # sslcainfo = /etc/pki/tls/certs/ca-bundle.crt # For KU # sslcainfo = /etc/ssl/certs/ca-bundle.crt # For AWI and IU # Only include for Linux and if you like the gedit or VIM editor better than the default editor #[core] # editor = gedit # editor = vim # NOTE: Only enable/include one diff/merge tool. Here are typical choices: # ======================================================================== # 1. Only include these lines if using TortoiseGit for diff/merge (nothing to include) # 2. Only include these lines if using meld for diff/merge # If using Git for Windows, but using meld instead of TortoiseGit: # 1. Install Meld-3.18.3-win32.msi # 2. Create C:\Windows\system32\meld text file with two lines: # #!/bin/bash # /c/Program\ Files\ \(x86\)/Meld/Meld.exe $@ #[diff] # tool = meld #[difftool] # prompt = false #[merge] # tool = meld #[mergetool] # prompt = false # 3. Only include these lines if using gvimdiff for diff/merge #[diff] # tool = gvimdiff #[difftool] # prompt = false # 4. Only include these line if using vimdiff for diff/merge #[diff] # tool = vimdiff #[difftool] # prompt = false # ========================================================================
- Clone cresis-toolbox (All Machines): This copies the software repository from https://github.com/CReSIS/ to your computer and sets up a working directory where you can edit and use the software. The default branch is "stable". Developers should branch from "master" and do pull requests to master.
- Open up a terminal (You may need to log in using ssh (e.g. PUTTY or Nxclient or Thinlinc) if you are doing this on a remote machine.)
- mkdir ~/scripts/
- cd ~/scripts/
- For read-only access:
- For read/write access:
- Setup ssh key: https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh
- git clone firstname.lastname@example.org:CReSIS/cresis-toolbox.git
- In your Documents (sometimes called "My Documents") folder, create a directory called "scripts". This is usually located at: C:\Users\USERNAME\Documents\scripts\cresis-toolbox
- Navigate to this "scripts" folder
- git clone the repository into this directory. This can be done several ways and depends on which git client interface you use.
TortoiseGit: Right click in the folder and choose "Git clone..." from the context menu. See Tortoise Git Clone example to the right. Replace username with your actual username and replace URL with https://GITHUB_USERNAME@github.com/CReSIS/cresis-toolbox
Instead of password, use personal access token: https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token
git for windows: Right click in the folder and choose "Git bash here...". Then run "git clone https://GITHUB_USERNAME@github.com/CReSIS/cresis-toolbox". Replace username with your github username.
- Create startup.m file (All Machines): Inside the cresis-toolbox repository, copy (do not move) the file HOME/scripts/cresis-toolbox/example_startup.m to the directory returned when you run "userpath" in Matlab. Rename the file in the userpath directory to "startup.m". The startup.m file adds the cresis-toolbox to the Matlab path so you can run all the cresis-toolbox functions and it also sets the gRadar variable (which holds your default cresis toolbox settings).
- Set the current profile in startup.m(All Machines): Inside the "startup.m" file, set the current profile to the profile that best matches your setup. Instructions on how to do this are at the top of the startup.m file. The recommended profile for Windows and Linux/Mac are:
- See profile list at the top of startup.m to choose the correct profile. Some institutions already have profiles defined, but may still require path changes in the profile.
- Windows and can map the KU CReSIS network drives: Set cur_profile to 5. No path changes.
- Linux and can mount KU CReSIS network drives: Set cur_profile to 3. No path changes. FYI: The userpath is usually "~/Documents/MATLAB/". At KU, the windows mount location for this is "H:\Documents\MATLAB\".
- Windows: Set cur_profile to 1 and adjust paths as necessary.
- Linux/Mac: Set cur_profile to 2 and adjust paths as necessary.
Modify the corresponding profile settings in the startup.m if necessary. Usually this is just changing the file paths to ensure they point to the folders where you have downloaded the git repositories, the data, and the metadata.
- Create "matlab" folder (All Machines):
- Under Windows: My Documents\scripts\matlab
- Under Linux/Mac: ~/scripts/matlab
This is your personal scratch space. You should keep temporary files which you do not plan to push to the Git repository in this space.
- Clone the ct_params and ct_docs repositories (All Machines): This is done the same way as cresis-toolbox, except the URLs are https://GITHUB_USERNAME@github.com/CReSIS/ct_params.git and https://GITHUB_USERNAME@github.com/CReSIS/ct_docs.git for https and
git clone email@example.com:CReSIS/ct_params.gitand
git clone firstname.lastname@example.org:CReSIS/ct_docs.gitfor ssh. The default branch is "stable". Developers should branch from "master" and do pull requests to master.
- Under Windows, in Windows Explorer navigate to My Documents\scripts\. Clone in this directory so that you should have ct_params and ct_docs in your scripts directory.
- Under Linux/Mac, change your working directory (cd) to ~/scripts/. Clone in this directory so that you should have ct_params and ct_docs in your scripts directory.
The read-only repository links are https://github.com/CReSIS/ct_params.git and https://github.com/CReSIS/ct_docs.git. Then you would typically do something like "cd ~/scripts; git clone https://github.com/CReSIS/ct_params.git" to clone the ct_params repository.
- Recommended Matlab Preferences (Optional):
- Set Matlab->Keyboard->Shortcuts: Set "Active settings" to "Windows Default Set" or "Macintosh Default Set"
- Set Matlab->Editor/Debugger: Uncheck "Save changes upon clicking away from a file"
- Set Matlab->Editor/Debugger: Uncheck "Reload unedited files that have been externally modified"
- Set Matlab->Editor/Debugger->Tab: Set "Tab size" and "Indent size" to 2
- Set Matlab->Editor/Debugger->Backup Files: Set "Location" to "Single directory" and choose a separate path such as "/tmp" in Linux/Mac or "C:\tmp" in windows.
- Set Matlab->General->Java Heap Memory: Set "Java Heap Size" to 2048 MB. This may not be required except for some OPS database operations like opsBulkInsert.
- Download the GIS data required by the toolbox (All Machines): Email email@example.com for URL. This is only required if the GIS data are not already available at your site. The following command may be useful to download (run from the folder where you have or want the "GIS_data" folder to be):
wget -m -nH --cut-dirs=3 ftp://data.cresis.ku.edu/data/temp/ct/GET_FOLDERNAME_FROM_CRESIS_DATA
- Obtain Google Map Platform API Key: To use Google Maps, you will need to specify a valid API key in startup.m by setting the ops.google_map_api_key field in your profile. Follow the instructions in header comment of google_map.m to obtain a key at https://developers.google.com/maps/.
- OPS user account (OPS Write Access Only): Request an Open Polar Server account if you are using imb.picker to make edits to layers in the OPS database. This is NOT required for browsing data. At KU, email firstname.lastname@example.org for an account.
CReSIS Toolbox Development Guides
Code for Multiple Segments
Commenting style example
CReSIS toolbox command reference
Debug output conventions
Matlab version specific code
Parameter Spreadsheet Guide
Variable naming conventions and Mathematical symbols
Mex Compiler Setup
General C/C++ Coding Notes
Linear Algebra Libraries
mex -v -largeArrayDims -lmwlapack -lmwblas wb_compute_cost.cpp
-v = verbose -largeArrayDims required for LAPACK and BLAS libraries
Linear Algebra PACKage (LAPACK) Information (used for matrix inverse, zgetrf and zgetri): http://www.netlib.org/lapack/explore-html/d7/d99/_v_a_r_i_a_n_t_s_2lu_2_c_r_2zgetrf_8f.html edit([matlabroot '/extern/include/lapack.h'])
Cross Platform Coding
For cross platform and 32/64 bit compatibility, use size_t and ptrdiff_t for referencing pointers.
Custom C compiler switches
Some switches must be passed through the options file rather than from the mex command line. The options file location can be found by adding -v option to the mex function call.
Windows minGW C/C++ Compiler
- Download and install 64 bit msys2:
http://www.msys2.org/ E.g. http://repo.msys2.org/distrib/x86_64/msys2-x86_64-20161025.exe
- From msys2 terminal run:
- After quitting msys2 when it asks you to by clicking the “x” to kill the shell, open msys2 again and run this:
- Then install g++ and support tools by running:
pacman -S --needed base-devel mingw-w64-i686-toolchain mingw-w64-x86_64-toolchain \ git subversion mercurial \ mingw-w64-i686-cmake mingw-w64-x86_64-cmake # https://github.com/orlp/dev-on-windows/wiki/Installing-GCC--&-MSYS2
- Then add g++ and the libraries to your Windows PATH environment variable. You can set the environment variables by pressing the start menu key, typing "environment", selecting "Edit the system environment variables" when it shows up in the suggestions list. This will open "System Properties". Open the "Environment variables..." button. Update the PATH variable in "System variables" to add the two new paths. Typically your two new paths will look like this:
Examples of VERSION are "6.2.0", "11.2.0", etc. and should match the version that you installed (just browse to your installation to see which is usually C:\msys64\mingw64\lib\gcc\x86_64-w64-mingw32\). Some versions of Windows will give you a nice editor for entering the two new paths. Some versions of Windows just show a long string and you just need to append the new paths to the PATH string with semicolons to delimit each new path; for example by adding this string to the end of the existing string ";C:\msys64\mingw64\bin;C:\msys64\mingw64\lib\gcc\x86_64-w64-mingw32\VERSION". </pre>
To gain access to all of the commands (e.g. "wget" which is used by the cresis-toolbox), then also add this folder to your path:
- Setup Matlab to use mingw by adding this Windows environment variable:
- Test in Matlab and set it up (restart matlab first and then run these two commands):
mex -setup mex -setup C++
Linux C++ Compiler Issues
gcc version mismatch
CReSIS specific: If you get "libstdc++.so.6: version CXXABI_1.3.8' not found" you may need to unload the gcc module so that you are using the OS installed gcc which is an older version that works with Matlab r2015b. The problem is that mex will link with the new version, but when the mex file is run it will only see the matlab environment. There may be a way to get the matlab environment to see the new libraries, but that solution is not documented yet.
C++11 Option Error
To compile C++ programs, run "mex -setup C++". This creates the options file that you may need to fix this file. If you get a C++11 option error, you may be using pre-G++ 4.7. You can check the g++ version with system('g++ --version'); To fix this, add -v option to mex function and look for a line like this:
Options file: ~/.matlab/R2015b/mex_C++_glnxa64.xml
In the options file, replace -std=c++11 with -std=c++0x (should occur in two places)
THIS PROJECT IS NOT ACTIVE.
The code review is an ongoing project to propose clean up of the code and supporting files. One task of the code review is to review all the functions in the toolbox and make sure the comments are up to date. If you help with this project, follow the link below: