Available Software¶

CCR maintains a suite of software programs, libraries, and toolchains commonly used in scientific computing. The HPC clusters can execute most software that runs under Linux. In many cases, the software you need will already be installed and available to you on the compute nodes. You access the software using what's called a "module". If the software you need is not available, you can ask our staff to install it for you or do it yourself.

New software infrastructure now available

CCR is moving to a new software infrastructure described in this document. Starting in the Summer of 2023 users will be required to migrate to the system. To start using the new software now, simply run this command touch ~/.ccr_new_modules. You will need to logout and log back in for the changes to take effect. If you'd like to temporarily go back to using the old modules in /util, simply remove that file rm ~/.ccr_new_modules.

Using Modules¶

Modules are simply configuration files that dynamically change your environment allowing you to run a particular application or provide access to a particular library. The module system allows CCR to provide multiple versions of software concurrently and enables users to easily switch between different versions without conflicts. Users can also build and maintain their own modules for more fine grained control over your groups environment.

Modules are managed using a tool called Lmod developed by TACC. For a more comprehensive overview of Lmod you can read the user guide here.

To see what software modules are available, ssh into the login node and run:

$ module avail

This will return a list of modules available to load into your environment.

Note

If a module has dependencies you may not see the module listed until dependencies are loaded. For example, software compiled with the gcc toolchain will not be shown until you load gcc or foss modules. See the hierarchical modules section for more information.

If you cannot load a module because of dependencies, you can use the module spider command to find what dependencies you need to load the module:

module spider some_module

# For example, openmpi requires gcc:
$ module spider openmpi
    You will need to load all module(s) on any one of the lines below before 
    the "openmpi/4.1.1" module is available to load.

      gcc/11.2.0

To load your chosen modules into the environment run:

$ module load some_module_name

# For example:
$ module load python

You can specify the version of the software by appending a / with the version number:

module load some_module/version 

# For example:
$ module load python/3.9.5

Hierarchical Modules¶

CCR uses a hierarchical module naming convention to support programs built with specific compiler, library, and CPU architecture requirements. For example, when you run module avail on an Intel Skylake compute node (avx512), you will see three hierarchical levels of modules that look like this:

---------------------------- MPI-dependent avx512 modules -----------------------------

-------------------------- Compiler-dependent avx512 modules --------------------------

------------------------------------ Core modules -------------------------------------

Core modules: compiler and architecture independent
Compiler-dependent modules: depend on a specific CPU architecture and compiler toolchain
MPI-dependent modules: depend on a specific CPU architecture, compiler toolchain, and MPI library

Only core modules are shown by default

When you run module avail only the core modules are shown by default. To see modules for a specific compiler toolchain you will need to first load the module for the specific compiler toolchain of interest.

CCR supports the following compiler toolchains:

Toolchain	Included compilers and libraries
intel	Intel compilers, Intel MPI, and Intel MKL
foss	GCC, OpenMPI, FlexiBLAS, OpenBLAS, LAPACK, ScaLAPACK, FFTW
GCC	GCC compiler only

In addition to the compiler toolchains, the hierarchical module system is also "CPU architecture" aware. The module system will auto-detect what type of CPU you're running on and display modules built for that specific architecture.

CCR supports the following CPU architectures:

Architecture	Supported CPUs
avx	Intel Sandy Bridge, Ivy Bridge
avx2	Intel Haswell, Broadwell
avx512	Intel Skylake-SP, Skylake-X, Cascade Lake-SP

For specific compiler versions, see our releases page.

Loading Modules in a Job Script¶

Modules in a job script can be loaded after your #SBATCH directives and before your actual executable is called. A sample job script that loads Python into the environment is shown below:

#!bin/bash
#SBATCH --nodes=1
#SBATCH --time=00:01:00
#SBATCH --ntasks=1
#SBATCH --job-name=test-job

# Load your modules here
module load python

python3 test-program.py

For more information, see the Running Jobs section.

Python¶

Several python modules are available for use: python, anaconda3, and scipy-bundle. We encourage users to checkout these modules as they include lots of common scientific python packages.

module	Included python packages
python	Bare python3 interpreter only
scipy-bundle	beniget, Bottleneck, deap, gast, mpi4py, mpmath, numexpr, numpy, pandas, ply, pythran, scipy
anaconda3	see here

We also recommend using virtual environments.

R¶

Two R modules are provided: r and r-bundle-bioconductor, both of which include many pre-built R libraries. To see a complete list of R libraries and packages included with each module run the spider command:

$ module spider r
$ module spider r-bundle-bioconductor

Perl¶

We provide a perl module which includes many pre-built CPAN modules. To see the complete list run:

$ module spider perl