This document is a work in progress and does not address all the possible issues you might run into compiling and installing software.

Easybuild¶

CCR staff utilize the Easybuild framework when installing software and this tool is made available to CCR users to install software themselves. Easybuild ensures that the compilation and linking that happens during a software installation is done correctly and will work with our systems. CCR staff do not offer courses on using Easybuild, nor are we able to troubleshoot issues you might run into. However, there is excellent documentation and tutorials provided by Easybuild developers. We provide this document to provide a bit of context to the Easybuild documentation and provide an example of using Easybuild to install a software package.

Easybuild and Modules¶

One of the key features of EasyBuild is that it automatically generates environment modules which can be used to make a software package available in your session. In addition to defining standard Linux environment variables such as PATH, CPATH and LIBRARY_PATH, EasyBuild also defines some environment variables specific to EasyBuild, two of which may be particularly interesting to users:

EBROOT<name>: Contains the full path to the location where the software <name> is installed.
EBVERSION<name>: Contains the full version of the software <name> loaded by this module.

For example, the module java/11.0.16 defines:
- EBROOTJAVA: /cvmfs/soft.ccr.buffalo.edu/versions/2023.01/easybuild/software/Core/java/11.0.16
- EBVERSIONJAVA: 11.0.16

You can see the environment variables defined by the java/11.0.16 module using:

CCRusername@login:~$ module show java/11.0.16 |grep EB  

You can use these as variables on the command line and in scripts. For example:

CCRusername@login:~$ cd $EBROOTJAVA
CCRusername@login:~$ /cvmfs/soft.ccr.buffalo.edu/versions/2023.01/easybuild/software/Core/java/11.0.16$  

Using Easybuild in your own account¶

EasyBuild can be used to install software packages in your own account. However, in most cases, it is preferable to ask our technical support to install the software centrally for you. This will ensure that the software package is available on all of our clusters for all users. It will also avoid using up all your quota. It is often difficult for us to troubleshoot problems when someone else is trying to install software, making it hard to provide support to you.

What is a recipe¶

Writing a recipe from scratch will not be discussed here; you can find more about this in the EasyBuild documentation. Modifying a recipe for your particular situation is easier, and it is easier still to find a suitable recipe and use it unmodified.

Recipes, also known as EasyConfig files, are text files containing the information EasyBuild needs to build a particular piece of software in a particular environment. They are named following this convention:

<name>-<version>-<toolchain name>-<toolchain version>.eb where <name> is the name of the package, <version> is its version, <toolchain name> is the name of the toolchain used and <toolchain version> is its version. More on toolchains later.

Installation recipes and logs¶

EasyBuild keeps a copy of the recipe used to install each software package, as well as a detailed log inside the installation directory. This is accessible in the directory $EBROOT/easybuild. For example, for the java/11.0.16 module, the installation directory contains, among other things:
- $EBROOTJAVA/easybuild/Java-11.0.16.eb
- $EBROOTJAVA/easybuild/easybuild-Java-11.0.16-20221206.215954.log.bz2

During your installation, you will see output indicating where the temporary log files for the build are located. After completion of the install, the path of the full build log is output by Easybuild. For more information see the Investigating Logs section below

An Easybuild Example¶

Easybuild Background¶

At CCR, we provide a set of supported toolchains and users are able to use Easybuild to install software for themselves without administrative privileges. We provide instructions to get started in CCR's Software Building documentation. Some packages install quite easily and the samtools example provided in our documentation is one of those packages. However, what happens if you want to install something that CCR doesn't offer, but you're not finding an exact match for a toolchain provided by CCR or the version of the software you want to run? It's a bit of trial and error, to be honest. And this is what CCR staff are doing on a daily basis as we work through the installation requests in our GitHub repository.

In this how-to section, we're going to take you through the process to provide you with the information you'll need to be off and running on your own Easybuild installation!

Though you don't need to understand everything about an Easybuild recipe file to install software using Easybuild, we do recommend you read the EasyBuild guide to writing easyconfig files and keep it handy to refer to should questions come up while you're editing these recipe files.

Easyconfig files are named to include the toolchain and version:
GNU compilers:
[...]-foss-2021b.eb
[...]-GCCcore-11.2.0.eb

Intel compilers:
[...]-intel-2022.00.eb
[...]-iimpi-2022.00.eb

The EasyBuild config repository contains a lot of recipes which may or may not compile with the toolchains CCR provides. As a reminder, these are the CCR supported toolchains. You must use a recipe for a supported toolchain when installing software using Easybuild. If you try a recipe for an unsupported toolchain, Easybuild will attempt to install that toolchain for you. You don't want this as it will take up too much disk space, take very long to build, and is highly likely that installation will fail.

What's in a toolchain¶

Toolchains are a combination of compiler, MPI implementation, CUDA version, and mathematical libraries, which are used to compile the software package. CCR offers toolchains as modules and you can see what is included in a toolchain by looking at the module. For example, if you load the foss/2021b module you'll see it includes:

depends_on("gcc/11.2.0")
depends_on("openmpi/4.1.1")
depends_on("flexiblas/3.0.4")
depends_on("fftw/3.3.10")
depends_on("scalapack/2.1.0-fb")

When using Easybuild, do NOT use the CCR login nodes. Always use a compile node or do this from a compute node in an OnDemand desktop session or interactive job. These installations can use a decent amount of disk space so we recommend you use your project directory for all software installations. We do not recommend using your home directory as these have small quotas. If you haven't already done so, create an easybuild directory in your group's project directory and set the Easybuild build prefix to that directory. Reminder: this is just an example; you'll need to set the path name to your group's shared project directory, which may not be in /projects/academic

CCRusername@login:~$ ssh compile  
CCRusername@compile ~ $ mkdir /projects/academic/[YourGroupName]/easybuild   
CCRusername@compile ~ $ export CCR_BUILD_PREFIX=/projects/academic/[YourGroupName]/easybuild  

Example Easybuild Installation¶

Refer to our CCR-examples repository for an example of rebuilding SAMtools-v1.18 with Easybuild.

Investigating Logs¶

Easybuild does a great job logging everything it does. You just have to spend time reviewing all these logs. At the start of the installation, Easybuild will give you the location and name of the log for the entire installation. This is accessible while the installation is running but is removed at completion time. The log file is zipped up and stored for future reference (see output shown in the example above):

== Temporary log file in case of crash /tmp/eb-_ago41dx/easybuild-8vf4s_8o.log

And with each step of the installation, you'll get a separate log and working directory, for example:

  >> running command:
        [started at: 2024-02-15 09:45:29]
        [working dir: /var/tmp/[CCRusername]/easybuild/build/.../...]
        [output logged in /tmp/eb-_ago41dx/easybuild-run_cmd-5y2xgnhf.log]

Frequently asked questions¶

Failed dep¶

Why am I still seeing failed to determine minimal toolchain for dep?
You've checked all the dependencies for your software and you know that they're already installed with the right versions by CCR (or you did this yourself). Yet, you're still seeing the error failed to determine minimal toolchain for dep when trying to install your software. Make sure that your EB recipe is formatted correctly. In the list of dependencies, you must match the name of the dependency with the EB recipe configs exactly. For example, if your recipe has a dependency for rust/1.54.0 you will see this is already installed by CCR. However, the name in the EB recipe must be Rust not rust. How can you tell how the name should be spelled? Look at the Easybuild recipe name for the module and use that spelling. In this example, the EB recipe for rust is: Rust-1.54.0-GCCcore-11.2.0.eb Also, ensure you're not using a module that is an extension to an existing module in your dependency lists. For more information see the Module extensions section below.

Another reason you might see this is because Easybuild can't find the dependency you've already built. If you've had to modify an Easybuild recipe to build a dependency, that recipe needs to be in a location where the Easybuild installation can find it. By default, Easybuild is looking in CCR's configuration directories where our recipes are stored. It will not look in your directory. To fix this, specify your directory in the Easybuild installation command using the --robot= option. For the example from ccr-examples repository, you can use the following command so Easybuild locates the previously built dependencies in your home directory: eb SAMtools-1.18-GCC-11.2.0.eb --robot=/user/[CCRusername]

Module extensions¶

Sometimes you'll see modules listed in the module spider output that have an (E) next to them like: matlab/1.0.2 (E) Do NOT use these versions in your Easybuild recipes. The (E) refers to the module being an extension of another module. If we used this in our EB recipe then the software we build will also depend on the "parent" module.

Install a different software version¶

How do I update the version of a software?
You have an EB recipe to try to install but you want to update the software version. This completely depends on the type of software you're installing. In our ccr-examples demo for SAMtools-v1.18, we can see in the EB recipe that this software is being downloaded from a Github repository.

source_urls = ['https://github.com/%(namelower)s/%(namelower)s/releases/download/%(version)s']
sources = [SOURCELOWER_TAR_BZ2]
checksums = ['d686ffa621023ba61822a2a50b70e85d0b18e79371de5adb07828519d3fc06e1']

You will see source locations and checksums are required in Python bundles as well. These can often be downloaded from PyPi

What if CCR installed software but I want a different version?
In most cases, we will only offer one version of a software package per software release. If you must use a different version, we recommend you check first to see if CCR has a modified EB recipe for that software. If so, it will most likely be easiest for you to copy that and modify it for your needs, rather than starting from the original Easybuild recipe.

Finding Easybuild Recipes¶

Is there anywhere else I can search for Easybuild recipes?
Yes! Although CCR syncs with the official Easybuild easyconfigs repository, our repository is not always the most up-to-date. You can use the Github interface to search easily and if you find a recipe you'd like to try, copy the contents and paste into a file, making sure to name the file correctly. We also highly recommend using the Digital Research Alliance of Canada (formerly Compute Canada) as a reference with many easyconfig recipes. If there is no EB recipe you will need to create one using other recipes as examples.

Easybuild Help¶

Where can I go for more help?
This guide was offered as an example and will not transfer over to each and every software package you try to install. Unfortunately, compiling and installing software is hard and each package is different from the next. Easybuild provides a reproducible structure and allows us to build software in a way that ensures it will run efficiently and with the highest performance on the CCR systems. However, it is not always easy. We recommend the Easybuild documentation and tutorials as the primary sources of information. It is often difficult for CCR staff to troubleshoot software installations run by others, even if you provide us log files. We have to go through the process of replicating your steps, at which point, it's often easier for us to just do the installation ourselves. If there is an existing EB recipe available and we don't already have a version of the package installed, we encourage you to submit a build request as per CCR's Software build requests documentation. If you run into problems while using Easybuild, it's possible we can provide some basic assistance through CCR Help. However, please note there are limits to the amount of individual help we can provide and you may need to work independently to resolve your issues.

Module Not Listed¶

Why isn't my module showing up?
Your Easybuild installation completes successfully but when you run module avail you do not see the software listed. This is due to CCR's Hierarchical Modules structure. module spider should be used to find installed modules across all toolchains. Unless your software does not depend on a specific toolchain, you must load that toolchain first, before you'll see the module listed using module avail Also, ensure you have the custom module paths setup as per CCR's "Building modules for your group" documentation.

Why isn't a CCR module showing up?
CCR has installed hundreds of software packages and not all of them are intended to be loaded and used frequently by users. They may be packages required just to install other packages or tools that are needed for using other software. To attempt to keep the module spider output cleaner, we have chosen to hide some modules. No fear! You can still use these modules just as any others. To find these, you can tell the module spider command to show any hidden modules that also exist using this command: module --show-hidden spider software_name

Setting permissions on installed software¶

Sometimes a software package requires certain unix file permissions in order to launch. This is usually when the software is a licensed product and the vendor wants to ensure only you can access your installation. If you see something in the documentation for the software you're installing or you attempt to install it with Easybuild and see an error similar to this:

# IMPORTANT!!  
# Needs to be installed with --umask=007  

Easybuild installation hangs¶

Sometimes an Easybuild installation will hang with no obvious errors or the installation logs are not updated with useful information. Easybuild is configured to run in parallel so there are times when the installation can step on itself or log files are not updated because another process has it locked. This is rare but if you come across this, one thing you can try is to force the Easybuild installation to run in serial. Add this option to your EB installation command: --parallel=1