(Please note that your HPC may have different locations for filesystems. Please consult the articles in your organization for your own specifics.)
Software installation has 3 main levels.
Software Stacks Available to all users as Modules
Software Stacks Available per project
Software Stacks customized per user
On an HPC system software is typically installed as modules.
Will give you a list of the possible software modules.
# Load the module module load my-module # Unload only one module module unload my-module # Unload all modules module purge
Install new software
The recommended way to install software on the cluster is through Conda. The easiest way to search for conda packages is through google, but you can also use
module load Miniconda3
You can then install software using conda. It is always a good idea to keep your software environments as small and isolated as possible to keep softwares from clobbering eachother.
Install sofware directly with conda
conda install -p /apps/users/$USER/software-name package-1 package-2 # or by name conda create -n some-name package-1 package-2
Once it’s installed conda will tell you to use
conda activate /path/to/env or name-of-env. Don’t do that. Use
source activate name-of-env or
source activate path-to-env.
Recommended - Create an environment definition
You can also create a conda-env definition file. This is the recommended way of doing things because you have a record of the software you’ve installed.
# r-single-cell.yaml name: r-single-cell channels: - bioconda - conda-forge dependencies: - r-seurat=3.0.2 - r-monocle3=0.2.0
conda env create -f=r-single-cell.yaml # You can also specify a directory # conda env create -f=r-single-cell.yaml -p /apps/users/$USER/software-stack source activate r-single-cell-env # or # source activate /apps/users/$USER/software-stack
After you’ve installed software it’s a good idea to clean out the package cache.
conda clean -y --all
The first thing to try is to clean out your package cache.
conda clean -y --all
If you are getting errors about packages or package manifests being corrupted, and you are using any modules you do not have write access to (Miniconda3) you will have to ask an admin to clean out the package cache there.
If you are trying to create an environment and it’s being killed, it is probably because the dependency tree of the packages you are trying to solve is too large for that node. Submit a SLURM job for a higher memory node, either through your Jupyterhub or through the command line, and install there.