Topics

Shared Conda Environment

For shared projects, we recommend creating a conda environment in a shared project space (multiple users can share this environment and there is less chance of running out of space) instead of your home directory which is not shareable. By default, conda creates a new environment folder and installs all the packages inside .conda folder in your home directory but this space is limited and cannot be shared with other users. In this topic guide, we will create a new conda environment in a shared project space for multiple researchers to use it.

First, load anaconda module and create a new environment with a specific python version (default is python 3.9 with anaconda3 module) but we can upgrade to 3.10 for example.

For this example, I will create a shared conda env in /zfs/gsb/intermediate-yens/conda but choose a path in your shared project space instead.

To make a new conda directory for the environment in a shared project space (this is where all the python packages will be installed to):

$ cd /zfs/gsb/intermediate-yens
$ mkdir conda
$ cd conda

You now should be in /zfs/gsb/intermediate-yens/conda directory. In the --prefix argument, specify the path to the conda folder and designate what your environment is about. For example, if this is my OCR project, I would add ocr at the end of the prefix to have a separate directory for that environment. Then, I can create multiple conda environments in /zfs/gsb/intermediate-yens/conda for different projects I am working on.

$ module load anaconda3
$ conda create --prefix /zfs/gsb/intermediate-yens/conda/ocr python=3.10

Answer y to Proceed when prompted. Creating a new conda environment takes a few minutes.

This will make a new conda environment in /zfs/gsb/intermediate-yens/conda/ocr.

Then, activate the environment. This adds the python path to our current environment so the python we are using is the new conda environment’s python:

$ source activate /zfs/gsb/intermediate-yens/conda/ocr

Check that the python binary is inside the newly created conda environment:

$ which python
 /zfs/gsb/intermediate-yens/conda/ocr/bin/python

Check that the version of python is as expected:

$ python --version
Python 3.10.6

List available conda environments (you should see base and any other environments you have):

$ conda info -e

You should see at least two conda environments:

# conda environments:
#
                      *  /zfs/gsb/intermediate-yens/conda/ocr
base                     /software/free/anaconda/2022.05

The default conda environment is base and the active conda env is indicated with *.

Install python packages for your project.

$ pip install -U pytesseract matplotlib pandas tqdm

After loading anaconda3 module and activating the environment, we install ipykernel package so that the active conda environment can be added as a kernel in your JupyterHub.

$ conda install ipykernel

Again, press y to Proceed when prompted.

Now to run our python script, all we have to do is load anaconda module and activate the conda environment before running python binary.

Note: If you want to create a personal conda environment and are okay using your home directory for it, you can use a shortcut:

$ conda create -n ocr python=3.10
$ source activate ocr

when creating a new conda environment.