Multiverse analysis on a high performance cluster

Multiverse analyses can be computationally expensive and thus may require a high performance cluster.

A simple example for running a multiverse within a SLURM environment is provided in this tutorial.

First, create a job script (e.g., multiverse.job) and fill it according to your requirements:

#!/bin/bash
#SBATCH --job-name=multiverse
#SBATCH --partition=<YOUR PARTITION>
#SBATCH --nodes=1
#SBATCH --cpus-per-task=128
#SBATCH --mem=512GB
#SBATCH --time=01:00:00
#SBATCH --output=%j.out
#SBATCH --error=%j.out

set -euo pipefail

# Use whatever is needed to load your Python environment (depending on your HPC setup)
module load Miniforge3
conda activate comet

# Start the multiverse from a Python script
python multiverse.py

Second, create your multiverse in a Python file (e.g., multiverse.py).

In the script, specify parallelisation with mverse.run(parallel=X). Comet will constrain every universe to a single CPU core, so the parallelisation should ideally be at most one less than the number of cores requested in the job script.

For example, as this multiverse creates 108 universes, we can run all of them in parallel:

from comet.multiverse import Multiverse

forking_paths = {
    "strings":  ["Hello", "world", "this", "is", "a", "multiverse"],
    "numbers":  [1, 2, 3, 4, 5, 6, 7, 8, 9],
    "booleans": [True, False]
}

def analysis_template():
    pass

mverse = Multiverse(name="example_mv_hpc")
mverse.create(analysis_template, forking_paths)
mverse.summary()
mverse.run(parallel=108)

Finally, the job can simply be submitted by running sbatch multiverse.job.

The multiverse and its results are stored in the directory from which the job was submitted.