Multiverse recipes

There are many ways of specifying and customizing complex multiverse analyses. The following code snippets aim to provide some examples/recipes for this.

Specifying classes/functions

You can define the method either as a direct function call or by passing the arguments separately in an args dictionary.
Both approaches are equivalent:
[8]:

from comet.multiverse import Multiverse

forking_paths = {
    "dfc": [
    {
        "name": "SW29",
        "func": "comet.connectivity.SlidingWindow(ts, windowsize=29).estimate()"
    },
    {
        "name": "SW49",
        "func": "comet.connectivity.SlidingWindow",
        "args": {
            "time_series": "ts",
            "windowsize": 49}
    }
]}

def analysis_template():
    import comet

    ts = comet.utils.load_example("time_series")
    dfc = {{dfc}}

mverse = Multiverse(name="example_mv_recipes")
mverse.create(analysis_template, forking_paths)
mverse.summary()
Universe Decision 1 Value 1
0 Universe_1 dfc SW29
1 Universe_2 dfc SW49
From a functional perspective one could even pass the entire function call as a literal (as indicated by the $ sign).
However, as this does not allow users to specify a custom name, it will look messy in the outputs.
[2]:

from comet.multiverse import Multiverse

forking_paths = {
    "dfc": ["$comet.connectivity.SlidingWindow(ts, windowsize=69).estimate()"]
}

def analysis_template():
    import comet

    ts = comet.utils.load_example("time_series")
    dfc = {{dfc}}

mverse = Multiverse(name="example_mv_recipes")
mverse.create(analysis_template, forking_paths)
mverse.summary()
Universe Decision 1 Value 1
0 Universe_1 dfc $comet.connectivity.SlidingWindow(ts, windowsi...

This works the same for graph related functions, and also any other function/class not included in Comet:

[3]:

from comet.multiverse import Multiverse

forking_paths = {
    "functions": [
    {
        "name": "comet clustering",
        "func": "comet.graph.clustering_coef(G)",
    },
    {
        "name": "bct clustering",
        "func": "bct.clustering_coef_wu(G)",
    },
    {
        "name": "scipy zscore",
        "func": "scipy.stats.zscore(G)",
    }
]}

def analysis_template():
    import comet
    import bct
    import scipy

    G = [[0, 1, 2], [1, 0, 3], [2, 3, 0]]
    result = {{functions}}

mverse = Multiverse(name="example_mv_recipes")
mverse.create(analysis_template, forking_paths)
mverse.summary()
Universe Decision 1 Value 1
0 Universe_1 functions comet clustering
1 Universe_2 functions bct clustering
2 Universe_3 functions scipy zscore

Varying parameters for same connectivity measures

One can either define the the complete measures in the forking paths, or only substitute the arguments in the analysis template:

[4]:

from comet.multiverse import Multiverse

forking_paths = {
    "sw": [
    {
        "name": "SW29",
        "func": "comet.connectivity.SlidingWindow(ts, windowsize=29).estimate()"
    },
    {
        "name": "SW39",
        "func": "comet.connectivity.SlidingWindow(ts, windowsize=39).estimate()"
    },
    {
        "name": "SW49",
        "func": "comet.connectivity.SlidingWindow(ts, windowsize=49).estimate()"
    },
]}

def analysis_template():
    import comet

    ts = comet.utils.load_example("time_series")
    dfc = {{sw}}

mverse = Multiverse(name="example_mv_recipes")
mverse.create(analysis_template, forking_paths)
mverse.summary()
Universe Decision 1 Value 1
0 Universe_1 sw SW29
1 Universe_2 sw SW39
2 Universe_3 sw SW49 
[5]:

from comet.multiverse import Multiverse

forking_paths = {
    "size": [29, 39, 49],
    "shape": ["rectangular", "gaussian"]
    }

def analysis_template():
    import comet

    ts = comet.utils.load_example("time_series")
    dfc = comet.connectivity.SlidingWindow(time_series=ts, windowsize={{size}}, shape={{shape}}).estimate()

mverse = Multiverse(name="example_mv_recipes")
mverse.create(analysis_template, forking_paths)
mverse.summary()
Universe Decision 1 Value 1 Decision 2 Value 2
0 Universe_1 size 29 shape rectangular
1 Universe_2 size 29 shape gaussian
2 Universe_3 size 39 shape rectangular
3 Universe_4 size 39 shape gaussian
4 Universe_5 size 49 shape rectangular
5 Universe_6 size 49 shape gaussian

A more complex design are different connectivity measures which have different parameters. Two potential solutions are:

  1. Give the full decision space in the forking paths:

[6]:

from comet.multiverse import Multiverse

forking_paths = {
    "sw": [
    {
        "name": "SW29",
        "func": "comet.connectivity.SlidingWindow(ts, windowsize=29).estimate()"
    },
    {
        "name": "SW39",
        "func": "comet.connectivity.SlidingWindow(ts, windowsize=39).estimate()"
    },
    {
        "name": "FLS50",
        "func": "comet.connectivity.FlexibleLeastSquares(ts, mu=50).estimate()"
    },
    {
        "name": "FLS100",
        "func": "comet.connectivity.FlexibleLeastSquares(ts, mu=100).estimate()"
    }
]}

def analysis_template():
    import comet

    ts = comet.utils.load_example("time_series")
    dfc = {{sw}}

mverse = Multiverse(name="example_mv_recipes")
mverse.create(analysis_template, forking_paths)
mverse.summary()
Universe Decision 1 Value 1
0 Universe_1 sw SW29
1 Universe_2 sw SW39
2 Universe_3 sw FLS50
3 Universe_4 sw FLS100

  1. Use a conditional logic in the analysis script combined with a config rule. Here, combinations of SlidingWindow and mu as well as FlexibleLeastSquares and windowsize are invalid, so we exclude them from the multiverse:

[7]:

from comet.multiverse import Multiverse

forking_paths = {
    "method": ["SW", "FLS"],
    "windowsize": [29, 39],
    "mu": [50, 100]
    }

config = {
    "exclude": [
        [{"method": "SW"}, "mu"],
        [{"method": "FLS"}, "windowsize"]
    ]
}

def analysis_template():
    import comet

    ts = comet.utils.load_example("time_series")
    if {{method}} == "SW":
        dfc = comet.connectivity.SlidingWindow(ts, windowsize={{windowsize}}).estimate()
    elif {{method}} == "FLS":
        dfc = comet.connectivity.FlexibleLeastSquares(ts, mu={{mu}}).estimate()

mverse = Multiverse(name="example_mv_recipes")
mverse.create(analysis_template, forking_paths, config)
mverse.summary()

Exclusion summary
-----------------
Total number of universes: 8
  - Removed 8 duplicate universes (set 'deduplicate' to False if you want to keep them)
  - Set 'mu' to NaN for universes matching {'method': 'SW'} (4 total).
  - Set 'windowsize' to NaN for universes matching {'method': 'FLS'} (4 total).

8 universes remain for analysis.
Universe Decision 1 Value 1 Decision 2 Value 2 Decision 3 Value 3
0 Universe_1 method SW windowsize 29.0 mu NaN
1 Universe_2 method SW windowsize 29.0 mu NaN
2 Universe_3 method SW windowsize 39.0 mu NaN
3 Universe_4 method SW windowsize 39.0 mu NaN
4 Universe_5 method FLS windowsize NaN mu 50.0
5 Universe_6 method FLS windowsize NaN mu 100.0
6 Universe_7 method FLS windowsize NaN mu 50.0
7 Universe_8 method FLS windowsize NaN mu 100.0