Multiverse basics

To conduct a multiverse analysis, the forking paths must be specified in a dictionary. Options can contain:

• strings

• numerical values

• boolean values

• comet dFC methods

• comet and bct graph measures

• any kind of function

from comet.multiverse import Multiverse

forking_paths = {
    "strings": ["Hello", "world"],
    "numbers": [3.14, 4, 5.2],
    "booleans": [True, False],
    "dfc_measures": [
        {"name": "LeiDA", "func": "comet.connectivity.LeiDA(time_series=ts).estimate()"},
        {"name": "JC11", "func": "comet.connectivity.Jackknife(time_series=ts, windowsize=11).estimate()"}],
    "graph_measures": [
        {"name": "efficiency", "func": "comet.graph.efficiency(G, local= False)"},
        {"name": "clustering", "func": "comet.graph.avg_clustering_onella(G)"}],
}

With the decisions and options defined, an analysis template has to be specified. This is similar to a standard analysis pipeline with three additional requirements:

• The template is required to be encapsulated in a dedicated function

• Required imports need to be within the template function

• Decision points need to be specified in double brackets: {{decision}}

In this brief example, the corresponding string, number, and boolean decision will be printed in each universe. Then, connectivity will be estimated with the corresponding dFC method, and a graph measure (local efficiency or clustering) is calculated:

def analysis_template():
    import comet

    print("Decision 1:", {{strings}})
    print("Decision 2:", {{numbers}})
    print("Decision 3:", {{booleans}})

    # Load example data and calculate dFC
    ts = comet.utils.load_example()
    dfc = {{dfc_measures}}

    # Calculate graph measure
    graph_measure = []
    for i in range(dfc.shape[2]):
        G = dfc[:, :, i]
        G = comet.graph.handle_negative_weights(G, type="absolute")
        G = comet.graph.threshold(G, type="density", density=0.5)
        G = comet.graph.binarise(G)
        gm = {{graph_measures}}
        graph_measure.append(gm)

    # Save the results
    result = {"graph_measure": graph_measure}
    comet.utils.save_universe_results(result)

The forking paths dictionary defines 5 decision points consisting of 2 options each. Thus, the resulting multiverse will contain 2⁵=32 universes. A Multiverse object has to be created and can then be used to create, run, summarize, and visualize the multiverse.

• mverse.create() will generate Python scripts for all 32 universes. These scripts will be saved in a newly created example_multiverse/ folder

• mverse.summary() will print the decisions for every universe. This information is also available as a .csv in the example_multiverse/results/ folder

• mverse.run() will either run all or individual universes. If the computational resources allow for it, this can be parallelized by using e.g. mverse.run(parallel=4)

mverse = Multiverse(name="example_mv_basics")
mverse.create(analysis_template, forking_paths)
mverse.summary()

	Universe	Decision 1	Value 1	Decision 2	Value 2	Decision 3	Value 3	Decision 4	Value 4	Decision 5	Value 5
0	Universe_1	strings	Hello	numbers	3.14	booleans	True	dfc_measures	LeiDA	graph_measures	efficiency
1	Universe_2	strings	Hello	numbers	3.14	booleans	True	dfc_measures	LeiDA	graph_measures	clustering
2	Universe_3	strings	Hello	numbers	3.14	booleans	True	dfc_measures	JC11	graph_measures	efficiency
3	Universe_4	strings	Hello	numbers	3.14	booleans	True	dfc_measures	JC11	graph_measures	clustering
4	Universe_5	strings	Hello	numbers	3.14	booleans	False	dfc_measures	LeiDA	graph_measures	efficiency
5	Universe_6	strings	Hello	numbers	3.14	booleans	False	dfc_measures	LeiDA	graph_measures	clustering
6	Universe_7	strings	Hello	numbers	3.14	booleans	False	dfc_measures	JC11	graph_measures	efficiency
7	Universe_8	strings	Hello	numbers	3.14	booleans	False	dfc_measures	JC11	graph_measures	clustering
8	Universe_9	strings	Hello	numbers	4.00	booleans	True	dfc_measures	LeiDA	graph_measures	efficiency
9	Universe_10	strings	Hello	numbers	4.00	booleans	True	dfc_measures	LeiDA	graph_measures	clustering
10	Universe_11	strings	Hello	numbers	4.00	booleans	True	dfc_measures	JC11	graph_measures	efficiency
11	Universe_12	strings	Hello	numbers	4.00	booleans	True	dfc_measures	JC11	graph_measures	clustering
12	Universe_13	strings	Hello	numbers	4.00	booleans	False	dfc_measures	LeiDA	graph_measures	efficiency
13	Universe_14	strings	Hello	numbers	4.00	booleans	False	dfc_measures	LeiDA	graph_measures	clustering
14	Universe_15	strings	Hello	numbers	4.00	booleans	False	dfc_measures	JC11	graph_measures	efficiency
15	Universe_16	strings	Hello	numbers	4.00	booleans	False	dfc_measures	JC11	graph_measures	clustering
16	Universe_17	strings	Hello	numbers	5.20	booleans	True	dfc_measures	LeiDA	graph_measures	efficiency
17	Universe_18	strings	Hello	numbers	5.20	booleans	True	dfc_measures	LeiDA	graph_measures	clustering
18	Universe_19	strings	Hello	numbers	5.20	booleans	True	dfc_measures	JC11	graph_measures	efficiency
19	Universe_20	strings	Hello	numbers	5.20	booleans	True	dfc_measures	JC11	graph_measures	clustering
20	Universe_21	strings	Hello	numbers	5.20	booleans	False	dfc_measures	LeiDA	graph_measures	efficiency
21	Universe_22	strings	Hello	numbers	5.20	booleans	False	dfc_measures	LeiDA	graph_measures	clustering
22	Universe_23	strings	Hello	numbers	5.20	booleans	False	dfc_measures	JC11	graph_measures	efficiency
23	Universe_24	strings	Hello	numbers	5.20	booleans	False	dfc_measures	JC11	graph_measures	clustering
24	Universe_25	strings	world	numbers	3.14	booleans	True	dfc_measures	LeiDA	graph_measures	efficiency
25	Universe_26	strings	world	numbers	3.14	booleans	True	dfc_measures	LeiDA	graph_measures	clustering
26	Universe_27	strings	world	numbers	3.14	booleans	True	dfc_measures	JC11	graph_measures	efficiency
27	Universe_28	strings	world	numbers	3.14	booleans	True	dfc_measures	JC11	graph_measures	clustering
28	Universe_29	strings	world	numbers	3.14	booleans	False	dfc_measures	LeiDA	graph_measures	efficiency
29	Universe_30	strings	world	numbers	3.14	booleans	False	dfc_measures	LeiDA	graph_measures	clustering
30	Universe_31	strings	world	numbers	3.14	booleans	False	dfc_measures	JC11	graph_measures	efficiency
31	Universe_32	strings	world	numbers	3.14	booleans	False	dfc_measures	JC11	graph_measures	clustering
32	Universe_33	strings	world	numbers	4.00	booleans	True	dfc_measures	LeiDA	graph_measures	efficiency
33	Universe_34	strings	world	numbers	4.00	booleans	True	dfc_measures	LeiDA	graph_measures	clustering
34	Universe_35	strings	world	numbers	4.00	booleans	True	dfc_measures	JC11	graph_measures	efficiency
35	Universe_36	strings	world	numbers	4.00	booleans	True	dfc_measures	JC11	graph_measures	clustering
36	Universe_37	strings	world	numbers	4.00	booleans	False	dfc_measures	LeiDA	graph_measures	efficiency
37	Universe_38	strings	world	numbers	4.00	booleans	False	dfc_measures	LeiDA	graph_measures	clustering
38	Universe_39	strings	world	numbers	4.00	booleans	False	dfc_measures	JC11	graph_measures	efficiency
39	Universe_40	strings	world	numbers	4.00	booleans	False	dfc_measures	JC11	graph_measures	clustering
40	Universe_41	strings	world	numbers	5.20	booleans	True	dfc_measures	LeiDA	graph_measures	efficiency
41	Universe_42	strings	world	numbers	5.20	booleans	True	dfc_measures	LeiDA	graph_measures	clustering
42	Universe_43	strings	world	numbers	5.20	booleans	True	dfc_measures	JC11	graph_measures	efficiency
43	Universe_44	strings	world	numbers	5.20	booleans	True	dfc_measures	JC11	graph_measures	clustering
44	Universe_45	strings	world	numbers	5.20	booleans	False	dfc_measures	LeiDA	graph_measures	efficiency
45	Universe_46	strings	world	numbers	5.20	booleans	False	dfc_measures	LeiDA	graph_measures	clustering
46	Universe_47	strings	world	numbers	5.20	booleans	False	dfc_measures	JC11	graph_measures	efficiency
47	Universe_48	strings	world	numbers	5.20	booleans	False	dfc_measures	JC11	graph_measures	clustering

You can now run individual universes by specifying a number, or run all of them (parallelization is also supported). This example will only evaluate the first two universes (universe=[1,2]).

mverse.run(universe=[1,2], parallel=2)

Starting analysis for universe(s): [1, 2]...

Decision 1: Hello
Decision 2: 3.14
Decision 3: True
Decision 1: Hello
Decision 2: 3.14
Decision 3: True
The multiverse analysis completed without any errors.

After this, the results from all universes can be loaded:

from matplotlib import pyplot as plt

results = mverse.get_results()

# Plot the results
fig, ax = plt.subplots()
for i, universe in enumerate(results.keys()):
    ax.plot(results[universe]["graph_measure"], label=f"Universe {i+1}")

ax.set(title="Multiverse results", ylabel="Graph measure", xlabel="time")
ax.legend();

Or alternatively also as a DataFrame:

results = mverse.get_results(as_df=True)
results

	universe	graph_measure	__decisions
1	universe_1	[0.517973602484472, 0.5365295031055901, 0.5318...	{'Decision 1': 'strings', 'Value 1': 'Hello', ...
2	universe_2	[0.7096208416867087, 0.7107480051537015, 0.708...	{'Decision 1': 'strings', 'Value 1': 'Hello', ...