{
"cells": [
{
"cell_type": "markdown",
"id": "76ad5adb",
"metadata": {},
"source": [
"# Multiverse recipes\n",
"\n",
"There are many ways of specifying and customizing complex multiverse analyses. The following code snippets aim to provide some examples/recipes for this."
]
},
{
"cell_type": "markdown",
"id": "8f8786f7",
"metadata": {},
"source": [
"## Specifying classes/functions\n",
"\n",
"You can define the method either as a direct function call or by passing the arguments separately in an `args` dictionary. \n",
"Both approaches are equivalent:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "47e2ce99",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Universe | \n",
" Decision 1 | \n",
" Value 1 | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" Universe_1 | \n",
" dfc | \n",
" SW29 | \n",
"
\n",
" \n",
" | 1 | \n",
" Universe_2 | \n",
" dfc | \n",
" SW49 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Universe Decision 1 Value 1\n",
"0 Universe_1 dfc SW29\n",
"1 Universe_2 dfc SW49"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from comet.multiverse import Multiverse\n",
"\n",
"forking_paths = {\n",
" \"dfc\": [\n",
" {\n",
" \"name\": \"SW29\",\n",
" \"func\": \"comet.connectivity.SlidingWindow(ts, windowsize=29).estimate()\"\n",
" },\n",
" {\n",
" \"name\": \"SW49\",\n",
" \"func\": \"comet.connectivity.SlidingWindow\",\n",
" \"args\": {\n",
" \"time_series\": \"ts\",\n",
" \"windowsize\": 49}\n",
" }\n",
"]}\n",
"\n",
"def analysis_template():\n",
" import comet\n",
" \n",
" ts = comet.utils.load_example(\"time_series\")\n",
" dfc = {{dfc}}\n",
"\n",
"mverse = Multiverse(name=\"example_mv_recipes\")\n",
"mverse.create(analysis_template, forking_paths)\n",
"mverse.summary()"
]
},
{
"cell_type": "markdown",
"id": "1d9353c5",
"metadata": {},
"source": [
"From a functional perspective one could even pass the entire function call as a literal (as indicated by the `$` sign). \n",
"However, as this does not allow users to specify a custom name, it will look messy in the outputs."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "11e1d637",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Universe | \n",
" Decision 1 | \n",
" Value 1 | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" Universe_1 | \n",
" dfc | \n",
" $comet.connectivity.SlidingWindow(ts, windowsi... | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Universe Decision 1 Value 1\n",
"0 Universe_1 dfc $comet.connectivity.SlidingWindow(ts, windowsi..."
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from comet.multiverse import Multiverse\n",
"\n",
"forking_paths = {\n",
" \"dfc\": [\"$comet.connectivity.SlidingWindow(ts, windowsize=69).estimate()\"]\n",
"}\n",
"\n",
"def analysis_template():\n",
" import comet\n",
" \n",
" ts = comet.utils.load_example(\"time_series\")\n",
" dfc = {{dfc}}\n",
"\n",
"mverse = Multiverse(name=\"example_mv_recipes\")\n",
"mverse.create(analysis_template, forking_paths)\n",
"mverse.summary()"
]
},
{
"cell_type": "markdown",
"id": "c6da8772",
"metadata": {},
"source": [
"This works the same for graph related functions, and also any other function/class not included in Comet:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "c14ddcb0",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Universe | \n",
" Decision 1 | \n",
" Value 1 | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" Universe_1 | \n",
" functions | \n",
" comet clustering | \n",
"
\n",
" \n",
" | 1 | \n",
" Universe_2 | \n",
" functions | \n",
" bct clustering | \n",
"
\n",
" \n",
" | 2 | \n",
" Universe_3 | \n",
" functions | \n",
" scipy zscore | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Universe Decision 1 Value 1\n",
"0 Universe_1 functions comet clustering\n",
"1 Universe_2 functions bct clustering\n",
"2 Universe_3 functions scipy zscore"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from comet.multiverse import Multiverse\n",
"\n",
"forking_paths = {\n",
" \"functions\": [\n",
" {\n",
" \"name\": \"comet clustering\",\n",
" \"func\": \"comet.graph.clustering_coef(G)\",\n",
" },\n",
" {\n",
" \"name\": \"bct clustering\",\n",
" \"func\": \"bct.clustering_coef_wu(G)\",\n",
" },\n",
" {\n",
" \"name\": \"scipy zscore\",\n",
" \"func\": \"scipy.stats.zscore(G)\",\n",
" }\n",
"]}\n",
"\n",
"def analysis_template():\n",
" import comet\n",
" import bct\n",
" import scipy\n",
" \n",
" G = [[0, 1, 2], [1, 0, 3], [2, 3, 0]]\n",
" result = {{functions}}\n",
"\n",
"mverse = Multiverse(name=\"example_mv_recipes\")\n",
"mverse.create(analysis_template, forking_paths)\n",
"mverse.summary()"
]
},
{
"cell_type": "markdown",
"id": "fb461606",
"metadata": {},
"source": [
"## Varying parameters for same connectivity measures\n",
"\n",
"One can either define the the complete measures in the forking paths, or only substitute the arguments in the analysis template:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "068434f7",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Universe | \n",
" Decision 1 | \n",
" Value 1 | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" Universe_1 | \n",
" sw | \n",
" SW29 | \n",
"
\n",
" \n",
" | 1 | \n",
" Universe_2 | \n",
" sw | \n",
" SW39 | \n",
"
\n",
" \n",
" | 2 | \n",
" Universe_3 | \n",
" sw | \n",
" SW49 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Universe Decision 1 Value 1\n",
"0 Universe_1 sw SW29\n",
"1 Universe_2 sw SW39\n",
"2 Universe_3 sw SW49"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from comet.multiverse import Multiverse\n",
"\n",
"forking_paths = {\n",
" \"sw\": [\n",
" {\n",
" \"name\": \"SW29\",\n",
" \"func\": \"comet.connectivity.SlidingWindow(ts, windowsize=29).estimate()\"\n",
" },\n",
" {\n",
" \"name\": \"SW39\",\n",
" \"func\": \"comet.connectivity.SlidingWindow(ts, windowsize=39).estimate()\"\n",
" },\n",
" {\n",
" \"name\": \"SW49\",\n",
" \"func\": \"comet.connectivity.SlidingWindow(ts, windowsize=49).estimate()\"\n",
" },\n",
"]}\n",
"\n",
"def analysis_template():\n",
" import comet\n",
" \n",
" ts = comet.utils.load_example(\"time_series\")\n",
" dfc = {{sw}}\n",
"\n",
"mverse = Multiverse(name=\"example_mv_recipes\")\n",
"mverse.create(analysis_template, forking_paths)\n",
"mverse.summary()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "f3b76f5e",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Universe | \n",
" Decision 1 | \n",
" Value 1 | \n",
" Decision 2 | \n",
" Value 2 | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" Universe_1 | \n",
" size | \n",
" 29 | \n",
" shape | \n",
" rectangular | \n",
"
\n",
" \n",
" | 1 | \n",
" Universe_2 | \n",
" size | \n",
" 29 | \n",
" shape | \n",
" gaussian | \n",
"
\n",
" \n",
" | 2 | \n",
" Universe_3 | \n",
" size | \n",
" 39 | \n",
" shape | \n",
" rectangular | \n",
"
\n",
" \n",
" | 3 | \n",
" Universe_4 | \n",
" size | \n",
" 39 | \n",
" shape | \n",
" gaussian | \n",
"
\n",
" \n",
" | 4 | \n",
" Universe_5 | \n",
" size | \n",
" 49 | \n",
" shape | \n",
" rectangular | \n",
"
\n",
" \n",
" | 5 | \n",
" Universe_6 | \n",
" size | \n",
" 49 | \n",
" shape | \n",
" gaussian | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Universe Decision 1 Value 1 Decision 2 Value 2\n",
"0 Universe_1 size 29 shape rectangular\n",
"1 Universe_2 size 29 shape gaussian\n",
"2 Universe_3 size 39 shape rectangular\n",
"3 Universe_4 size 39 shape gaussian\n",
"4 Universe_5 size 49 shape rectangular\n",
"5 Universe_6 size 49 shape gaussian"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from comet.multiverse import Multiverse\n",
"\n",
"forking_paths = {\n",
" \"size\": [29, 39, 49],\n",
" \"shape\": [\"rectangular\", \"gaussian\"]\n",
" }\n",
"\n",
"def analysis_template():\n",
" import comet\n",
" \n",
" ts = comet.utils.load_example(\"time_series\")\n",
" dfc = comet.connectivity.SlidingWindow(time_series=ts, windowsize={{size}}, shape={{shape}}).estimate()\n",
"\n",
"mverse = Multiverse(name=\"example_mv_recipes\")\n",
"mverse.create(analysis_template, forking_paths)\n",
"mverse.summary()"
]
},
{
"cell_type": "markdown",
"id": "ce904035",
"metadata": {},
"source": [
"A more complex design are different connectivity measures which have different parameters. Two potential solutions are:\n",
"\n",
"1) Give the full decision space in the forking paths:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "d604de0c",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Universe | \n",
" Decision 1 | \n",
" Value 1 | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" Universe_1 | \n",
" sw | \n",
" SW29 | \n",
"
\n",
" \n",
" | 1 | \n",
" Universe_2 | \n",
" sw | \n",
" SW39 | \n",
"
\n",
" \n",
" | 2 | \n",
" Universe_3 | \n",
" sw | \n",
" FLS50 | \n",
"
\n",
" \n",
" | 3 | \n",
" Universe_4 | \n",
" sw | \n",
" FLS100 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Universe Decision 1 Value 1\n",
"0 Universe_1 sw SW29\n",
"1 Universe_2 sw SW39\n",
"2 Universe_3 sw FLS50\n",
"3 Universe_4 sw FLS100"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from comet.multiverse import Multiverse\n",
"\n",
"forking_paths = {\n",
" \"sw\": [\n",
" {\n",
" \"name\": \"SW29\",\n",
" \"func\": \"comet.connectivity.SlidingWindow(ts, windowsize=29).estimate()\"\n",
" },\n",
" {\n",
" \"name\": \"SW39\",\n",
" \"func\": \"comet.connectivity.SlidingWindow(ts, windowsize=39).estimate()\"\n",
" },\n",
" {\n",
" \"name\": \"FLS50\",\n",
" \"func\": \"comet.connectivity.FlexibleLeastSquares(ts, mu=50).estimate()\"\n",
" },\n",
" {\n",
" \"name\": \"FLS100\",\n",
" \"func\": \"comet.connectivity.FlexibleLeastSquares(ts, mu=100).estimate()\"\n",
" }\n",
"]}\n",
"\n",
"def analysis_template():\n",
" import comet\n",
" \n",
" ts = comet.utils.load_example(\"time_series\")\n",
" dfc = {{sw}}\n",
"\n",
"mverse = Multiverse(name=\"example_mv_recipes\")\n",
"mverse.create(analysis_template, forking_paths)\n",
"mverse.summary()"
]
},
{
"cell_type": "markdown",
"id": "f1d2efe1",
"metadata": {},
"source": [
"2) Use a conditional logic in the analysis script combined with a config rule. Here, combinations of `SlidingWindow` and `mu` as well as `FlexibleLeastSquares` and `windowsize` are invalid, so we exclude them from the multiverse:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "0c11e440",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Exclusion summary\n",
"-----------------\n",
"Total number of universes: 8\n",
" - Removed 8 duplicate universes (set 'deduplicate' to False if you want to keep them)\n",
" - Set 'mu' to NaN for universes matching {'method': 'SW'} (4 total).\n",
" - Set 'windowsize' to NaN for universes matching {'method': 'FLS'} (4 total).\n",
"\n",
"8 universes remain for analysis.\n"
]
},
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Universe | \n",
" Decision 1 | \n",
" Value 1 | \n",
" Decision 2 | \n",
" Value 2 | \n",
" Decision 3 | \n",
" Value 3 | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" Universe_1 | \n",
" method | \n",
" SW | \n",
" windowsize | \n",
" 29.0 | \n",
" mu | \n",
" NaN | \n",
"
\n",
" \n",
" | 1 | \n",
" Universe_2 | \n",
" method | \n",
" SW | \n",
" windowsize | \n",
" 29.0 | \n",
" mu | \n",
" NaN | \n",
"
\n",
" \n",
" | 2 | \n",
" Universe_3 | \n",
" method | \n",
" SW | \n",
" windowsize | \n",
" 39.0 | \n",
" mu | \n",
" NaN | \n",
"
\n",
" \n",
" | 3 | \n",
" Universe_4 | \n",
" method | \n",
" SW | \n",
" windowsize | \n",
" 39.0 | \n",
" mu | \n",
" NaN | \n",
"
\n",
" \n",
" | 4 | \n",
" Universe_5 | \n",
" method | \n",
" FLS | \n",
" windowsize | \n",
" NaN | \n",
" mu | \n",
" 50.0 | \n",
"
\n",
" \n",
" | 5 | \n",
" Universe_6 | \n",
" method | \n",
" FLS | \n",
" windowsize | \n",
" NaN | \n",
" mu | \n",
" 100.0 | \n",
"
\n",
" \n",
" | 6 | \n",
" Universe_7 | \n",
" method | \n",
" FLS | \n",
" windowsize | \n",
" NaN | \n",
" mu | \n",
" 50.0 | \n",
"
\n",
" \n",
" | 7 | \n",
" Universe_8 | \n",
" method | \n",
" FLS | \n",
" windowsize | \n",
" NaN | \n",
" mu | \n",
" 100.0 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Universe Decision 1 Value 1 Decision 2 Value 2 Decision 3 Value 3\n",
"0 Universe_1 method SW windowsize 29.0 mu NaN\n",
"1 Universe_2 method SW windowsize 29.0 mu NaN\n",
"2 Universe_3 method SW windowsize 39.0 mu NaN\n",
"3 Universe_4 method SW windowsize 39.0 mu NaN\n",
"4 Universe_5 method FLS windowsize NaN mu 50.0\n",
"5 Universe_6 method FLS windowsize NaN mu 100.0\n",
"6 Universe_7 method FLS windowsize NaN mu 50.0\n",
"7 Universe_8 method FLS windowsize NaN mu 100.0"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from comet.multiverse import Multiverse\n",
"\n",
"forking_paths = {\n",
" \"method\": [\"SW\", \"FLS\"],\n",
" \"windowsize\": [29, 39],\n",
" \"mu\": [50, 100] \n",
" }\n",
"\n",
"config = {\n",
" \"exclude\": [\n",
" [{\"method\": \"SW\"}, \"mu\"],\n",
" [{\"method\": \"FLS\"}, \"windowsize\"]\n",
" ]\n",
"}\n",
"\n",
"def analysis_template():\n",
" import comet\n",
"\n",
" ts = comet.utils.load_example(\"time_series\")\n",
" if {{method}} == \"SW\":\n",
" dfc = comet.connectivity.SlidingWindow(ts, windowsize={{windowsize}}).estimate()\n",
" elif {{method}} == \"FLS\":\n",
" dfc = comet.connectivity.FlexibleLeastSquares(ts, mu={{mu}}).estimate()\n",
"\n",
"mverse = Multiverse(name=\"example_mv_recipes\")\n",
"mverse.create(analysis_template, forking_paths, config)\n",
"mverse.summary()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "comet-test",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.0"
}
},
"nbformat": 4,
"nbformat_minor": 5
}