acme-demo.qmd

---
title: "ACME"
subtitle: "Flexible Scalability for Research Software"
author:
  - Stefan </br> Fürtinger
  - Katharine Shapcott
  - Joscha Tapani Schmiedt
date: August 7 2023
date-format: long
institute:
  - Ernst Strüngmann Institute (ESI) Frankfurt
  - Ernst Strüngmann Institute (ESI) Frankfurt
  - Brain Research Institute Universität Bremen
format:
  revealjs:
    theme: [white, css/esi.scss]
    slide-number: c
    menu: true
    progress: true
    chalkboard: false
      # buttons: false
    preview-links: auto
    css: css/agenda.css
    code-line-numbers: false
    highlight-style: github
    auto-play-media: true
    embed-resources: false
    pdf-separate-fragments: true
include-after-body: css/esi_borders.html
revealjs-plugins:
  - attribution
filters:
  - reveal-auto-agenda
auto-agenda:
  bullets: none
  heading: Outline
---

<!--
Copyright (c) 2023 Ernst Strüngmann Institute (ESI) for Neuroscience
in Cooperation with Max Planck Society
SPDX-License-Identifier: CC-BY-NC-SA-1.0
-->

```{=html}
<section class="title-slide slide level1 agenda-slide center">
<div class="agenda-heading">
<p>Outline</p>
</div>
<div class="agenda">
<div class="agenda-active">
<p>I Big Data Is Great</p>
</div>
<div class="agenda-active">
<p>II Why ACME?</p>
</div>
<div class="agenda-active">
<p>III An Offer You Can Refuse</p>
</div>
</div>
</section>
```

# I Big Data Is Great

## Big Data: A Topical Example

::: {.r-stack}
![](imgs/fastMRI_website.png){.fragment .fade-in-then-semi-out .absolute width="85%" top="1.5em" left="1em"}
![](imgs/fastMRI_knee.png){.fragment .absolute width="40%" left="1em" top="4em"}
![](imgs/fastMRI_brain.png){.fragment .absolute width="45%" right="1em" top="1.5em"}
:::

::: {.attribution}
See [fastmri.med.nyu.edu](https://fastmri.med.nyu.edu/)
:::

# II Why ACME?

## Get SLURMed {transition="fade-in" auto-animate=true}

```{.python data-id="code1" code-line-numbers="|5-6,9-11,14-15"}
# Which subject do we want to analyze?
subIdx = 0

# Take stock of data on disk
data = datasets.fetch_development_fmri(age_group="adult")
atlasCoords = datasets.fetch_coords_power_2011()

# Extract fMRI time-series averaged within spheres @ atlas coords
mask = NiftiSpheresMasker(seeds=atlasCoords)
timeseries = mask.fit_transform(data.func[subIdx],
                                confounds=data.confounds[subIdx])

# Compute functional connectivity b/w brain regions
estimator = GraphicalLassoCV()
estimator.fit(timeseries)

# Inspect results
plotting.plot_connectome(estimator.covariance_)
```

::: {.r-stack}
![](imgs/subject0.png){.fragment fragment-index=1 .fade-in-then-out .absolute top="3em" left="5em" width="55%"}
:::

## Get SLURMed {auto-animate=true}

```{.python data-id="code1" filename="connectome.py" code-line-numbers="1-16|" style="font-size: 0.8em; "}
def compute_connectome(subIdx):

      # Take stock of data on disk
      data = datasets.fetch_development_fmri(age_group="adult")
      atlasCoords = datasets.fetch_coords_power_2011()

      # Extract fMRI time-series averaged within spheres @ atlas coords
      masker = NiftiSpheresMasker(seeds=atlasCoords)
      timeseries = masker.fit_transform(data.func[subIdx],
                                        confounds=data.confounds[subIdx])

      # Compute functional connectivity b/w brain regions
      estimator = GraphicalLassoCV()
      estimator.fit(timeseries)
      return estimator.covariance_

if __name__ == "__main__":

      # Compute functional connectivity of subject and save result
      con = compute_connectome(sys.argv[1])
      np.save("con_{}".format(sys.argv[1]), con)
```

## Get SLURMed {auto-animate=true}

```{.python data-id="code1" filename="connectome.py"}
```

. . .

```{.bash data-id="code3" filename="run_connectome.sh"}
#!/bin/bash
#
# SLURM script for computing per-subject connectomes
#
#SBATCH -J connectome_batch # Common name for the job-array
#SBATCH -p myPartition      # Partition
#SBATCH -c 2                # Use two cores
#SBATCH -t 0-2:00           # Max run-time of 2 hours
#SBATCH --mem 4000          # Request 4 GB of RAM
#SBATCH -o con_%A_%a.out    # Redirect stdout/stderr to file
#SBATCH --array=1-33        # Define job-array

source /path/to/conda/etc/profile.d/conda.sh
conda activate myenv
srun python connectome.py "$SLURM_ARRAY_TASK_ID"
```

## Get SLURMed {auto-animate=true}

```{.python data-id="code1" filename="connectome.py"}
```

```{.bash data-id="code3" filename="run_connectome.sh"}
```

. . .

```{data-id="code4"}
sbatch run_connectome.sh
```

## Get SLURMed {auto-animate=true}

```{.python data-id="code1" filename="connectome.py"}
```

```{.bash data-id="code3" filename="run_connectome.sh"}
```

```{data-id="code4"}
sbatch run_connectome.sh
Submitted batch job 21607933

squeue --me
ACCOUNT JOBID PARTITION NODELIST PRIORITY TIME STATE
fuertingers 21607933_3 8GBXS esi-svhpc46 25228545 0:11 RUNNING
fuertingers 21607933_4 8GBXS esi-svhpc46 25228545 0:11 RUNNING
fuertingers 21607933_5 8GBXS esi-svhpc29 25228545 0:11 RUNNING
fuertingers 21607933_2 8GBXS esi-svhpc24 25228545 0:15 RUNNING
fuertingers 21607933_1 8GBXS esi-svhpc24 25228545 0:16 RUNNING
...
```

## Get SLURMed {transition="fade"}

![](imgs/sbatch_schedmd.gif)

## It All Started With A gist...

![](imgs/acme_gist.png){.position .absolute height="80%" left="2em"}

## Asynchronous Computing Made ESI

![](imgs/acme_logo.png){.position .absolute left="0em" top="2em" width="22%"}

::: {.incremental style="font-size: 0.9em; " .position .absolute left="8em" right="0em" top="2.5em"}
- accelerates "simple", i.e., [embarassingly parallel]{.esi-emph}, workloads
- [wraps]{.esi-emph} sequential code and [maps]{.esi-emph} on parallel computing hardware
- DIY parallelization via [context manager]{.esi-emph} `ParallelMap` in Python
- built on top of [[dask](https://www.dask.org/)]{.esi-emph} and
  [[dask-jobqueue](https://jobqueue.dask.org/en/latest/)]{.esi-emph} to
  integrate with HPC clusters
  ![](imgs/dask.svg){.position .absolute left="-8em" top="6em" width="30%"}
  ![](imgs/dask-jobqueue.png){.position .absolute left="-8em" top="9em" width="30%"}
:::

. . .

::: {.position .absolute top="13.5em" left="3em"}
```{.bash style="font-size: 1.3em; "}
pip install esi-acme
conda install -c conda-forge esi-acme
```
:::

## Using ACME

:::: columns

::: {.column width="50%" .position .absolute left="1em"}

::: {.fragment .fade-in style="font-size: 0.6em; font-family: monospace; " .position .absolute top="2.5em"}
[def]{style="color: red; "} func(
  </br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  [arg1]{style="color: blue; "},
  </br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  [arg2]{style="color: blue; "},
  </br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  ...,
  </br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  [kwarg1=val1]{style="color: mediumorchid; "},
  </br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  [kwarg2=val2]{style="color: mediumorchid; "},
  </br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  ...):
  </br>&nbsp;&nbsp;&nbsp;...
  </br>&nbsp;&nbsp;&nbsp;do_this()
  </br>&nbsp;&nbsp;&nbsp;...
  </br>&nbsp;&nbsp;&nbsp;do_that()
  </br>&nbsp;&nbsp;&nbsp;...
  </br>&nbsp;&nbsp;&nbsp;[return]{style="color: red; "} something
:::

:::

::: {.column width="60%" .position .absolute right="0em"}

::: {.fragment .fade-in style="font-size: 0.6em; font-family: monospace; " .position .absolute top="-0.5em"}
from [acme]{.esi-color} import [ParallelMap]{.esi-color}
:::

::: {.fragment .fade-in style="font-size: 0.6em; font-family: monospace; " .position .absolute top="1.0em"}
with [ParallelMap]{.esi-color}(
:::

::: {.fragment .fade-up style="font-size: 0.6em; font-family: monospace; " .position .absolute top="1.0em"}
  </br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  func,
:::

::: {.fragment .fade-up style="font-size: 0.6em; font-family: monospace; " .position .absolute top="2.4em"}
  </br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  [[arg11,arg12,...]]{style="color: blue; "},
:::

::: {.fragment .fade-up style="font-size: 0.6em; font-family: monospace; " .position .absolute top="3.7em"}
  </br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  [[arg21,arg22,...]]{style="color: blue; "},
:::

::: {.fragment .fade-up style="font-size: 0.6em; font-family: monospace; " .position .absolute top="5.0em"}
  </br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  ...,
:::

::: {.fragment .fade-up style="font-size: 0.6em; font-family: monospace; " .position .absolute top="6.3em"}
  </br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  [kwarg1=[val11,val12,...]]{style="color: mediumorchid; "},
:::

::: {.fragment .fade-up style="font-size: 0.6em; font-family: monospace; " .position .absolute top="7.6em"}
  </br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  [kwarg2=[val21,val22,...]]{style="color: mediumorchid; "},
:::

::: {.fragment .fade-up style="font-size: 0.6em; font-family: monospace; " .position .absolute top="8.9em"}
  </br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  ...,
:::

::: {.fragment .fade-up style="font-size: 0.6em; font-family: monospace; " .position .absolute top="10.2em"}
  </br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  [kw1=v1]{.esi-color},
:::

::: {.fragment .fade-up style="font-size: 0.6em; font-family: monospace; " .position .absolute top="11.5em"}
  </br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  [kw2=v2]{.esi-color},
:::

::: {.fragment .fade-up style="font-size: 0.6em; font-family: monospace; " .position .absolute top="12.8em"}
  </br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  ...) as [pmap]{.esi-color}:
:::

::: {.fragment .fade-up style="font-size: 0.6em; font-family: monospace; " .position .absolute top="14.1em"}
  </br>&nbsp;&nbsp;&nbsp;&nbsp;
  [pmap]{.esi-color}.compute()
:::

::: {.fragment .fade-left style="font-size: 1.0em; " .position .absolute top="11em"}
  Now to [actually]{.esi-emph} using ACME
:::

:::

::::

## An Unpractical Example

```{.python}
def f(x, y, z=3):
    return (x + y) * z
```

[Objective:]{.esi-emph} Evaluate `f` for four different values of `x` and `y = 4`

. . .

`f(`[2]{.esi-emph}`, 4, z=3) = 18`

`f(`[4]{.esi-emph}`, 4, z=3) = 24`

`f(`[6]{.esi-emph}`, 4, z=3) = 30`

`f(`[8]{.esi-emph}`, 4, z=3) = 36`

## An Unpractical Example {auto-animate="true"}

```{.python}
def f(x, y, z=3):
    return (x + y) * z
```

## An Unpractical Example {auto-animate="true"}

```{.python}
from acme import ParallelMap

def f(x, y, z=3):
    return (x + y) * z

with ParallelMap(f, [2, 4, 6, 8], 4) as pmap:
    pmap.compute()
```

. . .

[[Demo Time!]{.esi-emph}]{.position .absolute bottom="4em" left="5em" style="font-size: 1.5em; "}

## Back To SLURM Connectomes...  {auto-animate="true"}

. . .

```{.python data-id="pmap" filename="connectome.py" style="font-size: 0.8em; "}
def compute_connectome(subIdx):

      # Take stock of data on disk
      data = datasets.fetch_development_fmri(age_group="adult")
      atlasCoords = datasets.fetch_coords_power_2011()

      # Extract fMRI time-series averaged within spheres @ atlas coords
      masker = NiftiSpheresMasker(seeds=atlasCoords)
      timeseries = masker.fit_transform(data.func[subIdx],
                                        confounds=data.confounds[subIdx])

      # Compute functional connectivity b/w brain regions
      estimator = GraphicalLassoCV()
      estimator.fit(timeseries)
      return estimator.covariance_

if __name__ == "__main__":

      # Compute functional connectivity of subject and save result
      con = compute_connectome(sys.argv[1])
      np.save("con_{}".format(sys.argv[1]), con)
```

## Back To SLURM Connectomes...  {auto-animate=true}

```{.python data-id="pmap" filename="connectome.py" style="font-size: 0.8em; "}
def compute_connectome(subIdx):

      # Take stock of data on disk
      data = datasets.fetch_development_fmri(age_group="adult")
      atlasCoords = datasets.fetch_coords_power_2011()

      # Extract fMRI time-series averaged within spheres @ atlas coords
      masker = NiftiSpheresMasker(seeds=atlasCoords)
      timeseries = masker.fit_transform(data.func[subIdx],
                                        confounds=data.confounds[subIdx])

      # Compute functional connectivity b/w brain regions
      estimator = GraphicalLassoCV()
      estimator.fit(timeseries)
      return estimator.covariance_

if __name__ == "__main__":

      # Compute functional connectivity of subject and save result
      with ParallelMap(compute_connectome, range(21)) as pmap:
          results = pmap.compute()
```

. . .

[...[or]{.esi-emph}]{.position .absolute left="10em" bottom="1em"}

## Back To SLURM Connectomes... {auto-animate="true"}

### ...in Jupyter

```{.python data-id="jupyter1" style="font-size: 0.8em; "}
[1]: from connectome import compute_connectome, subjectList
     from acme import ParallelMap
```

## Back To SLURM Connectomes... {auto-animate="true"}

### ...in Jupyter

```{.python data-id="jupyter1" style="font-size: 0.8em; "}
[1]: from connectome import compute_connectome, subjectList
     from acme import ParallelMap
     from dask_jobqueue import SLURMCluster
```

. . .

```{.python data-id="jupyter2" style="font-size: 0.8em; "}
[2]: cluster = SLURMCluster(queue="8GBXS", cores=1, memory="8GB", processes=1,
                            local_directory="/path/to/somewhere", python=sys.executable,
                            job_directives_skip=["-t", "--mem"])
     cluster.scale(10)
```

## Back To SLURM Connectomes... {auto-animate=true}

### ...in Jupyter

```{.python data-id="jupyter1" style="font-size: 0.8em; "}
[1]: from connectome import compute_connectome, subjectList
     from acme import ParallelMap
```

## Back To SLURM Connectomes... {auto-animate=true}

### ...in Jupyter

```{.python data-id="jupyter1" style="font-size: 0.8em; "}
[1]: from connectome import compute_connectome, subjectList
     from acme import ParallelMap, esi_cluster_setup
```

. . .

```{.python data-id="jupyter2" style="font-size: 0.8em; "}
[2]: myClient = esi_cluster_setup(n_workers=10, partition="8GBXS")
```

. . .

[[Demo Time!]{.esi-emph}]{.position .absolute bottom="4em" left="5em" style="font-size: 1.5em; "}

## Back To SLURM Connectomes...

### ...in Jupyter

```{.python data-id="jupyter1" style="font-size: 0.8em; "}
[1]: from connectome import compute_connectome, subjectList
     from acme import ParallelMap, esi_cluster_setup
```

```{.python data-id="jupyter2" style="font-size: 0.8em; "}
[2]: myClient = esi_cluster_setup(n_workers=10, partition="8GBXS")
```
:::{.fragment .fade-up style="font-size: 0.4em; font-family: monospace; "}
`<slurm_cluster_setup>`{style="color: black; "} SLURM workers ready: 10/10 	[elapsed time 00:13 | timeout at 01:00]</br>
[ACME [- INFO -]{style="color: DodgerBlue; "}]{style="font-weight: bold; "} Parallel computing client ready, dashboard accessible at http://10.100.32.17:37981/status
:::

. . .

```{.python data-id="jupyter3" style="font-size: 0.8em; "}
[3]: with ParallelMap(compute_connectome, subjectList, result_shape=(264, 264, None))) as pmap:
         results = pmap.compute()
```

:::{.fragment .fade-up style="font-size: 0.4em; font-family: monospace; "}
[ACME [> ANNOUNCE <]{style="color: DodgerBlue; "}  This is ACME v. 2023.4]{style="font-weight: bold; "}</br>
[ACME [- INFO -]{style="color: DodgerBlue; "}]{style="font-weight: bold; "} Attaching to parallel computing client <Client: 'tcp://10.100.32.17:32851' processes=10 threads=10, memory=74.50 GiB>
:::

::: {.fragment .fade-in data-id="box1" style="background: rgba(0,0,0,0); width: 95%; height: 1em; border: 0.1em solid red; " .absolute top="10.3em" left="-0.2em" .fade}
:::


## Back To SLURM Connectomes... {auto-animate=true auto-animate-unmatched="false" transition="none"}

### ...in Jupyter

```{.python data-id="jupyter1" style="font-size: 0.8em; "}
[1]: from connectome import compute_connectome, subjectList
     from acme import ParallelMap, esi_cluster_setup
```

```{.python data-id="jupyter2" style="font-size: 0.8em; "}
[2]: myClient = esi_cluster_setup(n_workers=10, partition="8GBXS")
```
:::{data-id="jupyter2" style="font-size: 0.4em; font-family: monospace; "}
`<slurm_cluster_setup>`{style="color: black; "} SLURM workers ready: 10/10 	[elapsed time 00:13 | timeout at 01:00]</br>
[ACME [- INFO -]{style="color: DodgerBlue; "}]{style="font-weight: bold; "} Parallel computing client ready, dashboard accessible at http://10.100.32.17:37981/status
:::

```{.python data-id="jupyter3" style="font-size: 0.8em; "}
[3]: with ParallelMap(compute_connectome, subjectList, result_shape=(264, 264, None))) as pmap:
         results = pmap.compute()
```
:::{.scrollable style="font-size: 0.4em; font-family: monospace; "}
[ACME [> ANNOUNCE <]{style="color: DodgerBlue; "} This is ACME v. 2023.4]{style="font-weight: bold; "}</br>
[ACME [- INFO -]{style="color: DodgerBlue; "}]{style="font-weight: bold; "} Attaching to parallel computing client <Client: 'tcp://10.100.32.17:32851' processes=10 threads=10, memory=74.50 GiB></br>
[ACME [- INFO -]{style="color: DodgerBlue; "}]{style="font-weight: bold; "} Preparing 21 parallel calls of `compute_connectome` using 10 workers</br>
100% |██████████| 21/21 [02:00<00:00]</br>
[ACME [> ANNOUNCE <]{style="color: DodgerBlue; "} SUCCESS!]{style="font-weight: bold; "}</br>
[ACME [- INFO -]{style="color: DodgerBlue; "}]{style="font-weight: bold; "} Results have been saved to /cs/home/fuertingers/ACME_20230720-122528-965199/compute_connectome.h5
:::

::: {data-id="box1" style="background: rgba(0,0,0,0); width: 95%; height: 1em; border: 0.1em solid red; " .absolute top="10.3em" left="-0.2em" .fade}
:::

## Back To SLURM Connectomes... {auto-animate=true auto-animate-unmatched="false"}

### ...in Jupyter

```{.python data-id="jupyter1" style="font-size: 0.8em; "}
[1]: from connectome import compute_connectome, subjectList
     from acme import ParallelMap, esi_cluster_setup
```

```{.python data-id="jupyter2" style="font-size: 0.8em; "}
[2]: myClient = esi_cluster_setup(n_workers=10, partition="8GBXS")
```
:::{data-id="jupyter2" style="font-size: 0.4em; font-family: monospace; "}
`<slurm_cluster_setup>`{style="color: black; "} SLURM workers ready: 10/10 	[elapsed time 00:13 | timeout at 01:00]</br>
[ACME [- INFO -]{style="color: DodgerBlue; "}]{style="font-weight: bold; "} Parallel computing client ready, dashboard accessible at http://10.100.32.17:37981/status
:::

```{.python data-id="jupyter3" style="font-size: 0.8em; "}
[3]: with ParallelMap(compute_connectome, subjectList, result_shape=(264, 264, None))) as pmap:
         results = pmap.compute()
```
:::{.scrollable data-id="jupyter3" style="font-size: 0.4em; font-family: monospace; "}
[ACME [> ANNOUNCE <]{style="color: DodgerBlue; "} This is ACME v. 2023.4]{style="font-weight: bold; "}</br>
[ACME [- INFO -]{style="color: DodgerBlue; "}]{style="font-weight: bold; "} Attaching to parallel computing client <Client: 'tcp://10.100.32.17:32851' processes=10 threads=10, memory=74.50 GiB></br>
[ACME [- INFO -]{style="color: DodgerBlue; "}]{style="font-weight: bold; "} Preparing 21 parallel calls of `compute_connectome` using 10 workers</br>
100% |██████████| 21/21 [02:00<00:00]</br>
[ACME [> ANNOUNCE <]{style="color: DodgerBlue; "} SUCCESS!]{style="font-weight: bold; "}</br>
[ACME [- INFO -]{style="color: DodgerBlue; "}]{style="font-weight: bold; "} Results have been saved to /cs/home/fuertingers/ACME_20230720-122528-965199/compute_connectome.h5
:::

::: {data-id="box1" style="background: rgba(0,0,0,0); width: 83%; height: 1em; border: 0.1em solid red; " .absolute top="12.9em" left="-0.2em" .fade}
:::

. . .

[...what?]{.position .absolute left="10em" bottom="1.5em"}

## Back To SLURM Connectomes...

### ...in Jupyter

```{.python data-id="jupyter4" style="font-size: 0.8em; "}
[4]: pmap.results_container
```
. . .

```{data-id="jupyter4" style="font-size: 0.4em; "}
'/cs/home/fuertingers/ACME_20230720-122528-965199/compute_connectome.h5'
```

. . .

```{.python data-id="jupyter5" style="font-size: 0.8em; "}
[5]: results
```

. . .

```{data-id="jupyter5" style="font-size: 0.4em; "}
['/cs/home/fuertingers/ACME_20230720-122528-965199/compute_connectome_payload/compute_connectome_0.h5',
 '/cs/home/fuertingers/ACME_20230720-122528-965199/compute_connectome_payload/compute_connectome_1.h5',
 '/cs/home/fuertingers/ACME_20230720-122528-965199/compute_connectome_payload/compute_connectome_2.h5',
 '/cs/home/fuertingers/ACME_20230720-122528-965199/compute_connectome_payload/compute_connectome_3.h5',
 '/cs/home/fuertingers/ACME_20230720-122528-965199/compute_connectome_payload/compute_connectome_4.h5']
```

::: {.incremental style="font-size: 0.85em; "}
- single [aggregate results file]{.esi-emph} `compute_connectome.h5`
  only points to data written by workers
- each worker saves results [independently]{.esi-emph} to files in the `*_payload` directory
- leverages [HDF5 Virtual Datasets]{.esi-emph}
:::


## Communication via Filesystem

![](imgs/virtual_dataset_illustration_ii.png){.fragment .fade-in-then-out .absolute top="2em" right="1em" width="100%"}


# III An Offer You Can Refuse

## Why Python? {auto-animate="true"}

. . .

Home to [modern ML/AI libraries]{.esi-emph}

```{.python style="font-size: 0.8em; "}
from torch.optim import Adam
from torch.nn import Sequential
from acme import ParallelMap

def hyper_par_tuning(learning_rate, epochs=10):
  model = Sequential(...)
  optimizer = Adam(model.parameters(), lr=learning_rate, betas=(0.5, 0.999))
  ...
  for epoch in range(epochs):
    classifications = model(inputs)
    loss = criterion(classifications, labels)
    loss.backward()
    optimizer.step()
    ...
```

## Why Python? {auto-animate="true"}

Home to [modern ML/AI libraries]{.esi-emph}

```{.python style="font-size: 0.8em; "}
from torch.optim import Adam
from torch.nn import Sequential
from acme import ParallelMap

def hyper_par_tuning(learning_rate, epochs=10):
  model = Sequential(...)
  optimizer = Adam(model.parameters(), lr=learning_rate, betas=(0.5, 0.999))
  ...
  for epoch in range(epochs):
    classifications = model(inputs)
    loss = criterion(classifications, labels)
    loss.backward()
    optimizer.step()
    ...
with ParallelMap(hyper_par_tuning, np.linspace(1e-3, 1e-6, 100)) as pmap:
  pmap.compute()
```

## Why Python? {auto-animate="true"}

[Interfaces]{.esi-emph}...

. . .

...with [R]{.esi-emph}

```{.R style="font-size: 0.8em; " filename="some_code.R"}
sample_stats <- function(d_mu, d_sd) {
    samp <- rnorm(10^6, d_mu, d_sd)
    c(s_mu = mean(samp), s_sd = sd(samp))
}
```

::: {style="height: 1em; "}
:::

. . .

```{.python style="font-size: 0.8em; "}
from rpy2.robjects.packages import STAP

def sample_stats(mu, sd):
  with open("some_code.R", "r") as Rfile:
    Rcode = Rfile.read()
  some_code = STAP(Rcode, "sample_stats")
  return np.array(some_code.sample_stats(mu, sd))

with ParallelMap(some_code.sample_stats, [2, 4, 6, 8], [3, 5, 7, 9]) as pmap:
  pmap.compute()
```

## Why Python? {auto-animate="true"}

[Interfaces]{.esi-emph}...

...with [system binaries]{.esi-emph}

```{.python style="font-size: 0.8em; "}
import subprocess

def freesurf_preproc(subIdx):
  cmd = "recon-all -s sub-{subj:d} -i sub-{subj:d}_ses-BL_T1w.nii.gz -all"
  subprocess.run(cmd.format(sub=subIdx), text=True, shell=True, check=True)

with ParallelMap(freesurf_preproc, np.arange(101, 151)) as pmap:
  pmap.compute()
```

. . .

...and [FORTRAN]{.esi-emph}, [Java]{.esi-emph}, [Octave]{.esi-emph}, ...

## ACME {visibilty="uncounted"}

## ACME is [not]{style="color: red"}

::: {.incremental style="font-size: 1.0em; " .absolute .position top="3em"}
- a general purpose parallelization framework
- up for tasks requiring inter-worker communication
- built for for processing shared distributed data structures
- the best way to tackle CFD problems/PDE discretizations
- tuned for bleeding edge performance
- another `sbatch` wrapper
:::

## ACME [is]{.esi-emph}

::: {.incremental style="font-size: 0.9em; " .absolute .position bottom="1em"}
- made for [easy-to-use parallelization]{.esi-emph} of code not written with concurrency in mind
- [application-agnostic]{.esi-emph}: has been used for math optimization,
  neural networks, fMRI and is the parallelization engine of
  ![](imgs/syncopy_logo.svg){.absolute .position top="3.7em" right="2.8em" width="15%"}
- based on existing [robust parallel computing libraries]{.esi-emph} (`dask`, `dask-jobqueue`)
- [small network footprint]{.esi-emph} by shifting concurrency to the filesystem
- fully [open-source on [GitHub](https://github.com/esi-neuroscience/acme)]{.esi-emph} PRs welcome!
- open to [[include more HPC centers]{.esi-epmh}]{.esi-emph}

  ``<your-institution-here>_cluster_setup``
:::

## {visibilty="uncounted"}

::: {style="height: 0.8em; "}
:::

::: {.r-fit-text}
Thank You For Your Attention!
:::

[**GitHub**]{.absolute top="10.5em" left="0em" style="font-size: 0.9em; "}

![](imgs/me.jpeg){.absolute top="4em" left="3em" width="16%"}
[Stefan]{.absolute top="9.5em" left="4.5em" style="font-size: 0.9em; "}
[@pantaray]{.absolute top="10.5em" left="3.5em" style="font-size: 0.9em; "}

![](imgs/katharine.jpg){.absolute top="4em" left="10.5em" width="16%"}
[Katharine]{.absolute top="9.5em" left="12em" style="font-size: 0.9em; "}
[@KatharineShapcott]{.absolute top="10.5em" left="10em" style="font-size: 0.9em; "}

![](imgs/joscha.jpeg){.absolute top="4em" left="18.7em" width="16%"}
[Joscha]{.absolute top="9.5em" left="21.7em" style="font-size: 0.9em; "}
[@joschaschmiedt]{.absolute top="10.5em" left="20em" style="font-size: 0.9em; "}

![](imgs/acme_logo.png){.absolute top="11em" left="2em" width="15%"}
[[https://github.com/esi-neuroscience/acme](https://github.com/esi-neuroscience/acme)]{.absolute top="14em" left="8em" style="font-size: 0.9em; "}

[**Contact**: [stefan.fuertinger@esi-frankfurt.de](stefan.fuertinger@esi-frankfurt.de)]{.absolute top="16.5em" left="5em" style="font-size: 0.9em; "}

## {background-image="imgs/acme_nilearn_demo.gif" background-opacity="0.2" visibilty="uncounted"}

::: {style="height: 0.8em; "}
:::

::: {.r-fit-text}
Thank You For Your Attention!
:::

[**GitHub**]{.absolute top="10.5em" left="0em" style="font-size: 0.9em; "}

![](imgs/me.jpeg){.absolute top="4em" left="3em" width="16%"}
[Stefan]{.absolute top="9.5em" left="4.5em" style="font-size: 0.9em; "}
[@pantaray]{.absolute top="10.5em" left="3.5em" style="font-size: 0.9em; "}

![](imgs/katharine.jpg){.absolute top="4em" left="10.5em" width="16%"}
[Katharine]{.absolute top="9.5em" left="12em" style="font-size: 0.9em; "}
[@KatharineShapcott]{.absolute top="10.5em" left="10em" style="font-size: 0.9em; "}

![](imgs/joscha.jpeg){.absolute top="4em" left="18.7em" width="16%"}
[Joscha]{.absolute top="9.5em" left="21.7em" style="font-size: 0.9em; "}
[@joschaschmiedt]{.absolute top="10.5em" left="20em" style="font-size: 0.9em; "}

![](imgs/acme_logo.png){.absolute top="11em" left="2em" width="15%"}
[[https://github.com/esi-neuroscience/acme](https://github.com/esi-neuroscience/acme)]{.absolute top="14em" left="8em" style="font-size: 0.9em; "}

[**Contact**: [stefan.fuertinger@esi-frankfurt.de](stefan.fuertinger@esi-frankfurt.de)]{.absolute top="16.5em" left="5em" style="font-size: 0.9em; "}