From 3625a2906d773ec287a58b3ef2cdedba2852623c Mon Sep 17 00:00:00 2001 From: Jason Boxman Date: Mon, 13 May 2024 22:40:17 -0400 Subject: [PATCH] Refresh Wave documentation --- docs/api.mdx | 14 ++ docs/cli/build-conda.mdx | 28 --- docs/cli/build-directory.mdx | 34 --- docs/cli/build-docker.mdx | 60 ------ docs/cli/build-freeze.mdx | 31 --- docs/cli/build-singularity.mdx | 47 ---- docs/cli/build-spack.mdx | 29 --- docs/cli/index.mdx | 2 +- docs/cli/install.mdx | 41 ---- docs/cli/reference.mdx | 271 ++++++++++++++++++++++++ docs/get-started.mdx | 377 +++++++++++++++++++++++++++++++++ docs/guide.mdx | 161 -------------- docs/index.mdx | 75 ++++--- docs/metrics.mdx | 8 +- docs/nextflow.mdx | 215 +++++++++++++++++++ docs/provisioning.mdx | 88 ++++++++ docs/sidebar.json | 34 ++- 17 files changed, 1036 insertions(+), 479 deletions(-) delete mode 100644 docs/cli/build-conda.mdx delete mode 100644 docs/cli/build-directory.mdx delete mode 100644 docs/cli/build-docker.mdx delete mode 100644 docs/cli/build-freeze.mdx delete mode 100644 docs/cli/build-singularity.mdx delete mode 100644 docs/cli/build-spack.mdx delete mode 100644 docs/cli/install.mdx create mode 100644 docs/cli/reference.mdx create mode 100644 docs/get-started.mdx delete mode 100644 docs/guide.mdx create mode 100644 docs/nextflow.mdx create mode 100644 docs/provisioning.mdx diff --git a/docs/api.mdx b/docs/api.mdx index 019a95b39..0d646c0b3 100644 --- a/docs/api.mdx +++ b/docs/api.mdx @@ -4,6 +4,20 @@ title: API reference This page summarizes the API provided by the Wave container service. +**API limits** + +The Wave service implements API rate limits for API calls. Authenticated users have higher rate limits than anonymous users. + +If an access token is provided, the following rate limits apply: + +- 100 container builds per hour +- 1,000 container pulls per minute + +If an access token isn't provided, the following rate limits apply: + +- 25 container builds per day +- 250 container pulls per hour + ## POST `/container-token` Deprecated endpoint allows you to submit a request to access a private container registry via Wave, or build a container image on-the-fly with a Dockerfile or Conda recipe file. diff --git a/docs/cli/build-conda.mdx b/docs/cli/build-conda.mdx deleted file mode 100644 index be13cb2dd..000000000 --- a/docs/cli/build-conda.mdx +++ /dev/null @@ -1,28 +0,0 @@ ---- -title: Build a container from Conda packages ---- - -The Wave CLI supports building a container from a list of [Conda] packages. - -## Related CLI arguments - -Conda builds support the following arguments: - -- `--conda-base-image`: A base image for installing Conda packages. The default value is `mambaorg/micromamba:1.5.1`. -- `--conda-channels`: One or more comma separated channels. The default value is ` seqera,bioconda,conda-forge,defaults`. -- `--conda-file`: A [Conda lock file][conda-lock]. Can be a local file or a URL to a remote file. -- `--conda-package`: A Conda package to install. Can be specified multiple times. Expression are supported, such as `bioconda::samtools=1.17` or `samtools>=1.0,<1.17`. -- ` --conda-run-command`: A Docker `RUN` command used when the container is built. Can be specified multiple times. - -## Example usage - -In the following example, a container with the `samtools` and `bamtools` packages is built: - -``` -wave \ - --conda-package bamtools=2.5.2 \ - --conda-package samtools=1.17 -``` - -[Conda]: https://anaconda.org/anaconda/repo -[conda-lock]: https://github.com/conda/conda-lock diff --git a/docs/cli/build-directory.mdx b/docs/cli/build-directory.mdx deleted file mode 100644 index c627d05ab..000000000 --- a/docs/cli/build-directory.mdx +++ /dev/null @@ -1,34 +0,0 @@ ---- -title: Augment a container with a directory ---- - -The Wave CLI supports container augmentation with a specified directory. You can use container augmentation to dynamically add a layer to your container, so you can inject scripts or configuration files as a new layer. - -The following limitations apply: - -- A file must be no larger than 1 MB each. -- A directory must be no larger than 10 MB, inclusive of all files. -- A base image must be specified. - -## Related CLI arguments - -The following arguments are used for a directory build: - -- `--layer`: A directory that contains layer content. -- `--image` or `-i`: An existing container image. The default image registry is `docker.io`. Specify an image name such as `alpine:latest` or an image URL such as `public.ecr.aws/docker/library/busybox`. - -## Example usage - -Create a new context directory: - -``` -mkdir -p new-layer/usr/local/bin -printf 'echo Hello world!' > new-layer/usr/local/bin/hello.sh -chmod +x new-layer/usr/local/bin/hello.sh -``` - -Use the CLI to build the image and run the result with Docker: - -``` -docker run $(wave -i alpine --layer new-layer) sh -c hello.sh -``` diff --git a/docs/cli/build-docker.mdx b/docs/cli/build-docker.mdx deleted file mode 100644 index 6399abcf5..000000000 --- a/docs/cli/build-docker.mdx +++ /dev/null @@ -1,60 +0,0 @@ ---- -title: Build a container from a Dockerfile ---- - -The Wave CLI supports building a container from a `Dockerfile`. Specifying an optional build context allows the use of `ADD` and `COPY` commands in a Dockerfile. - -:::note -Building a Dockerfile that requires `--build-arg` for build time variables isn't currently supported. -::: - -## Related CLI arguments - -- `--containerfile` or `-f`: A Dockerfile to build. Build args aren't currently supported. -- `--context`: A directory that contains the context for the build. - -## Example usage - -In the following example `Dockerfile`, several packages are installed: - -``` -cat << EOF > ./Dockerfile -FROM alpine - -RUN apk update && apk add bash cowsay \ - --update-cache \ - --repository https://alpine.global.ssl.fastly.net/alpine/edge/community \ - --repository https://alpine.global.ssl.fastly.net/alpine/edge/main \ - --repository https://dl-3.alpinelinux.org/alpine/edge/testing -EOF -``` - -Build and run the container based on the Dockerfile in the previous example by running the following command: - -``` -container=$(wave --containerfile ./Dockerfile) -docker run --rm $container cowsay "Hello world" -``` - -In the following example `Dockerfile`, a local context is used: - -``` -cat << EOF > ./Dockerfile -FROM alpine -ADD hello.sh /usr/local/bin/ -EOF -``` - -Create the shell script referenced in the previous example by running the following commands in your terminal: - -``` -mkdir -p build-context/ -printf 'echo Hello world!' > build-context/hello.sh -chmod +x build-context/hello.sh -``` - -Build and run the container based on the Dockerfile in the previous example by running the following command: - -``` -docker run $(wave -f Dockerfile --context build-context) sh -c hello.sh -``` diff --git a/docs/cli/build-freeze.mdx b/docs/cli/build-freeze.mdx deleted file mode 100644 index 72f83482e..000000000 --- a/docs/cli/build-freeze.mdx +++ /dev/null @@ -1,31 +0,0 @@ ---- -title: Build a container and freeze to a container registry ---- - -The Wave CLI supports building a container and persisting the container to a container registry, such as DockerHub. You can refer to this frozen container image in a Dockerfile or [Nextflow] pipeline in the same way as any other container. - -To freeze a container, you must ensure the following conditions are met: - -- You created a Seqera Platform access token. -- You specified the destination container registry credentials in Seqera Platform. -- You specify the Seqera Platform access token via either the `TOWER_ACCESS_TOKEN` environment variable or the `--tower-token` Wave command line option. - -## Related CLI arguments - -The following arguments are used to freeze a container build: - -- `--build-repo`: A target repository to save the built container to. -- `--freeze`: Enable a container freeze. -- `--tower-token`: A Seqera Platform auth token so that Wave can access your private registry credentials. Not required if the `TOWER_ACCESS_TOKEN` environment variable is set. -- `--tower-workspace-id`: A Seqera Platform workspace ID, such as `1234567890`, where credentials may be stored. - -## Example usage - -In the following example, the `alpine` container image is frozen to a private DockerHub image registry. The `--tower-token` argument is not required if the `TOWER_ACCESS_TOKEN` environment variable is defined. - -``` -wave -i alpine --freeze \ - --build-repo docker.io/user/repo --tower-token -``` - -[Nextflow]: https://www.nextflow.io/ diff --git a/docs/cli/build-singularity.mdx b/docs/cli/build-singularity.mdx deleted file mode 100644 index 54a361ecb..000000000 --- a/docs/cli/build-singularity.mdx +++ /dev/null @@ -1,47 +0,0 @@ ---- -title: Build a Singularity container ---- - -The Wave CLI supports building a [Singularity]. A target build repository, specified with the `--build-repo` argument, is required to build a Singularity container. You can build a Singularity container from several sources: - -- A [SingularityCE] def file -- A Docker container image with an optional local context directory -- Conda packages -- Spack packages - -The following limitations apply: - -- The `linux/arm64` platform is not currently supported - -## Related CLI arguments - -The following arguments are used to build a Singularity container: - -- `--build-repo`: A target repository to save the built container to. -- `--freeze`: Enable a container freeze. -- `--singularity` and `-s`: Build a Singularity container. -- `--tower-token`: A Seqera Platform auth token so that Wave can access your private registry credentials. Not required if the `TOWER_ACCESS_TOKEN` environment variable is set. -- `--tower-workspace-id`: A Seqera Platform workspace ID, such as `1234567890`, where credentials may be stored. - -## Example usage - -In the following example, a Docker base imagine is augmented: - -``` -wave -i alpine --layer context-dir/ --build-repo docker.io/user/repo -``` - -In the following example, a SingularityCE def file is specified: - -``` -wave -f hello-world.def --singularity --freeze --build-repo docker.io/user/repo -``` - -In the following example, two Conda packages are specified: - -``` -wave --conda-package bamtools=2.5.2 --conda-package samtools=1.17 --freeze --singularity --build-repo docker.io/user/repo -``` - -[Singularity]: https://docs.sylabs.io/guides/latest/user-guide/introduction.html -[SingularityCE]: https://docs.sylabs.io/guides/latest/user-guide/definition_files.html diff --git a/docs/cli/build-spack.mdx b/docs/cli/build-spack.mdx deleted file mode 100644 index a11acbd9a..000000000 --- a/docs/cli/build-spack.mdx +++ /dev/null @@ -1,29 +0,0 @@ ---- -title: Build a container from Spack packages ---- - -**Note**: Spack support will be removed in future releases. - -The Wave CLI supports building a container from a list of [Spack] packages. - -:::caution -Support for Spack packages is currently experimental. -::: - -## Related CLI arguments - -Spack builds support following arguments: - -- `--spack-file`: A Spack YAML file that specifies packages and their dependencies. -- `--spack-package`: A Spack package to install. Can be specified multiple times. Version expression are supported, such as `curl@7.42.1`. -- `--spack-run-command`: A Docker `RUN` command used when the container is built. Can be specified multiple times. - -## Example usage - -In the following example, a container with the `curl` package is built: - -``` -wave --spack-package curl@7.42.1 -``` - -[Spack]: https://spack.io/ diff --git a/docs/cli/index.mdx b/docs/cli/index.mdx index 562d3846f..6581f628d 100644 --- a/docs/cli/index.mdx +++ b/docs/cli/index.mdx @@ -1,5 +1,5 @@ --- -title: Wave CLI +title: CLI overview --- The Wave CLI is a convenient wrapper around the Wave API. diff --git a/docs/cli/install.mdx b/docs/cli/install.mdx deleted file mode 100644 index ddf9574f1..000000000 --- a/docs/cli/install.mdx +++ /dev/null @@ -1,41 +0,0 @@ ---- -title: Install the Wave CLI ---- - -To install the `wave` CLI for your platform, complete the following steps: - -1. Download the [latest version of the Wave CLI][download] for your platform. - -2. In a new terminal, complete the following steps: - - 1. Move the executable from your downloads folder to a location in your `PATH`, such as `~/bin`. For example: `mv wave-cli-0.8.0-macos-x86_64 ~/bin/wave` - 2. Ensure that the executable bit is set. For example: `chmod u+x ~/bin/wave` - -3. You can also use [Homebrew](https://brew.sh/) in macos and linux, you can install like this: - ```bash - brew install seqeralabs/tap/wave-cli - ``` - -4. Verify that you can build containers with Wave: - - 1. Create a basic `Dockerfile`: - - ``` - cat << EOF > ./Dockerfile - FROM busybox:latest - EOF - ``` - - 2. Use the CLI to build the container: - - ``` - wave -f Dockerfile - ``` - - Example output: - - ``` - wave.seqera.io/wt/xxxxxxxxxxxx/wave/build:xxxxxxxxxxxxxxxx - ``` - -[download]: https://github.com/seqeralabs/wave-cli/releases diff --git a/docs/cli/reference.mdx b/docs/cli/reference.mdx new file mode 100644 index 000000000..fc419a2cf --- /dev/null +++ b/docs/cli/reference.mdx @@ -0,0 +1,271 @@ +--- +title: CLI reference +--- + +## Install the Wave CLI + +To install the `wave` CLI for your platform, complete the following steps: + +1. Download latest version of the Wave CLI for your platform: + + - To install the latest release from GitHub: + + 1. Download the [latest version of the Wave CLI][download] for your platform. + + 1. In a new terminal, complete the following steps: + + 1. Move the executable from your downloads folder to a location in your `PATH`, such as `~/bin`. For example: `mv wave-cli-0.8.0-macos-x86_64 ~/bin/wave` + 1. Ensure that the executable bit is set. For example: `chmod u+x ~/bin/wave` + + - To install the latest version with [Homebrew]: + + ```bash + brew install seqeralabs/tap/wave-cli + ``` + +1. Verify that you can build containers with Wave: + + 1. Create a basic `Dockerfile`: + + ``` + cat << EOF > ./Dockerfile + FROM busybox:latest + EOF + ``` + + 1. Use the CLI to build the container: + + ``` + wave -f Dockerfile + ``` + + Example output: + + ``` + wave.seqera.io/wt/xxxxxxxxxxxx/wave/build:xxxxxxxxxxxxxxxx + ``` + +[download]: https://github.com/seqeralabs/wave-cli/releases +[Homebrew]: https://brew.sh/ + +## Build a container + +With the Wave CLI you can build Docker and Singularity containers from a variety of sources, including a Dockerfile, Singularity def file, file system directory, and Conda packages. + +The following sections describe several common usage cases. To get started by creating an example Nextflow pipeline that uses Wave CLI, see [Get started][start]. + +[start]: ../get-started.mdx#wave-cli + +### Augment a container with a directory + +The Wave CLI supports container augmentation with a specified directory. You can use container augmentation to dynamically add a layer to your container, so you can inject scripts or configuration files as a new layer. + +
+**Augment a container with a directory** + +The following limitations apply: + +- A file must be no larger than 1 MB each. +- A directory must be no larger than 10 MB, inclusive of all files. +- A base image must be specified. + +**Related CLI arguments** + +The following arguments are used for a directory build: + +- `--layer`: A directory that contains layer content. +- `--image` or `-i`: An existing container image. The default image registry is `docker.io`. Specify an image name such as `alpine:latest` or an image URL such as `public.ecr.aws/docker/library/busybox`. + +**Example usage** + +Create a new context directory: + +``` +mkdir -p new-layer/usr/local/bin +printf 'echo Hello world!' > new-layer/usr/local/bin/hello.sh +chmod +x new-layer/usr/local/bin/hello.sh +``` + +Use the CLI to build the image and run the result with Docker: + +``` +docker run $(wave -i alpine --layer new-layer) sh -c hello.sh +``` +
+ +### Build a container from Conda packages + +The Wave CLI supports building a container from a list of [Conda] packages. + +
+**Build a container from Conda packages** + +**Related CLI arguments** + +Conda builds support the following arguments: + +- `--conda-base-image`: A base image for installing Conda packages. The default value is `mambaorg/micromamba:1.5.1`. +- `--conda-channels`: One or more comma separated channels. The default value is ` seqera,bioconda,conda-forge,defaults`. +- `--conda-file`: A [Conda lock file][conda-lock]. Can be a local file or a URL to a remote file. +- `--conda-package`: A Conda package to install. Can be specified multiple times. Expression are supported, such as `bioconda::samtools=1.17` or `samtools>=1.0,<1.17`. +- ` --conda-run-command`: A Docker `RUN` command used when the container is built. Can be specified multiple times. + +**Example usage** + +In the following example, a container with the `samtools` and `bamtools` packages is built: + +``` +wave \ + --conda-package bamtools=2.5.2 \ + --conda-package samtools=1.17 +``` + +[Conda]: https://anaconda.org/anaconda/repo +[conda-lock]: https://github.com/conda/conda-lock +
+ +### Build a container from a Dockerfile + +The Wave CLI supports building a container from a `Dockerfile`. Specifying an optional build context allows the use of `ADD` and `COPY` commands in a Dockerfile. + +:::note +Building a Dockerfile that requires `--build-arg` for build time variables isn't currently supported. +::: + +
+**Build a container from a Dockerfile** + +**Related CLI arguments** + +- `--containerfile` or `-f`: A Dockerfile to build. Build args aren't currently supported. +- `--context`: A directory that contains the context for the build. + +**Example usage** + +In the following example `Dockerfile`, several packages are installed: + +``` +cat << EOF > ./Dockerfile +FROM alpine + +RUN apk update && apk add bash cowsay \ + --update-cache \ + --repository https://alpine.global.ssl.fastly.net/alpine/edge/community \ + --repository https://alpine.global.ssl.fastly.net/alpine/edge/main \ + --repository https://dl-3.alpinelinux.org/alpine/edge/testing +EOF +``` + +Build and run the container based on the Dockerfile in the previous example by running the following command: + +``` +container=$(wave --containerfile ./Dockerfile) +docker run --rm $container cowsay "Hello world" +``` + +In the following example `Dockerfile`, a local context is used: + +``` +cat << EOF > ./Dockerfile +FROM alpine +ADD hello.sh /usr/local/bin/ +EOF +``` + +Create the shell script referenced in the previous example by running the following commands in your terminal: + +``` +mkdir -p build-context/ +printf 'echo Hello world!' > build-context/hello.sh +chmod +x build-context/hello.sh +``` + +Build and run the container based on the Dockerfile in the previous example by running the following command: + +``` +docker run $(wave -f Dockerfile --context build-context) sh -c hello.sh +``` +
+ +### Build a Singularity container + +The Wave CLI supports building a [Singularity]. A target build repository, specified with the `--build-repo` argument, is required to build a Singularity container. You can build a Singularity container from several sources. + +
+**Build a Singularity container** + +- A [SingularityCE] def file +- A Docker container image with an optional local context directory +- Conda packages + +The following limitations apply: + +- The `linux/arm64` platform is not currently supported + +**Related CLI arguments** + +The following arguments are used to build a Singularity container: + +- `--build-repo`: A target repository to save the built container to. +- `--freeze`: Enable a container freeze. +- `--singularity` and `-s`: Build a Singularity container. +- `--tower-token`: A Seqera Platform auth token so that Wave can access your private registry credentials. Not required if the `TOWER_ACCESS_TOKEN` environment variable is set. +- `--tower-workspace-id`: A Seqera Platform workspace ID, such as `1234567890`, where credentials may be stored. + +**Example usage** + +In the following example, a Docker base imagine is augmented: + +``` +wave -i alpine --layer context-dir/ --build-repo docker.io/user/repo +``` + +In the following example, a SingularityCE def file is specified: + +``` +wave -f hello-world.def --singularity --freeze --build-repo docker.io/user/repo +``` + +In the following example, two Conda packages are specified: + +``` +wave --conda-package bamtools=2.5.2 --conda-package samtools=1.17 --freeze --singularity --build-repo docker.io/user/repo +``` + +[Singularity]: https://docs.sylabs.io/guides/latest/user-guide/introduction.html +[SingularityCE]: https://docs.sylabs.io/guides/latest/user-guide/definition_files.html +
+ +### Build a container and freeze to a container registry + +The Wave CLI supports building a container and persisting the container to a container registry, such as DockerHub. You can refer to this frozen container image in a Dockerfile or [Nextflow] pipeline in the same way as any other container. + +
+**Build a container and freeze to a container registry** + +To freeze a container, you must ensure the following conditions are met: + +- You created a Seqera Platform access token. +- You specified the destination container registry credentials in Seqera Platform. +- You specify the Seqera Platform access token via either the `TOWER_ACCESS_TOKEN` environment variable or the `--tower-token` Wave command line option. + +**Related CLI arguments** + +The following arguments are used to freeze a container build: + +- `--build-repo`: A target repository to save the built container to. +- `--freeze`: Enable a container freeze. +- `--tower-token`: A Seqera Platform auth token so that Wave can access your private registry credentials. Not required if the `TOWER_ACCESS_TOKEN` environment variable is set. +- `--tower-workspace-id`: A Seqera Platform workspace ID, such as `1234567890`, where credentials may be stored. + +**Example usage** + +In the following example, the `alpine` container image is frozen to a private DockerHub image registry. The `--tower-token` argument is not required if the `TOWER_ACCESS_TOKEN` environment variable is defined. + +``` +wave -i alpine --freeze \ + --build-repo docker.io/user/repo --tower-token +``` + +[Nextflow]: https://www.nextflow.io/ +
diff --git a/docs/get-started.mdx b/docs/get-started.mdx new file mode 100644 index 000000000..e3ef0e2f0 --- /dev/null +++ b/docs/get-started.mdx @@ -0,0 +1,377 @@ +--- +title: Get started +--- + +Wave is versatile and you can leverage it in your Nextflow pipelines in several ways. The following guides describe how to quickly get started with [Seqera Containers and Nextflow](#nextflow-and-seqera-containers), [Nextflow and Wave integration](#nextflow), and the [Wave CLI](#wave-cli). + +## Nextflow and Seqera Containers + +You can provision containers that include [Conda packages][conda] through [Seqera Containers][sc] and use them directly in your Nextflow pipelines. + +**Prerequisites** + +- You have either [Docker Desktop] or [Podman] installed locally +- You have [Nextflow] 23.10.x or newer installed locally + +In this guide, you'll request a containerized Conda package from Seqera Containers. + +### Request a Conda package as a Seqera Container + +1. Open [Seqera Containers][sc] in a browser. +1. In the search box, enter `faker`. +1. In the search results, click **Add** in the `conda-forge::faker` result, and then click **Get Container** to initiate the container build. +1. From the **Fetching container** modal, copy the the durable container image URI that Seqera Containers provides. +1. Optional: Click **View build details** to watch Seqera Containers build the requested container in realtime. + +### Create a Nextflow pipeline that uses the container + +Nextflow can use the container that Seqera Containers built in the previous section. Use the container URI from Seqera Containers in the `container` directive. + +1. In a terminal window, create a new directory for the Nextflow pipeline. +1. Create a `nextflow.config` file with the following contents: + + ```groovy + docker { + enabled = true + } + ``` + +1. Create a `main.nf` file with the following contents: + + ```groovy + process FAKER { + container '' + debug true + + """ + faker address + """ + } + + workflow { + FAKER() + } + ``` + + Substitute `` for the container URI that you received from Seqera Containers in the previous section. + +### Run the Nextflow pipeline + +To confirm that the `faker` command is available from your pipeline, run the following command: + +``` +nextflow run main.nf +``` + +The output from a successful execution is displayed in the following example: + +``` +Launching `main.nf` [jolly_edison] DSL2 - revision: 5c414bd927 + +executor > local (1) +[86/0d56e8] faker | 1 of 1 ✔ +234 Nicholas Circle +Masonport, MS 98018 +``` +## Nextflow + +You can provision containers with Wave directly from your Nextflow pipelines. + +**Prerequisites** + +- You have an account with a container registry, such as DockerHub, and an access token that provides write access to your container repository +- You have a [Seqera Cloud](https://cloud.seqera.io/login) or Platform account, to store your container registry credentials for Wave to use +- You have either [Docker Desktop] or [Podman] installed locally +- You have [Nextflow] 23.10.x or newer installed locally + +In this guide, you'll build a container from a Nextflow module and freeze that package in your private container repository. + +### Create your Seqera access token + +1. Log in to Seqera. +1. From your personal workspace: Go to the user menu and select **Settings > Your tokens**. +1. Select **Add token**. +1. Enter a unique name for your token, then select **Add**. +1. Copy and store your token securely. + + :::caution + The access token is displayed only once. Save the token value before closing the **Personal Access Token** window. + ::: + +1. In a terminal window, assign your access token to the `TOWER_ACCESS_TOKEN` environment variable: + + ``` + export TOWER_ACCESS_TOKEN= + ``` + +### Add your container registry credentials to Seqera + +When freezing a container to the build repository that you specify, Wave uses Seqera to obtain your registry access credentials. If you use Docker as your container registry, completing the following steps: + +To create your access token in Docker Hub: + +1. Log in to [Docker Hub](https://hub.docker.com/). +2. Select your username in the top right corner and select **Account Settings**. +3. Select **Security -> New Access Token**. +4. Enter a token description and select **Read-only** from the Access permissions drop-down menu, then select **Generate**. +5. Copy and save the generated access token (this is only displayed once). + +To add your credentials to Seqera: + +1. Add your credentials to your organization or personal workspace: + - From an organization workspace: Go to **Credentials > Add Credentials**. + - From your personal workspace: From the user menu, go to **Your credentials > Add credentials**. + +2. Complete the following fields: + + - **Name**: Specify a unique name for the credentials using alphanumeric characters, dashes, or underscores. For example, `my-registry-creds`. + - **Provider**: Select **Container registry**. + - **User name**: Specify your Docker username. For example, `user1`. + - **Password**: Specify your personal access token (PAT). For example, `1fcd02dc-...215bc3f3`. + - **Registry server**: Specify the container registry hostname, excluding the protocol. For example, `docker.io`. + +3. After you've completed all the form fields, select **Add**. The new credential is now listed under the **Credentials** tab. + +Seqera supports other container registries, such as GitHub and Quay.io. + +### Create a Nextflow pipeline that uses Wave + +Nextflow can use Wave to seamlessly build a container directly from a Dockerfile in your pipeline. + +1. In a terminal window, create a new directory for the Nextflow pipeline. + +1. Create a `nextflow.config` file with the following contents: + + ```groovy + docker { + enabled = true + } + + wave { + build.repository = '' + wave.freeze = true + } + + tower { + accessToken = "$TOWER_ACCESS_TOKEN" + } + ``` + + Substitute `` for your private container repository hosted by the container registry that you input credentials for into Seqera. + +1. Create a `wave.nf` file with the following contents: + + ```groovy + include { HELLO } from './modules/gamma' + + workflow { + HELLO() + } + ``` + +1. Create a directory for the module: + ``` + mkdir -p modules/gamma + ``` + +1. Create the `modules/gamma/main.nf` file for the module: + + ``` + process HELLO { + debug true + + """ + cowsay Hello! + """ + } + ``` + +1. Create the `modules/gamma/Dockerfile` file for the module: + + ``` + FROM alpine + + RUN apk update && apk add bash cowsay \ + --update-cache \ + --repository https://alpine.global.ssl.fastly.net/alpine/edge/community \ + --repository https://alpine.global.ssl.fastly.net/alpine/edge/main \ + --repository https://dl-3.alpinelinux.org/alpine/edge/testing + + RUN echo hello + ``` + +### Run the Nextflow pipeline + +To run the pipeline, and initiate the Wave container build, enter the following command: + +``` +nextflow run wave.nf -with-wave +``` + +The output from a successful execution is displayed in the following example: + +``` +Launching `wave.nf` [naughty_wiles] DSL2 - revision: 3756d705d9 + +executor > local (1) +[c1/6d7d9d] HELLO | 1 of 1 ✔ + ________ +< Hello! > + -------- + \ ^__^ + \ (oo)\_______ + (__)\ )\/\ + ||----w | + || || +``` + +## Wave CLI + +With the Wave CLI, you can provision containers for later use in your Nextflow pipelines. + +**Prerequisites** + +For this guide, you must satisfy the following prerequisites: + +- You have an account with a container registry, such as DockerHub, and an access token that provides write access to your container repository +- You have a [Seqera Cloud](https://cloud.seqera.io/login) or Platform account, to store your container registry credentials for Wave to use +- You have either [Docker Desktop] or [Podman] installed locally +- You have [Nextflow] 23.10.x or newer installed locally +- You have the [Wave CLI][wave-cli] installed locally + +In this guide, you'll build a container from a [Conda package][conda] and freeze that package in your private container repository. By freezing the container, you ensure that you can always use it in your Nextflow pipelines. + +### Create your Seqera access token + +1. Log in to Seqera. +1. From your personal workspace: Go to the user menu and select **Settings > Your tokens**. +1. Select **Add token**. +1. Enter a unique name for your token, then select **Add**. +1. Copy and store your token securely. + + :::caution + The access token is displayed only once. Save the token value before closing the **Personal Access Token** window. + ::: + +1. In a terminal window, assign your access token to the `TOWER_ACCESS_TOKEN` environment variable: + + ``` + export TOWER_ACCESS_TOKEN= + ``` + +### Add your container registry credentials to Seqera + +When freezing a container to the build repository that you specify, Wave uses Seqera to obtain your registry access credentials. If you use Docker as your container registry, completing the following steps: + +To create your access token in Docker Hub: + +1. Log in to [Docker Hub](https://hub.docker.com/). +2. Select your username in the top right corner and select **Account Settings**. +3. Select **Security -> New Access Token**. +4. Enter a token description and select **Read-only** from the Access permissions drop-down menu, then select **Generate**. +5. Copy and save the generated access token (this is only displayed once). + +To add your credentials to Seqera: + +1. Add your credentials to your organization or personal workspace: + - From an organization workspace: Go to **Credentials > Add Credentials**. + - From your personal workspace: From the user menu, go to **Your credentials > Add credentials**. + +2. Complete the following fields: + + - **Name**: Specify a unique name for the credentials using alphanumeric characters, dashes, or underscores. For example, `my-registry-creds`. + - **Provider**: Select **Container registry**. + - **User name**: Specify your Docker username. For example, `user1`. + - **Password**: Specify your personal access token (PAT). For example, `1fcd02dc-...215bc3f3`. + - **Registry server**: Specify the container registry hostname, excluding the protocol. For example, `docker.io`. + +3. After you've completed all the form fields, select **Add**. The new credential is now listed under the **Credentials** tab. + +Seqera supports other container registries, such as GitHub and Quay.io. + +### Create and freeze a container + +Wave lets you build a container from any conda package or set of conda packages that you specify. In this section, you'll use Wave to build a container that includes the `faker` conda package. + +In the same terminal window from the previous section, run the Wave CLI to build and freeze a container with the `faker` conda package to the repository that you specify. Specify the URI for the repository for which you added an access token to Seqera, so that Wave can push the built container. + +``` +wave --conda-package 'faker' --freeze --build-repo --await +``` + +Example output: + +``` +docker.io/example-user/repo:faker--2aa7a4d826a76301 +``` + +After the container build completes, Seqera emails you a build status report, including the build logs for the container. + +### Create a Nextflow pipeline that uses the container + +Nextflow can use the container that Wave froze to the build repository that you provided in the previous section. The Wave CLI outputs the URI for the container image and the image tag. Use these values in the `container` directive. + +1. In a terminal window, create a new directory for the Nextflow pipeline. +1. Create a `nextflow.config` file with the following contents: + + ```groovy + docker { + enabled = true + } + + tower { + accessToken = "$TOWER_ACCESS_TOKEN" + } + ``` + +1. Create a `main.nf` file with the following contents: + + ```groovy + process FAKER { + container 'docker.io/example-user/repo:faker--2aa7a4d826a76301' + debug true + + """ + faker address + """ + } + + workflow { + FAKER() + } + ``` + +### Run the Nextflow pipeline + +To confirm that the `faker` command is available from your pipeline, run the following command: + +``` +nextflow run main.nf +``` + +The output from a successful execution is displayed in the following example: + +``` +Launching `./main.nf` [happy_leavitt] DSL2 - revision: 03b4e42ba3 + +executor > local (1) +[1d/120069] FAKER | 1 of 1 ✔ +1287 Eric Grove +Reneechester, AK 75644 +``` + +## Next steps + +- Explore additional [Wave CLI][wave-build] use cases +- Review the available [Nextflow configuration][nf-config] options for Wave + + +[sc]: https://seqera.io/containers/ +[conda]: https://docs.conda.io/en/latest/ +[Nextflow]: https://www.nextflow.io/docs/latest/install.html +[Podman]: https://podman.io/docs/installation +[Docker Desktop]: https://www.docker.com/products/docker-desktop/ + +[wave-build]: ./cli/build.mdx +[nf-config]: ./nextflow.mdx#configuration-options +[wave-cli]: ./cli/index.mdx diff --git a/docs/guide.mdx b/docs/guide.mdx deleted file mode 100644 index 32d5dc9b2..000000000 --- a/docs/guide.mdx +++ /dev/null @@ -1,161 +0,0 @@ ---- -title: User guide ---- - -## Getting started - -Wave containers can be used with any container runtime supporting the [Docker Registry API v2](https://docs.docker.com/registry/spec/api/) and it's smoothly integrated with [Nextflow](https://www.nextflow.io/) and [Seqera Platform](https://cloud.tower.nf/). - -This feature requires Nextflow `22.10.0` or later. - -### Nextflow installation - -If you've already installed Nextflow, update to the latest version using this command:: - -```bash -nextflow -self-update -``` - -If you don't have Nextflow already installed, install it with the command below: - -```bash -curl get.nextflow.io | bash -``` - -### Wave configuration - -Wave can be used in any Nextflow pipeline by adding the following snippet to your `nextflow.config` file: - -```groovy -wave { - enabled = true -} - -tower { - accessToken = '' -} -``` - -:::tip -The use of the Seqera access token is not mandatory, however, it's required to enable access to private repositories and it allows higher service rate limits compared to anonymous users. -::: - -## API limits - -The Wave service implements API rate limits for API calls. Authenticated users have higher rate limits than anonymous users. - -If an access token is provided, the following rate limits apply: - -- 100 container builds per hour -- 1,000 container pulls per minute - -If an access token isn't provided, the following rate limits apply: - -- 25 container builds per day -- 250 container pulls per hour - -## Known limitation - -### Use of sha256 digest in the image name - -The Wave does not support the use of sha256 digest in the image name, e.g. `ubuntu@sha256:3235...ce8f`, when using -the augmentation process to extend container images. - -In order to reference a container via sha256 digest in the image name with Wave you will need to *freeze* image mode -that will force the creation of a new container image using the container you have specified as base image. - -In your pipeline configuration, ensure that you specify the following settings: - -```groovy -wave.enabled = true -wave.freeze = true -wave.strategy = ['dockerfile'] -wave.build.repository = 'docker.io//' -``` - -## Tutorials - -### Authenticate private repositories - -Wave allows the use of private repositories in your Nextflow pipelines. The repository access keys must be provided via [Seqera credentials](https://help.tower.nf/23.1/credentials/overview/). - -When you have created the credentials, you only need to specify your [Seqera access token](https://help.tower.nf/23.1/api/overview/#authentication) in your pipeline configuration file. If the credentials were created in a Seqera organization workspace, specify the workspace ID as well in the config file as shown below: - -```groovy -tower { - accessToken = '' - workspaceId = '' -} -``` - -That's it. When launching the pipeline execution, Wave will allow Nextflow to access the private container repositories defined in your pipeline configuration, using the credentials stored in the Seqera Platform credentials manager. - -### Build Nextflow module containers - -Wave can build and provision container images on-demand for your Nextflow pipelines. - -To enable this feature, add the `Dockerfile` of the container to be built in the [module directory](https://www.nextflow.io/docs/latest/dsl2.html#module-binaries) where the pipeline process is defined. When Wave is enabled, it automatically uses the Dockerfile to build the required container, upload to the registry, and uses the container to carry out the tasks defined in the module. - -:::tip -Make sure the process does not declare a `container` directive, otherwise it will take precedence over the Dockerfile definition. -::: - -If a process uses a `container` directive and you still want to build the container using the `Dockerfile` provided in the module directory, add the following setting to the pipeline config file: - -```groovy -wave.strategy = ['dockerfile','container'] -``` - -The above line instructs Wave to give the module `Dockerfile` priority over process `container` directives. - -:::caution -Wave currently does not support `ADD`, `COPY` and other Dockerfile commands that access files in the host file system. -::: - -### Build Conda based containers - -Wave allows the provisioning of containers based on the [`conda` directive](https://www.nextflow.io/docs/latest/process.html#conda) used by the processes in your Nextflow pipeline. This is a quick alternative to building Conda packages in the local computer. Moreover, this enables the use of Conda packages in your pipeline when deploying it in cloud-native platforms such as AWS Batch and Kubernetes, which do not allow the (easy) use of the Conda package manager. - -Having Wave enabled in your pipeline, there's nothing else to do other than define the `conda` requirements in the pipeline processes provided the same process does not also specify a `container` directive or a Dockerfile. - -In the latter case, add the following setting to your pipeline configuration: - -```groovy -wave.strategy = ['conda'] -``` - -The above setting instructs Wave to only use the `conda` directive to provision the pipeline containers, ignoring the use of the `container` directive and any `Dockerfile`(s). - -### Store container images into a private repository - -Containers built by Wave are uploaded to the Wave default repository hosted on AWS ECR with name `195996028523.dkr.ecr.eu-west-1.amazonaws.com/wave/build`. The images in this repository are automatically deleted 1 week from the date of their push. - -If you want to store Wave containers in your own container repository use the following settings in the Nextflow configuration file: - -```groovy -wave.build.repository = 'example.com/your/build-repo' -wave.build.cacheRepository = 'example.com/your/cache-repo' -``` - -The first repository is used to store the built container images. The second one is used to store the individual image layers for caching purposes. - -The repository access keys need to be specified using the Seqera Platform credentials manager as specified in the [Authenticate private repositories](#Authenticate private repositories) section. - -## Advanced settings - -The following configuration options are available: - -| Method | Description | -| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `wave.enabled` | Enable/disable the execution of Wave containers | -| `wave.endpoint` | The Wave service endpoint (default: `https://wave.seqera.io`) | -| `wave.build.repository` | The container repository where image built by Wave needs to be uploaded (note: the corresponding credentials need to be provided in your Seqera Platform account). | -| `wave.build.cacheRepository` | The container repository used to cache image layers build by the Wave service (note: the corresponding credentials need to be provided in your Seqera Platform account). | -| `wave.conda.mambaImage` | The Mamba container image is used to build Conda based container. This is expected to be [micromamba-docker](https://github.com/mamba-org/micromamba-docker) image. | -| `wave.conda.commands` | One or more commands to be added to the Dockerfile used by build a Conda based image. | -| `wave.strategy` | The strategy to be used when resolving ambiguous Wave container requirement (default: `'container,dockerfile,conda'`) | -| `wave.freeze` | When `freeze` mode is enabled containers provisioned by Wave are stored permanently in the repository specified via the setting `wave.build.repository`. | - -## More examples - -Check out the [Wave showcase repository](https://github.com/seqeralabs/wave-showcase) for more examples how to use Wave containers. diff --git a/docs/index.mdx b/docs/index.mdx index 8695b60f1..a9dc1017f 100644 --- a/docs/index.mdx +++ b/docs/index.mdx @@ -1,52 +1,77 @@ --- -title: Wave containers +title: Wave overview --- -Containers are an essential part of data analysis in the cloud. Building and delivering optimized, context-aware container images slows down development. +Containers are an essential part of modern data analysis pipelines in bioinformatics. They encapsulate applications and dependencies in portable, self-contained packages that can be easily distributed across diverse computing environments. Containers are also key to enabling predictable and reproducible scientific results. However, workflows can comprise dozens of distinct container images. Pipeline developers must manage and maintain these container images and ensure that their functionality precisely aligns with the requirements of every pipeline task, creating unnecessary friction in the maintenance and deployment of data pipelines. -Wave is a container provisioning service designed for use with data analysis applications such as Nextflow. +Wave solves this problem by provisioning containers on-demand during the pipeline execution. This allows the delivery of container images that are defined precisely depending on the requirements of each pipeline task in terms of dependencies and platform architecture. This process is completely transparent and fully automated, removing all the plumbing and friction commonly needed to create, upload, and maintain dozens of container images that might be required by a pipeline execution. -It allows for the on-demand assembly, augmentation, and deployment of containerized images based on task requirements. +To get started with Wave: -The Wave container service itself is not a container registry. All containers builds are stored in a Seqera-hosted image registry for a limited time or frozen to a user-specified container registry. +1. See the [Get started][started] guide. +1. Learn about [Nextflow integration][nf]. +1. Learn about the [Wave CLI][cli]. :::note Wave is also available as hosted service on [Seqera Platform](https://cloud.seqera.io/). For Seqera Enterprise customers, a licensed self-hosted Wave solution is also available. Contact us [contact us](https://seqera.io/contact-us/) for more information. ::: -## Features +[started]: ./get-started.mdx +[nf]: ./nextflow.mdx +[cli]: ./cli/index.mdx -### Private container registries +## Wave features -Container registry authentication is the new norm. Yet when it comes to authenticating against cloud-specific container registries, the process is hardly hassle free. -Wave integrates with Seqera Platform credentials management enabling seamless access and publishing to private registries. +### Container registries -### Augment existing containers +#### Private container registries + +Wave integrates with [Seqera Platform credentials management][private] enabling seamless access and publishing to private registries. + +[private]: ./nextflow.mdx#access-private-container-repositories + +#### Seqera Containers - The community container registry -Regulatory and security requirements sometimes dictate specific container images, but additional context is often needed. -Wave enables any existing container to be extended without rebuilding it. Developers can add user-provided content such as custom scripts and logging agents, providing greater flexibility in the container’s configuration. +[Seqera Containers] is a free to use service operated for the community by Seqera. -Wave offers a flexible approach to container image management. It allows you to dynamically add custom layers to existing docker images, creating new images tailored to your specific needs. +It uses Wave to build images from Conda / PyPI packages on demand, either through the [web interface](https://seqera.io/containers/) or using the [Wave CLI](./cli/index.mdx) / [Nextflow integration](./nextflow.mdx). -#### An example of Wave augmentation +These images are cached and hosted permanently, being served through a [Docker Distribution][docker] registry and hosted on AWS infrastructure. Images are cached and served via Cloudflare CDN. -Imagine you have a base Ubuntu image in a container registry. Wave acts as a proxy between your Docker client and the registry. When you request an augmented image, Wave intercepts the process. +Images are publicly accessible to anyone for free and will be stored for at least 5 years. They can be pulled using any infrastructure (local, HPC, cloud) as Docker or native Singularity images. Images can be built for both `linux/aarch64` and `linux/arm64` architectures. -1. Base image layers download: The Docker client downloads the standard Ubuntu layers from the registry. -2. Custom layer injection: Wave injects your custom layer, denoted by "ω", which could represent application code, libraries, configurations etc. -3. New image creation: Wave combines the downloaded Ubuntu layers with your custom layer, effectively creating a new image on the fly. +:::note +Seqera Containers does not work with custom container files, augmentation, or authorization. It provides only Conda based containers. +::: -![](_images/wave_container_augmentation.png) +[docker]: https://github.com/distribution/distribution +[Seqera Containers]: https://seqera.io/containers/ + +### Augment existing containers -#### Benefits of Wave augmentation +Wave offers a flexible approach to container image management. It allows you to [dynamically add custom layers][augment] to existing docker images, creating new images tailored to your specific needs. +Any existing container can be extended without rebuilding it. You can add user-provided content such as custom scripts and logging agents, providing greater flexibility in the container’s configuration. -1. Streamlined workflows: Wave simplifies your workflow by eliminating the need to manually build and manage custom images. -2. Flexibility: You can easily modify the custom layer for different use cases, allowing for greater adaptability. +[augment]: ./provisioning.mdx#container-augmentation ### Conda-based containers -Package management systems such as Conda and Bioconda simplify the installation of scientific software. However, there’s considerable friction when it comes to using those tools to deploy pipelines in cloud environments. -Wave enables dynamic provisioning of container images from any Conda or Bioconda recipe. Just declare the Conda packages in your Nextflow pipeline and Wave will assemble the required container. +Package management systems such as Conda and Bioconda simplify the installation of scientific software. +Wave enables dynamic provisioning of container images from any Conda or Bioconda recipe. Just [declare the Conda packages][conda] in your Nextflow pipeline and Wave will assemble the required container. + +[conda]: ./nextflow.mdx#build-conda-based-containers + +### Singularity containers + +Singularity and Apptainer use a proprietary format called _Singularity Image Format_ (SIF). The Wave service can [provision containers based on the Singularity image format][singularity] either by using a `Singularityfile` file or Conda packages. The resulting Singularity image file is stored as an ORAS artifact in an OCI-compliant container registry of your choice or the Wave Community registry. + +The advantage of this approach is that Singularity and Apptainer engines can pull and execute those container images natively without requiring extra conversion steps, as needed when using Docker images with those two engines. + +:::note +Considering the Singularity image format's peculiarities, Wave's freeze mode is mandatory when provisioning Singularity images. +::: + +[singularity]: ./nextflow.mdx#build-singularity-containers ### Deploying containers across multi-clouds @@ -60,7 +85,7 @@ Builds for OCI-compliant container images are automatically scanned for known se ### Optimize workloads for specific architectures -Modern data pipelines can be deployed across different data centers having different hardware architectures. e.g., amd64, arm64, and others. This requires curating different collections of containers for each architecture. +Modern data pipelines can be deployed across different data centers having different hardware architectures such as amd64, arm64, and others. This requires curating different collections of containers for each architecture. Wave allows for the on-demand provisioning of containers, depending on the target execution platform (in development). ### Near caching diff --git a/docs/metrics.mdx b/docs/metrics.mdx index cd504f33a..1d3104818 100644 --- a/docs/metrics.mdx +++ b/docs/metrics.mdx @@ -1,5 +1,5 @@ --- -title: Wave usage metrics +title: Usage metrics --- Wave uses Redis to store its usage metrics for a specific date and/or a specific organization. @@ -50,7 +50,7 @@ Then, if the pulled container uses fusion, it increments the values of following ## How keys are created - When a request is made to wave, first it increments the key with current date. e.g. `builds/d/2024-04-23`. -- Keys with organisation are only incremented if the user is authenticated means there is Seqera platform token in the request. +- Keys with organization are only incremented if the user is authenticated means there is Seqera platform token in the request. - Wave extract the domain from the user email id (For example: `test_metrics@seqera.io`), which it gets from Seqera platform using the access token. -- In this case, The organisation value will be `seqera.io`. -- Then it increments the key with organisation. For example: `builds/o/seqera.io/d/2024-04-23` and `builds/o/seqera.io`. +- In this case, The organization value will be `seqera.io`. +- Then it increments the key with organization. For example: `builds/o/seqera.io/d/2024-04-23` and `builds/o/seqera.io`. diff --git a/docs/nextflow.mdx b/docs/nextflow.mdx new file mode 100644 index 000000000..ec9d246e8 --- /dev/null +++ b/docs/nextflow.mdx @@ -0,0 +1,215 @@ +--- +title: Nextflow integration +--- + +You can use Wave directly from your Nextflow pipelines, with full support for private repositories, container freezing, and conda packages. + +This feature requires Nextflow `22.10.0` or later. + +You can use Wave can in any Nextflow pipeline by adding the following snippet to your `nextflow.config` file: + +```groovy +wave { + enabled = true +} + +tower { + accessToken = '' +} +``` + +The use of the Seqera access token is not mandatory. However, it grants the following additional capabilities: + +- Access to private repositories +- More API requests than is permitted for anonymous users + +For the entire list of configuration options, see [Configuration options](#configuration-options). + +## Use Wave with Nextflow + +The following sections describe several common usage cases. To get started by creating an example pipeline that uses Wave, see [Get started][start]. + +[start]: ./get-started.mdx#nextflow + +### Access private container repositories + +Wave allows the use of private repositories in your Nextflow pipelines. The repository access keys must be provided in the form of [Seqera Platform credentials][credentials]. + +
+**Access private container repositories** + +After creating the credentials, specify your [personal access token][pat] in your pipeline configuration file. If the credentials were created in a Seqera Platform organization workspace, specify the workspace ID as well in the config file as shown below: + +```groovy +tower { + accessToken = '' + workspaceId = '' +} +``` + +Containers built by Wave are uploaded to the Wave default repository hosted on AWS ECR with name `195996028523.dkr.ecr.eu-west-1.amazonaws.com/wave/build`. The images in this repository are automatically deleted 1 week from the date of their push. + +If you want to store Wave containers in your own container repository use the following settings in the Nextflow configuration file: + +```groovy +wave.build.repository = 'example.com/your/build-repo' +wave.build.cacheRepository = 'example.com/your/cache-repo' +``` + +The first repository is used to store the built container images. The second one is used to store the individual image layers for caching purposes. + +When launching the pipeline execution, Wave allows Nextflow to access the private container repositories defined in your pipeline configuration, by using the credentials stored in the Seqera Platform credentials manager. + + +[credentials]: /platform_versioned_docs/version-23.4.0/credentials/overview +[pat]: /platform_versioned_docs/version-23.4.0/api/overview#authentication +
+ +### Build Nextflow modules containers + +Wave can build and provision container images on-demand for your Nextflow pipelines. + +
+**Build Nextflow modules containers** + +To enable this feature, add the Dockerfile of the container to be built in the [module directory][module-directory] where the pipeline process is defined. When Wave is enabled, it automatically uses the Dockerfile to build the required container, upload to the registry, and it uses the container to execute the script defined in the process. + +Make sure the process does not declare a `container` directive, otherwise it will take precedence over the Dockerfile definition. + +If a process uses a `container` directive and you still want to build the container using the Dockerfile provided in the module directory, add the following setting to the pipeline config file: + +```groovy +wave.strategy = ['dockerfile','container'] +``` + +This setting instructs Wave to prioritize the module Dockerfile over process `container` directives. + +:::warning +When building containers, Wave currently does not support `ADD`, `COPY`, or any other Dockerfile commands that access files in the host file system. +::: + +[module-directory]: https://www.nextflow.io/docs/latest/module.html#module-directory +
+ +### Build Conda-based containers + +Wave allows the provisioning of containers based on the `process-conda` [directive][process-conda] used by the processes in your pipeline. This is a quick alternative to building Conda packages in the local computer. Moreover, this enables the use of Conda packages in your pipeline when deploying in cloud-native platforms such as AWS Batch and Kubernetes, which do not allow the convenient use of the Conda package manager. + +
+**Build Conda-based containers** + +With Wave enabled in your pipeline, simply define the `conda` requirements in the pipeline processes, provided the same process does not also specify a `container` directive or a Dockerfile. + +In the latter case, add the following setting to your pipeline configuration: + +```groovy +wave.strategy = ['conda'] +``` + +The above setting instructs Wave to use the `conda` directive to provision the pipeline containers and ignore the `container` directive and any Dockerfile(s). + +For versions of Nextflow 23.10.x or newer, when a container is provisioned, the `conda-forge::procps-ng` package is included automatically. This package includes the `ps` command. + +Some configuration options in the `conda` scope are used when Wave is used to build Conda-based containers. +For example, the Conda channels and their priority can be set with `conda.channels`: + +```groovy +wave.strategy = ['conda'] +conda.channels = 'seqera,conda-forge,bioconda,defaults' +``` + +[process-conda]: https://www.nextflow.io/docs/latest/process.html#conda +
+ +### Build Singularity containers + +Nextflow can build Singularity native images on-demand using `Singularityfile`, +Conda packages. The Singularity images are automatically uploaded in a container registry OCI compliant +of your choice and stored as a [ORAS artifact](https://oras.land/). + +:::note +Available as of Nextflow version 23.09.0-edge. +::: + +
+**Build Singularity containers** + +:::note +This feature requires a version of Singularity (or Apptainer) that supports pulling images using the `oras:` pseudo-protocol. +::: + +For example to enable the provisioning of Singularity images in your pipeline use the following configuration snippet: + +```groovy +singularity.enabled = true +wave.enabled = true +wave.freeze = true +wave.strategy = ['conda'] +wave.build.repository = 'docker.io/user/repo' +``` + +In the above configuration replace `docker.io/user/repo` with a repository of your choice where Singularity image files +should be uploaded. + +When using a private repository, the repository access keys must be provided via the Seqera Platform credentials manager. For more information, see [Authenticate private repositories][private]. + +Moreover the access to the repository must be granted in the compute nodes by using the command `singularity remote login `. +Please see Singularity documentation for further details. + +In order to build Singularity native images, both `singularity.ociAutoPull` and `singularity.ociMode` must be disabled in the configuration. For more information, see the Nextflow [configuration][config] documentation. + +[private]: https://docs.seqera.io/platform/24.1/credentials/overview +[config]: https://www.nextflow.io/docs/latest/config.html#config-singularity +
+ +### Use Wave with Fusion + +Wave containers allows you to run your containerized workflow with the Fusion file system. + +
+**Use Wave with Fusion** + +This enables the use of an object storage bucket such as AWS S3 or Google Cloud Storage as your pipeline work directory, simplifying and speeding up many operations on local, AWS Batch, Google Batch or Kubernetes executions. + +For more information, refer to the following documentation pages: + +- [Fusion documentation][fusion] +- [Nextflow Fusion integration documentation][nextflow-fusion] + +[fusion]: https://docs.seqera.io/fusion +[nextflow-fusion]: https://www.nextflow.io/docs/latest/fusion.html +
+ +## Configuration options + +The following configuration options are available: + +| Method | Description | +| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `wave.enabled` | Enable/disable the execution of Wave containers | +| `wave.endpoint` | The Wave service endpoint (default: `https://wave.seqera.io`) | +| `wave.build.repository` | The container repository where image built by Wave needs to be uploaded (note: the corresponding credentials need to be provided in your Seqera Platform account). | +| `wave.build.cacheRepository` | The container repository used to cache image layers build by the Wave service (note: the corresponding credentials need to be provided in your Seqera Platform account). | +| `wave.conda.mambaImage` | The Mamba container image is used to build Conda based container. This is expected to be [micromamba-docker](https://github.com/mamba-org/micromamba-docker) image. | +| `wave.conda.commands` | One or more commands to be added to the Dockerfile used by build a Conda based image. | +| `wave.strategy` | The strategy to be used when resolving ambiguous Wave container requirement (default: `'container,dockerfile,conda'`) | +| `wave.freeze` | When `freeze` mode is enabled containers provisioned by Wave are stored permanently in the repository specified via the setting `wave.build.repository`. | + +## Limitations + +### Use of sha256 digest in the image name + +The Wave does not support the use of sha256 digest in the image name, e.g. `ubuntu@sha256:3235...ce8f`, when using +the augmentation process to extend container images. + +In order to reference a container via sha256 digest in the image name with Wave you will need to *freeze* image mode +that will force the creation of a new container image using the container you have specified as base image. + +In your pipeline configuration, ensure that you specify the following settings: + +```groovy +wave.enabled = true +wave.freeze = true +wave.strategy = ['dockerfile'] +wave.build.repository = 'docker.io//' +``` diff --git a/docs/provisioning.mdx b/docs/provisioning.mdx new file mode 100644 index 000000000..caa975874 --- /dev/null +++ b/docs/provisioning.mdx @@ -0,0 +1,88 @@ +--- +title: How container provisioning works +--- + +In the container lifecycle, images are generally created (*built*) and uploaded (*pushed*) to a container registry, and then these images are downloaded (*pulled*) for the execution of a specific pipeline. + +The Wave container provisioning process streamlines the container lifecycle by delivering container images on-demand during the pipeline execution and making sure each container includes the dependencies required by the requesting actor, such as a pipeline task or a user. + +Wave provides the following container provisioning capabilities: + +- Container augmentation +- Container freezing + +## Container augmentation + +The container augmentation provisioning mode allows _extending_ the content of a container image without rebuilding it. Instead, this mechanism modifies a container image during the pull phase made by a Docker-compatible client. Augmented containers are ephemeral: they are not stored in a container repository, and they can only be accessed for a short period of time. The extended content added by Wave is served from a CDN. The augmentation process does not perform any _build_ operation behind the scenes. + +This approach supports use cases such as the following: + +- Authenticate access to your private repositories with Seqera Platform credentials +- Extend existing containers by adding infrastructure and pipeline dependencies on the fly, without rebuilding and maintaining additional container images + +Container augmentation works as follows: + +1. The client, either Nextflow or Wave CLI, submits a container request specifying: + 1. The Seqera Platform user identity + 1. The container image to be augmented + 1. The container extension configuration, which can be either a custom payload, one or more extension layers, or container images. +1. The Wave service validates the request and authorizes the user submitting a request to the Platform service. +1. The Wave service responds with an ephemeral container image name e.g. `wave.seqera.io/wt//library/alpine:latest`. The `` is uniquely assigned and is used to identify and authorize the following container request. +1. The Docker client uses the returned image name to pull the container binary content of the upstream image directly from the target registry. +1. The content added by Wave as one or more layer extensions is shipped by the Wave service. + +Notable parts of this workflow include: + +- Wave acts as a proxy between the Docker client and the target registry that hosts the container image. +- Wave modifies, if needed, the container manifest to add the new content as specified by the request, but it does not (and cannot) alter the container layer blob files that have a unique checksum, which is preserved. +- Image blobs are downloaded directly from the target registry, not from Wave. + +## Container freezes + +The container _freeze mode_ allows the provisioning of non-ephemeral containers that are stored permanently in a container registry of your choice. When using the freeze mode, the Wave service transparently carries out a regular container build. + +This approach supports use cases such as the following: + +- Create container images on-demand from Conda packages +- Deliver multi-architecture (AMD64 and ARM64) and multi-format (Docker and Singularity) container collections +- Deliver container images in the same region where compute is performed + +Wave freeze mode works as follows: + +1. The client, either Nextflow or the Wave CLI, submits a container request specifying: + 1. The Seqera Platform user identity + 1. The container image to augment + 1. The container extension configuration, which can be either a custom payload, one or more extension layers, or container images + 1. The target repository where the built container should be uploaded +1. The Wave service validates the request and authorizes the user via a request to the Platform service. +1. The Wave service checks if the container image already exists in the target registry. +1. If the image does not exist, Wave launches a container build job and pushes the resulting image to the target registry. +1. The Wave service responds with the container image name e.g. `example.com/some/image/build:1234567`. + +Notable parts of this workflow include: + +- Container images provisioned with freeze mode are regular container builds. +- Each container image is associated with a unique ID that is obtained by hashing the following elements: + - The Container file + - Any package dependencies + - The target platform, which is either AMD64 or ARM64 + - The target repository name +- When a request for the same container is made, the same ID is assigned to it and therefore, the build is skipped. +- The resulting images are hosted in your selected repository and not cached locally, unless a cache repository is specified. +- The container images are stored permanently unless the repository owner deletes them. + +## Container provisioning capability matrix + +Wave supports the following types of container builds: + +|Type|Provisioning mode|Source|Freeze|Build repo|Accessibility|Format| +|--- |--- |--- |--- |--- |--- |--- | +|Ephemeral|Augmentation|Container image|No|n/a|Temporary token|Docker| +|Ephemeral|Build|Container file|No|Default|Temporary token|Docker| +|Ephemeral|Build|Conda package|No|Default|Temporary token|Docker| +|Ephemeral|Build|Container file|No|Custom|Temporary token|Docker| +|Ephemeral|Build|Conda package|No|Custom|Temporary token|Docker| +|Durable|Build|Container file|Yes|Custom|Docker auth|Docker /Singularity| +|Durable|Build|Conda package|Yes|Custom|Docker auth|Docker /Singularity| +|Community (durable)|Build|Container file|Yes|Default|Public|Docker /Singularity| +|Community (durable)|Build|Conda package|Yes|Default|Public|Docker /Singularity| diff --git a/docs/sidebar.json b/docs/sidebar.json index cb6df70fa..e41da8226 100644 --- a/docs/sidebar.json +++ b/docs/sidebar.json @@ -1,28 +1,26 @@ { "sidebar": [ - "index", { "type": "category", - "label": "Wave CLI", + "label": "Wave", "collapsed": false, - "link": { - "type": "doc", - "id": "cli/index" - }, "items": [ - "cli/install", - "cli/build-directory", - "cli/build-docker", - "cli/build-conda", - "cli/build-spack", - "cli/build-singularity", - "cli/build-freeze" + "index", + "get-started", + "provisioning" ] }, - "guide", - "api", - "metrics", - "faq", - "troubleshoot" + "nextflow", + { + "type": "category", + "label": "Developer tools", + "collapsed": false, + "items": [ + "cli/index", + "cli/reference", + "api" + ] + }, + "faq" ] }