2.11.13

Features

Configurable default Kubernetes resources

This release introduces configuration options for setting default values for cpu / memory / disk when running on Kubernetes. These can be set either with environment variables

METAFLOW_KUBERNETES_CPU=
METAFLOW_KUBERNETES_MEMORY=
METAFLOW_KUBERNETES_DISK=

or in a Metaflow profile

{
  "KUBERNETES_CPU": "",
  "KUBERNETES_MEMORY": "",
  "KUBERNETES_DISK": "",
}

These values will be overruled by specifying a value through the @kubernetes or @resources decorators.

Improvements

Support for wider foreach flows with Argo Workflows

This release changes the way task ids are generated on Argo Workflows in order to solve an issue where extremely wide foreach splits could not execute correctly due to hard limits on input parameters size on Argo Workflows.

What's Changed

• fix: Deterministic foreach task id's for Argo Workflows by @saikonen in #1704
• Make Kubernetes default resources configurable by @martinhausio in #1800
• Add METAFLOW_ESCAPE_HATCH_WARNING for s3op by default. by @romain-intel in #1825
• bump version to 2.11.13 by @saikonen in #1826

New Contributors

• @martinhausio made their first contribution in #1800

Full Changelog: 2.11.12...2.11.13

2.11.12

What's Changed

• Fix: JSON Reference Path Error in AWS Step Functions Distributed Map by @nidhinnru in #1822
• fix import of the new escape hatch flag by @wangchy27 in #1823

New Contributors

• @nidhinnru made their first contribution in #1822

Full Changelog: 2.11.11...2.11.12

2.11.11

What's Changed

• Add support for getting EC2 metadata through IMDSv2. by @trhodeos in #1808
• Initialize token for IMDSv1 ec2-metadata by @trhodeos in #1809
• allow @conda to accept nvidia as a compute target by @savingoyal in #1811
• fix: github tests by @saikonen in #1813
• fix: R tests by @saikonen in #1814
• Add a flag to control escape hatch log by @wangchy27 in #1819
• upgrade package versions by @wangchy27 in #1821

New Contributors

• @trhodeos made their first contribution in #1808

Full Changelog: 2.11.10...2.11.11

2.11.10

Improvements

Argo Events trigger improvements for parameters with default values

This release fixes an issue where partial or empty argo event payloads would incorrectly overwrite the default values for the parameters of a triggered flow.

For example a flow with

@trigger(events=["params_event"])
class DefaultParamEventFlow(FlowSpec):

    param_a = Parameter(
        name="param_a",
        default="default value A",
    )

    param_b = Parameter(
        name="param_b",
        default="default value B",
    )

will now correctly have the default values for its parameters when triggered by

from metaflow.integrations import ArgoEvent
ArgoEvent('params_event').publish()

or a default value for param_b and the supplied value for param_a when triggered by

ArgoEvent('params_event').publish({"param_a": "custom-value"})

What's Changed

• [Ready for review] replace pull_request_target by @darinyu in #1790
• [bug fix] Flow decorator click option names fix by @valayDave in #1775
• [@kubernetes port] Allow configurable port number by @valayDave in #1793
• Bump vite from 5.0.12 to 5.0.13 in /metaflow/plugins/cards/ui by @dependabot in #1791
• feature: add metadata for argo workflows template owner by @saikonen in #1798
• fix: support default parameter values with argo events by @saikonen in #1797
• release 2.11.10 by @saikonen in #1799

Full Changelog: 2.11.9...2.11.10

2.11.9

What's Changed

• check for falsy values instead of just None by @madhur-ob in #1786
• Bump version to 2.11.9 by @madhur-ob in #1787

Full Changelog: 2.11.8...2.11.9

2.11.8

What's Changed

• fix: support specifying subdirectory with PyPI vcs sources. by @saikonen in #1783
• bump version to 2.11.8 by @saikonen in #1784
• inject keyrings.google-artifactregistry-auth for GCP PyPI by @madhur-ob in #1782

Full Changelog: 2.11.7...2.11.8

2.11.7

What's Changed

• fix access to name and kwargs by @madhur-ob in #1773
• fix log options by @madhur-ob in #1777
• Bump version to 2.11.7 by @madhur-ob in #1780

Full Changelog: 2.11.6...2.11.7

2.11.6

What's Changed

• Generate stubs properly for extensions with lazy loaded modules by @romain-intel in #1765
• ability to customise log driver and log options by @madhur-ob in #1767
• Fix an issue when unpickling functions through the escape hatch by @romain-intel in #1770

Full Changelog: 2.11.5...2.11.6

2.11.5

What's Changed

• fix: allow dot files with --package-suffixes by @saikonen in #1742
• feat: add ephemeral_storage parameter for Batch (Fargate-only) by @stev-0 in #1739
• add security policy by @saikonen in #1508
• fix mypy tests failing by @madhur-ob in #1760
• [Ready for review] make default s3 worker count configurable by @darinyu in #1759
• [Ready for review] add an option to make sidecar send msg func threadsafe by @darinyu in #1757
• [Ready for review] Make Clone-only resume more efficient. by @darinyu in #1719
• [Ready for review] Retrieve original run tasks and clone successful ones directly by @darinyu in #1728
• feature: add alias for batch decorator inferentia by @saikonen in #1762
• Various fixes in the env escape code by @romain-intel in #1734
• upgrade versions to 2.11.5 by @wangchy27 in #1763
• fix format error for py3.5 by @wangchy27 in #1764

New Contributors

• @stev-0 made their first contribution in #1739

Full Changelog: 2.11.4...2.11.5

2.11.4

What's Changed

• fix: update list of reserved parameter names by @saikonen in #1736
• [card-client] make Card.get_data more efficient by @valayDave in #1750

Full Changelog: 2.11.3...2.11.4