Releases: Netflix/metaflow
2.10.5
What's Changed
- fix nulls in pandas columns on default cards by @amerberg in #1606
- CatchDecorator: Fix function signature by @tfurmston in #1615
- fix: remove leftover use of LooseVersion by @saikonen in #1617
- bump version to 2.10.5 by @saikonen in #1618
Full Changelog: 2.10.4...2.10.5
2.10.4
Features
Support for tracing
With this release it is possible to gather telemetry data using an opentelemetry endpoint.
Specifying an endpoint in one of the environment variables
METAFLOW_OTEL_ENDPOINT
METAFLOW_ZIPKIN_ENDPOINT
will enable the corresponding tracing provider.
Some additional dependencies are required for the tracing functionality in the execution environment. These can be installed in the base Docker image, or supplied through a conda environment. The relevant packages are
opentelemetry-sdk, opentelemetry-api, opentelemetry-instrumentation, opentelemetry-instrumentation-requests
and depending on your endpoint, either opentelemetry-exporter-otlp
or opentelemetry-exporter-zipkin
Custom index support for the pypi
decorator
The pypi
decorator now supports using a custom index in the users Pip configuration under global.index-url
.
This enables using private indices, even ones that require authentication.
For example the following would set up one authenticated and two extra non-authenticated indices for package resolution
pip config set global.index-url "https://user:token@example.com"
pip config set global.extra-index-url "https://extra.example.com https://extra2.example.com"
Specify Kubernetes job ephemeral storage size through resources
decorator
It is now possible to specify the ephemeral storage size for Kubernetes jobs when using the resources
decorator with the disk=
attribute.
Introduce argo-workflows status
command
Adds a command for easily checking the current status of a workflow on Argo workflows.
python flow.py argo-workflows status [run-id]
Improvements
Add more randomness to Kubernetes pod names to avoid collisions
There was an issue where relying solely on the Kubernetes apiserver for generating random pod names was resulting in significant collisions with sufficiently large number of executions.
This release adds more randomness to the pod names besides what is generated by Kubernetes.
Fix issues with resources
decorator in combination with step functions
This release fixes an issue where deploying flows on AWS Step Functions was failing in the following cases
@resources(shared_memory=)
with any value- combining
@resources
and@batch(use_tmpfs=True)
What's Changed
- Bump postcss from 8.4.24 to 8.4.31 in /metaflow/plugins/cards/ui by @dependabot in #1599
- introduce argo-workflows status by @savingoyal in #1600
- remove dependency on wget for micromamba by @savingoyal in #1601
- fix: resource decorator step functions issue by @saikonen in #1610
- feature: add status grouping to argo-workflows output by @saikonen in #1604
- fix: apply only supported attributes from resources decorator in batch by @saikonen in #1586
- fix: use correct deco for limiting keys. by @saikonen in #1611
- Otel oss integration by @wangchy27 in #1462
- Remove use of distutils from code (not setup.py yet) by @romain-intel in #1585
- Add a uid for the kubernetes job creation by @tylerpotts in #1588
- feature: support custom index config for pypi decorator by @saikonen in #1613
- Adding 'disk' parameter to @resources decorator by @jaypond in #1500
- Bump version to 2.10.4 by @saikonen in #1614
New Contributors
Full Changelog: 2.10.3...2.10.4
2.10.3
What's Changed
- Safely parse
pandas.DataFrame
indexes for default card by @amerberg in #1574 - search for micromamba in more locations by @savingoyal in #1581
- fix timezone in default card time format string by @amerberg in #1575
- change writable location for local micromamba by @savingoyal in #1583
- introduce triggers in annotations by @savingoyal in #1569
- Fix escape hatch test by @romain-intel in #1558
- EFA with pre-commit formatting by @emattia in #1584
- return event identifier in response to
ArgoEvent.publish
by @savingoyal in #1587 - Add parameter info to argo workflow template annotations by @rohanrebello in #1549
- avoid race condition with argo-workflows by @savingoyal in #1595
- Bump version to 2.10.3 by @saikonen in #1598
New Contributors
- @rohanrebello made their first contribution in #1549
Full Changelog: 2.10.2...2.10.3
2.10.2
2.10.2
Features
-
New configuration option to use same headers as metadata service for argo events webhook calls by @oavdeev in #1560 . Default behavior is the same as before.
-
Metaflow CLI now supports
list-workflow-templates
command to list deployed argo workflows by @saikonen in #1577
Full Changelog: 2.10.0...2.10.2
2.10.0
Coming soon!
Full Changelog: 2.9.15...2.10.0
2.9.15
Improvements
Improve the performance of parallel_map
We now check for processes in the order in which they complete not in the order in which they are launched. This also increases the likelihood of failing fast.
Fix issues with the environment escape mechanism
Deadlocks and errors could occur when using the environment escape mechanism in two cases: (a) GC would occur at an inopportune moment or (b) subprocesses were involved. Both issues were fixed.
What's Changed
- Fix two possible issues with the environment escape communication by @romain-intel in #1555
- Check for the first finished parallel proccess by @maxzheng in #1546
New Contributors
Full Changelog: 2.9.14...2.9.15
2.9.14
Improvements
Fixes merging of log lines
This release fixes an issue with merging broken log lines.
Fix issue with using LD_LIBRARY_PATH
with Conda environments
In a Conda environment, it is sometimes necessary to set LD_LIBRARY_PATH
to first include the Conda's environment libraries before anything else. Prior to this release, this used to cause issues with the escape hatch.
What's Changed
- fix: GPU scheduling issue by @saikonen in #1520
- vend packaging by @savingoyal in #1506
- fix: vendor older version of packaging for python 3.5 by @saikonen in #1530
- Fix an issue when LD_LIBRARY_PATH is overriden in a Conda environment by @romain-intel in #1540
- Fix the merging of borked lines with MFLOG by @romain-intel in #1533
- Revert "fix: GPU scheduling issue" by @savingoyal in #1542
- Bump version to 2.9.14 by @saikonen in #1543
Full Changelog: 2.9.13...2.9.14
2.9.13
Bug fix
Revert annotations changes to fix a regression
The recent annotations feature introduced an issue where project
, flow_name
or user
annotations are not being populated for Kubernetes. This release reverts the changes.
What's Changed
- Revert "Adds custom annotations via env variables" by @savingoyal in #1516
- Bump to 2.9.13 by @savingoyal in #1517
Full Changelog: 2.9.12...2.9.13
2.9.12
Known issues
The annotations feature introduced in this release has an issue where project, flow_name or user annotations are not being populated for Kubernetes. This has been reverted in the next release.
Features
Custom annotations for K8S and Argo Workflows
This release enables users to add custom annotations to the Kubernetes resources that Flows create. The annotations can be configured much in the same way as custom labels
- Globally with an environment variable. For example with
export METAFLOW_KUBERNETES_ANNOTATIONS="first=A,second=B"
- At a step level by passing a dictionary to the Kubernetes decorator.
@kubernetes(annotations={"first": "A", "second": "B"})
What's Changed
- Adds custom annotations via env variables by @tylerpotts in #1442
- Pass the user-defined executable to environment's
executable
by @romain-intel in #1454 - Remove validate_environment from task lifecycle by @savingoyal in #1507
- Fix/863 - Improve error message in metaflow.S3 class when DATATOOLS_S3ROOT is not configured. by @tfurmston in #1491
- Fix an issue where 0 was not considered False for extension debug opt… by @romain-intel in #1511
- Bump version to 2.9.12 by @saikonen in #1514
Full Changelog: 2.9.11...2.9.12
2.9.11
Bug Fix
Fix regression for @Batch decorator introduced by v2.9.10
This release reverts a validation fix introduced in 2.9.10, which prevented executions of Metaflow tasks on AWS Batch
What's Changed
- Revert "fix: validate required configuration for Batch" by @savingoyal in #1486
- Bump version to 2.9.11 by @savingoyal in #1487
Full Changelog: 2.9.10...2.9.11