Releases: Netflix/metaflow
2.12.6
Improvements
Fix Argo Workflows issue with long static splits
This release fixes an issue where a join step of a static split would fail on Argo Workflows in rare cases where the length of step names exceeded a threshold.
Argo Events support for parameter names with dashes
Fixes an issue where values from an Argo Event payload did not correctly map to flow Parameters if the parameter name contained dashes.
More specific PyPI errors for package resolving
The errors for @pypi
should now be more clear in cases where it is unable to successfully resolve an environment due to not finding a suitable package.
What's Changed
- Bump braces from 3.0.2 to 3.0.3 in /metaflow/plugins/cards/ui by @dependabot in #1898
- feature: polish pypi package errors by @saikonen in #1905
- fix: long names with static split fails join step on Argo Workflows by @saikonen in #1907
- feature: Conda env extension hooks by @saikonen in #1902
- fix: support for dashed parameters through argo events by @saikonen in #1908
- bump version to 2.12.6 by @saikonen in #1909
Full Changelog: 2.12.5...2.12.6
2.12.5
What's Changed
- add timestamps to conda debug logs by @savingoyal in #1889
- fix: decorator attributes being modified in get_environment by @saikonen in #1895
- bump version to 2.12.5 by @saikonen in #1896
- [cards] bug fix with error card renders by @valayDave in #1893
Full Changelog: 2.12.4...2.12.5
2.12.4
What's Changed
- Fix escape hatch to make getattr behavior more standard by @romain-intel in #1883
- Fix/remember namespace by @romain-intel in #1873
- Remove global nature of parameters and flow decorators by @romain-intel in #1886
- Install PyPI packages in interpreter's site packages by @savingoyal in #1890
- bump version to 2.12.4 by @saikonen in #1891
Full Changelog: 2.12.3...2.12.4
2.12.3
2.12.2
What's Changed
- Fix issue with setting metadata when using the runner by @romain-intel in #1875
- Bump version to 2.12.2 by @romain-intel in #1876
Full Changelog: 2.12.1...2.12.2
2.12.1
Features
Configurable default decorators
This release adds the ability to configure default decorators that will be applied to all steps. This is achieved by setting the decospecs as a value (space separated) for METAFLOW_DECOSPECS
either as an environment variable or in a config.json
The following example would add retry and kubernetes decorators with a custom memory value to all steps:
export METAFLOW_DECOSPECS="kubernetes:memory=4096 retry"
Defining a decorator with the --with
keyword will override the defaults configured. Same applies for explicitly adding a decorator in the flow file.
Improvements
Correctly clean up Argo Workflow sensors when using @project
This release fixes an issue where argo-workflows delete
did not correctly remove possible sensors associated with the workflow if the workflow used the @project
decorator.
What's Changed
- Add the possibility of defining default decorators for steps by @romain-intel in #1837
- bugfix: properly deletes Argo Events trigger sensors when
@project
is used by @gabriel-rp in #1871 - S3PubObject was not used properly after #1807 by @romain-intel in #1872
- bump version to 2.12.1 by @saikonen in #1874
New Contributors
- @gabriel-rp made their first contribution in #1871
Full Changelog: 2.12.0...2.12.1
2.12.0
Features
Support running flows in notebooks and through Python scripts
This release introduces a new Runner API that makes it simple to run flows inside Notebooks, or as part of Python code.
Read the blog post on the feature, or dive straight into the documentation to start using it.
What's Changed
- Remove dead code by @savingoyal in #1853
- initial runner api by @madhur-ob in #1732
- synchronous run and resume functionality + nbrun for runner API by @madhur-ob in #1845
- Tests for runner by @savingoyal in #1859
- minor nbrun fixes by @madhur-ob in #1860
- fix leaked message by @madhur-ob in #1861
- raise exception instead by @madhur-ob in #1862
- allow output to be hidden in nbrun() by @tuulos in #1864
- show_output as True by default by @madhur-ob in #1865
- add explicit cleanup() methods in Runners by @tuulos in #1863
- Runner docstring fixes by @tuulos in #1866
- release 2.12.0 by @savingoyal in #1867
Full Changelog: 2.11.16...2.12.0
2.11.16
Features
Support GCP Secret Manager
This release adds support for using GCP Secret manager to supply secret values for steps environment.
In order to enable the secret manager, you should specify the type by setting METAFLOW_DEFAULT_SECRETS_BACKEND_TYPE
to gcp-secret-manager
or specifying it in the decorator
@secrets(sources=[{"type": "gcp-secret-manager", "id": "some-secret-key"}])
METAFLOW_GCP_SECRET_MANAGER_PREFIX
can be set in order to not have to write full secret locations.
Support Azure Key Vault
This release also adds support for Azure Key Vault as a secrets backend. Specify az-key-vault
as the secret backend type to use this.
Same as with the other secret managers, we provide a prefix config to avoid having to repeat common parts in the secret keys. Configure this by setting METAFLOW_AZURE_KEY_VAULT_PREFIX
Note: Currently only Secret
object types are supported when using Azure Key Vault.
@parallel
for Kubernetes
This release adds support for @parallel
when flows are run --with kubernetes
Example:
@step
def start(self):
self.next(self.parallel_step, num_parallel=3)
@kubernetes(cpu=1, memory=512)
@parallel
@step
def parallel_step(self):
...
Configurable runtime limits
It is now possible to configure the default timeout for the @timeout
decorator. This can be done by setting METAFLOW_DEFAULT_RUNTIME_LIMIT
in the environment, or in a config.json
Improvements
Resumed flows should record task competions correctly
Fixes an issue where tasks that were cloned from a previous run by resume
would not show up as completed on the Metaflow UI due to missing metadata
Fix accessing task index of a foreach task
There was an issue accessing the index of a foreach task via the client. With this release it is possible to do the following
from metaflow import Task
task = Task("ForeachFlow/123/foreach_step/task-00000000")
task.index
What's Changed
- [runtime limits] make runtime limits configurable by @valayDave in #1834
- [Ready for Review] fix bug where client can not access foreach stack by @darinyu in #1766
- [Ready for Review] add attempt_ok flag so that UI will not show up unknown node by @darinyu in #1830
- pluggable azure credentials provider by @oavdeev in #1756
- Secret Backend Support for Azure Key Vault by @iamsgarg-ob in #1839
- reducing the dep version to 4.7.0 (#47) by @iamsgarg-ob in #1840
- [
@parallel
on Kubernetes] support for Jobsets by @valayDave in #1804 - Pluggable GCP auth by @oavdeev in #1841
- GCP secret manager support by @oavdeev in #1842
- [jobsets] py3.5 compatibility fixes. by @valayDave in #1844
- Support Python 3.5 for tests by @savingoyal in #1843
- py3.5 compatibility fixes [azure/gcp/jobsets] by @valayDave in #1848
- [version bump] for release by @valayDave in #1847
- [OB-625] adding metaflow/cron annotation to argo workflows by @iamsgarg-ob in #1852
New Contributors
- @iamsgarg-ob made their first contribution in #1839
Full Changelog: 2.11.15...2.11.16
2.11.15
Features
Displaying task attempt logs
When running a task with the @retry
decorator, previously we were able to only view the logs of the latest attempt of the task.
With this release it is now possible to target a specific attempt with the --attempt
option for the logs command
python example.py logs 123/retry_step/456 --attempt 2
Scrubbing log contents
This release introduces a new command for scrubbing log contents of tasks in case they contain sensitive information that needs to be redacted.
Simplest use case is scrubbing the latest task logs. By default both stdout
and stderr
are scrubbed
python example.py logs scrub 123/example/456
There are also options to target only a specific log stream
python example.py logs scrub 123/example/456 --stderr
python example.py logs scrub 123/example/456 --stdout
when using the@retry
decorator, tasks can have multiple attempts with separate logs that require scrubbing. By default only the latest attempt is scrubbed. There are options to make scrubbing multiple attempts easier
# scrub specific attempt
python example.py logs scrub 123/retry_step/456 --attempt 1
# scrub all attempts
python example.py logs scrub 123/retry_step/456 --all
# scrub specified attempt and all prior to it (this would scrub attempts 0,1,2,3)
python example.py logs scrub 123/retry_step/456 --all --attempt 3
The command also accepts only specifying a step for scrubbing. This is useful for steps with multiple tasks, like a foreach split.
python example.py logs scrub 123/foreach_step
all the above options also apply when targeting a step for scrubbing.
Note: Log scrubbing for running tasks is not recommended, and is actively protected against. There can be occasions where a task has failed in such a way that it still counts as not completed. For such a case you can supply the --include-not-done
option to try and scrub it as well.
What's Changed
- feature: scrub logs by @saikonen in #1802
- Fix issue #1805 by @narayanacharya6 in #1807
- bug fix: unused arg for batch step --help by @mae5357 in #1817
- bump version to 2.11.15 by @saikonen in #1829
New Contributors
- @narayanacharya6 made their first contribution in #1807
- @mae5357 made their first contribution in #1817
Full Changelog: 2.11.14...2.11.15
2.11.14
What's Changed
- Increase Azure Blobstore connection_timeout. by @shrinandj in #1827
- bump version to 2.11.14 by @shrinandj in #1828
Full Changelog: 2.11.13...2.11.14