Releases: Netflix/metaflow
2.11.3
What's Changed
- remove 3.5 and 3.6 from classifiers for stubs by @madhur-ob in #1727
- Distributed Map Support in AWS Step Functions by @savingoyal in #1720
- [card] bug fix when card type doesn't exist by @valayDave in #1725
- fix: add otel endpoint to batch env by @saikonen in #1747
- expose resolved dependencies through .environment_info by @savingoyal in #1740
- fix stubs version check by @madhur-ob in #1748
- Bump version to 2.11.3 by @madhur-ob in #1749
- [Ready for review] Add explicit max length control for foreach value. by @darinyu in #1733
- [Ready for review] Ignore _foreach_values in merge artifacts by @darinyu in #1729
- add alert details using slack block kit by @russellbrooks in #1751
Full Changelog: 2.11.2...2.11.3
2.11.2
What's Changed
- fix github action for publishing stubs by @madhur-ob in #1707
- add ability to terminate execution of a step-fn state machine by @madhur-ob in #1695
- Increase timeout to get card to reduce test failures by @romain-intel in #1717
- [kubernetes] Use KUBECONFIG if set by @leogargu in #1716
- publish action should be dependent on stubs tests by @madhur-ob in #1718
- fix overwrite of neuron devices when efa devices are also specified by @emattia in #1656
- Add support for shared_memory for Kubernetes workloads. by @shrinandj in #1709
- add repr and str methods to Parameter by @raybellwaves in #1714
- Release 2.11.2 by @saikonen in #1721
- fix missing long_description for setup.py for stubs by @madhur-ob in #1722
- fix pypi URL for stubs by @madhur-ob in #1723
- new token for publishing stubs by @madhur-ob in #1724
New Contributors
- @leogargu made their first contribution in #1716
- @raybellwaves made their first contribution in #1714
Full Changelog: 2.11.1...2.11.2
2.11.1
Features
Auto generated stubs
package
You can now install metaflow[stubs]
to install an additional metaflow-stubs
package that provides type hints for Metaflow decorators (and other objects). Some hints were previously included inline but this package expands their use.
What's Changed
- Set parameters for running flow instead of any imported by @madhur-ob in #1691
- Bump vite from 5.0.8 to 5.0.12 in /metaflow/plugins/cards/ui by @dependabot in #1692
- [Ready for Review] Support more convenient foreach variable/value search-ability. by @darinyu in #1664
- Add auto-generated pyi files to Metaflow by @romain-intel in #1557
- Bump version to 2.11.1 by @romain-intel in #1706
New Contributors
Full Changelog: 2.11.0...2.11.1
2.11.0
2.10.11
What's Changed
- [realtime-card] pre-release bug Fixes. by @valayDave in #1686
- handle race condition with local packages meta by @savingoyal in #1688
Full Changelog: 2.10.10...2.10.11
2.10.10
Features
Support git repositories as @pypi
dependencies
This release adds support for git repositories as dependencies for the @pypi
decorator.
Usage
You can pin the version of the dependency with "@branch"
, "@commit"
, "@tag"
or the head of the main branch with ""
.
Under the hood @pypi
will resolve the dependency to a commit at the moment of creating the environment.
Limitations
Currently conda and pypi environments are created in advance before deploying on remote platforms, this includes gathering all the dependencies and bundling them up. The dependencies for pypi have been limited to binaries only in order to support cross-platform deployments.
For the git sources, we build a wheel from the source and check whether it is compatible with the target platform. This limits using sources where the output wheel is a platform specific binary, for example if building on an ARM platform but trying to deploy on x86.
What's Changed
- support git repos for
@pypi
by @savingoyal in #1681 - bump version to 2.10.10 by @saikonen in #1684
- chore: cleanup pypi bootstrap by @saikonen in #1685
Full Changelog: 2.10.9...2.10.10
2.10.9
Features
EFS Volume support for @batch
This release adds support for mounting existing EFS volumes with @batch
Usage
You can specify one or multiple volumes to be mounted with the batch decorators efs_volumes=
attribute. The volumes will be mounted under /mnt/volume-name
by default, but you can also specify a custom mount point separated by a semicolon with the volume.
Examples
@batch(efs_volumes="fs-001")
@step
...
@batch(efs_volumes=["fs-002", "fs-003"])
@step
...
@batch(efs_volumes="fs-003:/mnt/custom-mountpoint"])
@step
Support custom mount point for host volumes with @batch
This release also adds support for custom mount points for host volumes with @batch
.
Example
@batch(host_volumes="/home:/mnt/host-homedir")
@step
Improvements
@conda
and @pypi
changes
@pypi
can now handle dependencies with special characters in versions
Fixes a @conda
bug where environments were not appending to PATH
during runtime if the decorator was applied implicitly to the step. This bug affected dependencies that install a binary, causing them to not be found during runtime due to the environment missing from the PATH
.
Decorator lifecycle improvement
Previously the lifecycle methods task_pre_step
and task_decorate
were being called consecutively on a per-decorator basis. This release changes the lifecycle so that task_pre_step
is called on all decorators before any task_decorate
is called.
Better checking of datastore dependencies
This release improves the way that datastore dependencies are checked, in order to avoid an issue where previously run metadata could be published to the metadata service, but the execution fails to start due to missing dependencies for pushing data to the datastore.
What's Changed
- [@card] Realtime cards (1/N) by @valayDave in #1550
- fix rendering of zero-length DataFrames in default card by @amerberg in #1652
- fix: pypi decorator handling packages with special characters by @saikonen in #1643
- fix: conda environment counts as disabled with implicit decorator by @saikonen in #1668
- fix: decorator init order by @saikonen in #1669
- don't create run ids if dependencies for data store are missing i.e. fail fast by @madhur-ob in #1666
- Bump follow-redirects from 1.15.2 to 1.15.4 in /metaflow/plugins/cards/ui by @dependabot in #1675
- [@card] Realtime cards (2/N) by @valayDave in #1551
- AWS Batch decorator: EFS volumes support (and custom mount point) by @enricomarchesin in #1650
- add linting for EFS volume mount by @madhur-ob in #1679
- Bump version to 2.10.9 by @saikonen in #1680
New Contributors
- @madhur-ob made their first contribution in #1666
- @enricomarchesin made their first contribution in #1650
Full Changelog: 2.10.8...2.10.9
Metaflow 2.10.8
Improvements
- Warnings about regex patterns not being raw strings were fixed.
- An issue with how the environment escape dealt with aliased modules was fixed.
What's Changed
- fix: change regex patterns to raw strings by @saikonen in #1645
- raw strings to fix deprecation warnings by @russellbrooks in #1646
- Fix an issue with aliases in environment escape server mappings. by @romain-intel in #1649
New Contributors
- @russellbrooks made their first contribution in #1646
Full Changelog: 2.10.7...2.10.8
2.10.7
Improvements
pypi decorator enhancements
This release fixes support for pip environment variables that specify a custom location for the config file (PIP_CONFIG_FILE
or PIP_CONFIG
).
The release also adds support for defining a custom index-url through the pip supported environment variable PIP_INDEX_URL
What's Changed
- Add deployed-at timestamp in annotations for argo-workflows by @rohanrebello in #1633
- fix: pip config issues with pypi decorator by @saikonen in #1626
- Bump version to 2.10.7 by @saikonen in #1636
Full Changelog: 2.10.6...2.10.7
2.10.6
Improvements
Fix environment activation issue with pypi
decorator
The pypi
decorator had a bug that caused it to be treated as disabled
unless specifically passing disabled=False
as an attribute to it.
This release fixes the default case so that pypi
environments activate correctly.
Add debug flag to tracing
This release adds a METAFLOW_DEBUG_TRACING
environment variable to toggle more verbose output for tracing related issues.
By default any errors related to missing tracing dependencies are now silenced completely, in order to not affect platforms that might want tracing environment variables present for all deployments, whether they have the required dependencies or not.
What's Changed
- fix: output ImportError for tracing module to stderr by @saikonen in #1619
- feature: add debug_tracing flag by @saikonen in #1621
- fix: pypi environment issues by @saikonen in #1623
- Bump version to 2.10.6 by @saikonen in #1625
Full Changelog: 2.10.5...2.10.6