Releases: Netflix/metaflow
2.4.0 (Oct 4th 2021)
Metaflow 2.4.0 Release Notes
The Metaflow 2.4.0 release is a minor release and includes a breaking change
Breaking Changes
Change return type of created_at/finished_at in the client (#692)
Prior to this release, the return type for created_at
and finished_at
properties in the Client API was a timestamp
string. This release changes this to a datetime
object, as the old behavior is considered an unintentional mis-feature
(see below for details).
How to retain the old behavior
To keep the old behavior, append an explicit string conversion, .strftime('%Y-%m-%dT%H:%M:%SZ')
, to
the created_at
and finshed_at
calls, e.g.
run.created_at.strftime('%Y-%m-%dT%H:%M:%SZ')
Background
The first versions of Metaflow (internal to Netflix) returned a datetime
object in all calls dealing with timestamps in
the Client API to make it easier to perform operations between timestamps. Unintentionally, the return type was changed
to string in the initial open-source release. This release introduces a number of internal changes, removing all
remaining discrepancies between the legacy version of Metaflow that was used inside Netflix and the open-source version.
The timestamp change is the only change affecting the user-facing API. While Metaflow continues to make a strong promise
of backwards compatibility of user-facing features and APIs, the benefits of one-time unification outweigh the cost of this
relatively minor breaking change.
Bug Fixes
Better error messages in case of a Conda issue (#706)
Conda errors printed to stderr
were not surfaced to the user; this release addresses this issue.
Fix error message in Metadata service (#690)
The code responsible for printing error messages from the metadata service had a problem that could cause it to be unable to print the correct error message and would instead raise another error that obfuscated the initial error. This release addresses this issue and errors from the metadata service are now properly printed.
New Features
S3 retry counts are now configurable (#700)
This release allows you to set the number of times S3 access are retried (the default is 7). The relevant environment variable is: METAFLOW_S3_RETRY_COUNT
.
New datastore implementation resulting in improved performance (#580)
The datastore implementation was reworked to make it easier to extend in the future. It also now uploads artifacts in parallel to S3 (as opposed to sequentially) which can lead to better performance. The changes also contribute to a notable improvement in the speed of resume
which can now start resuming a flow twice as fast as before. Documentation can be found here.
S3 datatools performance improvements (#697)
The S3 datatools better handles small versus large files by using the download_file
command for larger files and using get_object
for smaller files to minimize the number of calls made to S3.
2.3.6 (Sep 8th, 2021)
Metaflow 2.3.6 Release Notes
The Metaflow 2.3.6 release is a patch release.
Bug Fixes
Fix recursion error when METAFLOW_DEFAULT_ENVIRONMENT
is set to conda
Prior to this release, setting default execution environment to conda
through METAFLOW_DEFAULT_ENVIRONMENT
would result in a recursion error.
METAFLOW_DEFAULT_ENVIRONMENT=conda python flow.py run
File "/Users/savin/Code/metaflow/metaflow/cli.py", line 868, in start
if e.TYPE == environment][0](ctx.obj.flow)
File "/Users/savin/Code/metaflow/metaflow/plugins/conda/conda_environment.py", line 27, in __init__
if e.TYPE == DEFAULT_ENVIRONMENT][0](self.flow)
File "/Users/savin/Code/metaflow/metaflow/plugins/conda/conda_environment.py", line 27, in __init__
if e.TYPE == DEFAULT_ENVIRONMENT][0](self.flow)
File "/Users/savin/Code/metaflow/metaflow/plugins/conda/conda_environment.py", line 27, in __init__
if e.TYPE == DEFAULT_ENVIRONMENT][0](self.flow)
[Previous line repeated 488 more times]
File "/Users/savin/Code/metaflow/metaflow/plugins/conda/conda_environment.py", line 24, in __init__
from ...plugins import ENVIRONMENTS
RecursionError: maximum recursion depth exceeded
This release fixes this bug.
Allow dots in host_volumes
attribute for @batch
decorator
Dots in volume names - @batch(host_volumes='/path/with/.dot')
weren't being santized properly resulting in errors when a Metaflow task launched on AWS Batch. This release fixes this bug.
2.3.5 (Aug 23rd, 2021)
Metaflow 2.3.5 Release Notes
The Metaflow 2.3.5 release is a patch release.
Features
Enable mounting host volumes in AWS Batch
With this release, you can now mount and access instance host volumes within a Metaflow task running on AWS Batch. To access a host volume, you can add host-volumes
argument to your @batch
decorator -
@batch(host_volumes=['/home', '/var/log'])
Bug Fixes
Fix input values for Parameters of type list
within a Metaflow Foreach task
The following flow had a bug where the value for self.input
was being imputed to None
rather than the dictionary element. This release fixes this issue -
from metaflow import FlowSpec, Parameter, step, JSONType
class ForeachFlow(FlowSpec):
numbers_param = Parameter(
"numbers_param",
type=JSONType,
default='[1,2,3]'
)
@step
def start(self):
# This works, and passes each number to the run_number step:
#
# self.numbers = self.numbers_param
# self.next(self.run_number, foreach='numbers')
# But this doesn't:
self.next(self.run_number, foreach='numbers_param')
@step
def run_number(self):
print(f"number is {self.input}")
self.next(self.join)
@step
def join(self, inputs):
self.next(self.end)
@step
def end(self):
pass
if __name__ == '__main__':
ForeachFlow()
2.3.4 (Aug 11th, 2021)
Metaflow 2.3.4 Release Notes
The Metaflow 2.3.4 release is a patch release.
Bug Fixes
Fix execution of step-functions create
when using an IncludeFile
parameter
PR #607 in Metaflow 2.3.3
introduced a bug with step-functions create
command for IncludeFile
parameters. This release rolls back that PR. A subsequent release will reintroduce a modified version of PR #607.
2.3.3 (Jul 29th, 2021)
Metaflow 2.3.3 Release Notes
The Metaflow 2.3.3 release is a patch release.
Features
Support resource tags for Metaflow's integration with AWS Batch
Metaflow now supports setting resource tags for AWS Batch jobs and propagating them to the underlying ECS tasks. The following tags are attached to the AWS Batch jobs now -
metaflow.flow_name
metaflow.run_id
metaflow.step_name
metaflow.user
/metaflow.owner
metaflow.version
metaflow.production_token
To enable this feature, set the environment variable (or alternatively in the metaflow config
) METAFLOW_BATCH_EMIT_TAGS
to True
. Keep in mind that the IAM role (MetaflowUserRole
, StepFunctionsRole
) submitting the jobs to AWS Batch will need to have the Batch:TagResource
permission.
Bug Fixes
Properly handle None
as defaults for parameters for AWS Step Functions execution
Prior to this release, a parameter specification like -
Parameter(name="test_param", type=int, default=None)
will result in an error even though the default has been specified
Flow failed:
The value of parameter test_param is ambiguous. It does not have a default and it is not required.
This release fixes this behavior by allowing the flow to execute as it would locally.
Fix return value of IncludeFile
artifacts
The IncludeFile
parameter would return JSONified metadata about the file rather than the file contents when accessed through the Metaflow Client
. This release fixes that behavior by returning instead the file contents, just like any other Metaflow data artifact.
2.3.2 (Jun 29th 2021)
Metaflow 2.3.2 Release Notes
The Metaflow 2.3.2 release is a minor release.
- Features
step-functions trigger
command now supports--run-id-file
option
Features
step-functions trigger
command now supports --run-id-file
option
Similar to run
, you can now pass --run-id-file
option to step-function trigger
. Metaflow then will write the triggered run id to the specified file. This is useful if you have additional scripts that require the run id to examine the run or wait until it finishes.
2.3.1 (Jun 23rd 2021)
Metaflow 2.3.1 Release Notes
The Metaflow 2.3.1 release is a minor release.
Features
Performance optimizations for merge_artifacts
Prior to this release, FlowSpec.merge_artifacts
was loading all of the merged artifacts into memory after doing all of the consistency checks with hashes. This release now avoids the memory and compute costs of decompressing, de-pickling, re-pickling, and recompressing each merged artifact - resulting in improved performance of merge_artifacts
.
2.3.0 (May 27th. 2021)
Metaflow 2.3.0 Release Notes
The Metaflow 2.3.0 release is a minor release.
Features
Coordinate larger Metaflow projects with @project
It's not uncommon for multiple people to work on the same workflow simultaneously. Metaflow makes it possible by keeping executions isolated through independently stored artifacts and namespaces. However, by default, all AWS Step Functions deployments are bound to the name of the workflow. If multiple people call step-functions create
independently, each deployment will overwrite the previous one.
In the early stages of a project, this simple model is convenient but as the project grows, it is desirable that multiple people can test their own AWS Step Functions deployments without interference. Or, as a single developer, you may want to experiment with multiple independent AWS Step Functions deployments of their workflow.
This release introduces a @project
decorator to address this need. The @project
decorator is used at the FlowSpec
-level to bind a Flow to a specific project. All flows with the same project name belong to the same project.
from metaflow import FlowSpec, step, project, current
@project(name='example_project')
class ProjectFlow(FlowSpec):
@step
def start(self):
print('project name:', current.project_name)
print('project branch:', current.branch_name)
print('is this a production run?', current.is_production)
self.next(self.end)
@step
def end(self):
pass
if __name__ == '__main__':
ProjectFlow()
python flow.py run
The flow works exactly as before when executed outside AWS Step Functions and introduces project_name
, branch_name
& is_production
in the current
object.
On AWS Step Functions, however, step-functions create
will create a new workflow example_project.user.username.ProjectFlow
(where username
is your user name) with a user-specific isolated namespace and a separate production token.
For deploying experimental (test) versions that can run in parallel with production, you can deploy custom branches with --branch
python flow.py --branch foo step-functions create
To deploy a production version, you can deploy with --production
flag (or pair it up with --branch
if you want to run multiple variants in production)
python project_flow.py --production step-functions create
Note that the isolated namespaces offered by @project
work best when your code is designed to respect these boundaries. For instance, when writing results to a table, you can use current.branch_name to choose the table to write to or you can disable writes outside production by checking current.is_production.
Hyphenated-parameters support in AWS Step Functions
Prior to this release, hyphenated parameters in AWS Step Functions weren't supported through CLI.
from metaflow import FlowSpec, Parameter, step
class ParameterFlow(FlowSpec):
foo_bar = Parameter('foo-bar',
help='Learning rate',
default=0.01)
@step
def start(self):
print('foo_bar is %f' % self.foo_bar)
self.next(self.end)
@step
def end(self):
print('foo_bar is still %f' % self.foo_bar)
if __name__ == '__main__':
ParameterFlow()
Now, users can create their flows as usual on AWS Step Functions (with step-functions create
) and trigger the deployed flows through CLI with hyphenated parameters -
python flow.py step-functions trigger --foo-bar 42
State Machine execution history logging for AWS Step Functions
Metaflow now logs State Machine execution history in AWS CloudWatch Logs for deployed Metaflow flows. You can enable it by specifying --log-execution-history
flag while creating the state machine
python flow.py step-functions create --log-execution-history
Note that you would need to set the environment variable (or alternatively in your Metaflow config) METAFLOW_SFN_EXECUTION_LOG_GROUP_ARN
to your AWS CloudWatch Logs Log Group ARN to pipe the execution history logs to AWS CloudWatch Logs
2.2.13 (May 19th, 2021)
Metaflow 2.2.13 Release Notes
The Metaflow 2.2.13 release is a minor patch release.
Bug Fixes
Handle regression with @batch
execution on certain docker images
Certain docker images override the entrypoint by executing eval
on the user-supplied command. The 2.2.10
release impacted these docker images where we modified the entrypoint to support datastore based logging. This release fixes that regression.
2.2.12 (May 18th, 2021)
Metaflow 2.2.12 Release Notes
The Metaflow 2.2.12 release is a minor patch release.
Features
Add capability to override AWS Step Functions state machine name while deploying flows to AWS Step Functions
Prior to this release, the State Machines created by Metaflow while deploying flows to AWS Step Functions had the same name as that of the flow. With this release, Metaflow users can now override the name of the State Machine created by passing in a --name
argument : python flow.py step-functions --name foo create
or python flow.py step-functions --name foo trigger
.
Introduce heartbeats for Metaflow flows
Metaflow now registers heartbeats at the run level and the task level for all flow executions (with the exception of flows running on AWS Step Functions where only task-level heartbeats are captured). This provides the necessary metadata to ascertain if a run/task has been lost. Subsequent releases of Metaflow will expose this information through the client.
Bug Fixes
Handle regression with Click >=8.0.x
The latest release of Click (8.0.0) broke certain idempotency assumptions in Metaflow which PR #526 addresses.