2.2.8 (Mar 15th, 2021)

Metaflow 2.2.8 Release Notes

The Metaflow 2.2.8 release is a minor patch release.

• Bug Fixes
  • Fix @environment behavior for conflicting attribute values
  • Fix environment is not callable error when using @environment

Bugs

Fix @environment behavior for conflicting attribute values

Metaflow was incorrectly handling environment variables passed through the @environment decorator in some specific instances. When @environment decorator is specified over multiple steps, the actual environment that's available to any step is the union of attributes of all the @environment decorators; which is incorrect behavior. For example, in the following workflow -

from metaflow import FlowSpec, step, batch, environment
import os
class LinearFlow(FlowSpec):
    @environment(vars={'var':os.getenv('var_1')})
    @step
    def start(self):
        print(os.getenv('var'))
        self.next(self.a)
    @environment(vars={'var':os.getenv('var_2')})
    @step
    def a(self):
        print(os.getenv('var'))
        self.next(self.end)
    @step
    def end(self):
        pass
if __name__ == '__main__':
    LinearFlow()

var_1=foo var_2=bar python flow.py run

will result in

Metaflow 2.2.7.post10+gitb7d4c48 executing LinearFlow for user:savin
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
2021-03-12 20:46:04.161 Workflow starting (run-id 6810):
2021-03-12 20:46:04.614 [6810/start/86638 (pid 10997)] Task is starting.
2021-03-12 20:46:06.783 [6810/start/86638 (pid 10997)] foo
2021-03-12 20:46:07.815 [6810/start/86638 (pid 10997)] Task finished successfully.
2021-03-12 20:46:08.390 [6810/a/86639 (pid 11003)] Task is starting.
2021-03-12 20:46:10.649 [6810/a/86639 (pid 11003)] foo
2021-03-12 20:46:11.550 [6810/a/86639 (pid 11003)] Task finished successfully.
2021-03-12 20:46:12.145 [6810/end/86640 (pid 11009)] Task is starting.
2021-03-12 20:46:15.382 [6810/end/86640 (pid 11009)] Task finished successfully.
2021-03-12 20:46:15.563 Done!

Note the output for the step a which should have been bar. PR #452 fixes the issue.

Fix environment is not callable error when using @environment

Using @environment would often result in an error from pylint - E1102: environment is not callable (not-callable). Users were getting around this issue by launching their flows with --no-pylint. PR #451 fixes this issue.