-
Notifications
You must be signed in to change notification settings - Fork 766
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[parallel-fixes] core + test changes
- fix tests for interplay of @Secrets and @parallel - Local runtime will allow secrets only on worker tasks - Secrets will be set on all kinds of tasks when run remotely (control/worker) - fix tests for ubf based on new changes to core. - fix tests for tag-catch for ubf based on new changes to core. - internal ubf decorator has a internal_task_type set to it. - [feedback] register metadata in parallel decorator - [feedback]@parallel inject in current: - move `current.parallel` from `metaflow_current` to `parallel_decorator` - [feedback] appropariately setting task-metadata for parallel stuff - The 'world size' metadata will be set in the @parallel decorator. - The 'node-index' metadata, however, varies depending on the type of computing environment executing the task so it will be set in the appropriate compute decorators. - One reason to specify 'node-index' within compute decorators is that the parallel implementation in the compute decorator might not directly set the required environment variables (`MF_PARALLEL_*`). Instead, these values may be established during the `task_pre_step` phase of the compute decorator using other environment variables set during the implementation. - adding some aws batch changes - [feedback] safety check for _parallel_buf_iter in task_pre_step for @parallel - [feedback] set `is_parallel` in current to denote a step is running under an `@parallel` decorator.
- Loading branch information
Showing
8 changed files
with
193 additions
and
82 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.