Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

@pypi and faster @conda #1505

Merged
merged 47 commits into from
Oct 4, 2023
Merged

@pypi and faster @conda #1505

merged 47 commits into from
Oct 4, 2023

Conversation

savingoyal
Copy link
Collaborator

@savingoyal savingoyal commented Aug 23, 2023

To try it out

python linear_flow.py --environment=conda run --with kubernetes

@savingoyal
Copy link
Collaborator Author

All test failures are due to a (un)related issue with how we vendor packages.

@savingoyal savingoyal added the in review Currently under review label Aug 28, 2023
metaflow/plugins/pypi/conda_decorator.py Show resolved Hide resolved
metaflow/plugins/pypi/conda_environment.py Outdated Show resolved Hide resolved
metaflow/plugins/pypi/__init__.py Show resolved Hide resolved
metaflow/plugins/pypi/__init__.py Show resolved Hide resolved
metaflow/plugins/pypi/utils.py Outdated Show resolved Hide resolved
metaflow/plugins/pypi/utils.py Outdated Show resolved Hide resolved
saikonen
saikonen previously approved these changes Oct 3, 2023
@saikonen saikonen dismissed their stale review October 3, 2023 23:15

butterfingered approval, will still run through some tests but changes I requested are ok :)

saikonen
saikonen previously approved these changes Oct 3, 2023
Copy link
Collaborator

@saikonen saikonen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

expected functionality seems to be in place. tested both conda and pypi decorator usage with sfn,batch,argo, kubernetes, and local execution. No issues found so approving from my end :)

Copy link
Contributor

@romain-intel romain-intel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not done. Pushing current comments as requested.

metaflow/plugins/__init__.py Show resolved Hide resolved
metaflow/plugins/pypi/bootstrap.py Show resolved Hide resolved
# Create Conda environment.
cmds = [
# TODO: check if mamba or conda are already available on the image
f"""if ! command -v ./micromamba >/dev/null 2>&1; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why just in current dir?

metaflow/plugins/pypi/conda_environment.py Show resolved Hide resolved
metaflow/plugins/pypi/conda_decorator.py Show resolved Hide resolved
metaflow/plugins/pypi/micromamba.py Show resolved Hide resolved
metaflow/plugins/pypi/micromamba.py Show resolved Hide resolved
metaflow/plugins/pypi/micromamba.py Outdated Show resolved Hide resolved
metaflow/plugins/pypi/pip.py Show resolved Hide resolved
metaflow/plugins/pypi/conda_environment.py Show resolved Hide resolved
Copy link
Contributor

@romain-intel romain-intel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. I think I finished looking at it. It's mostly minor stuff. I think the pinned dep stuff warrants fixing since I don't see a compelling reason to change it (unless I am missing something). It feels gratuitous to break something that people may be relying on (as you say, if it's in the code :) ).

There is some concern around the general way things are done in a not very multiple-process safe kind of way (even removing some of the previous locks that were in the previous implementation) so I suspect that we will hit some issues there at some point but it probably won't impact users directly as it is not very common.

metaflow/plugins/pypi/micromamba.py Show resolved Hide resolved
metaflow/plugins/pypi/micromamba.py Show resolved Hide resolved
@savingoyal
Copy link
Collaborator Author

#1567 re-enables pinned_conda_libs. re - removing the locks - it shouldn't be an issue anymore. Dangling lock files were a big headache in the previous implementation - now everything happens in a multi-process safe manner - so we should be good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in review Currently under review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants