Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: freeze environment packages before bootstrapping #1502

Closed

Conversation

saikonen
Copy link
Collaborator

This aims to resolve issue #906 where installing code package bootstrapping dependencies inadvertently upgrades dependencies, possibly breaking existing libraries that depend on older versions.
e.g.

aiobotocore==1.4.2
botocore==1.20.106

$ pip install boto3
# observe that botocore was upgraded, breaking the installed version of aiobotocore

Alternative approach to #1490

Testing notes:

  • by freezing the environment packages and appending them to the pip install commands, it successfully guards all existing packages from upgrades, instead trying to resolve the minimum viable version for the requested packages, e.g. awscli, boto3
  • because this relies on pip to resolve the dependencies, it can be quite slow in cases where environment packages could introduce conflicts. With a test case I observed a ~4min overhead with the pip install.
  • the regular use case where awscli and boto3 can be installed without concerns remains fast.

…n order to protect existing dependencies from upgrades
@saikonen
Copy link
Collaborator Author

Seeing the overhead of the pip install is quite straightforward from the flow logs, as it is more or less the time delta between
Setting up task environment. and Downloading code package...

A major concern with this approach is the time it takes pip to resolve the dependencies in complex cases.

@tylerpotts
Copy link
Contributor

tylerpotts commented Aug 15, 2023

While this bash line is a little lengthy, it would get around doing any pip install operations if boto3 and awscli are present

pip show awscli &> /dev/null; x1=$?; pip show boto3 &> /dev/null; x2=$?; if ! (( $(($x1 + $x2)) == 0 )); then pip install awscli boto3; fi

@saikonen
Copy link
Collaborator Author

While this bash line is a little lengthy, it would get around doing any pip install operations if boto3 and awscli are present

pip show awscli &> /dev/null; x1=$?; pip show boto3 &> /dev/null; x2=$?; if ! (( $(($x1 + $x2)) == 0 )); then pip install awscli boto3; fi

To my understanding the regular pip install behavior would match that of the bash script, as trying to install a package without version pinning should result in a no-op.

This approach would still break with the initial example, as in it boto3 is not present, so the pip install would kick off, resulting in botocore being upgraded, breaking the existing version of aiobotocore.

@savingoyal
Copy link
Collaborator

postponing this PR for now since #1490 has been closed.

@savingoyal savingoyal closed this Aug 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants