Upload local Pants PEXs to release #19450

thejcannon · 2023-07-11T20:39:29Z

From #19443 and fixes #18434 (lol)

This change does 2 things:

Eliminates the PEX building from release.py in favor of just ./pants package src/python/pants:pants-pex. It isn't universal, but we're moving away from the universal PEX.
Tweaks CI to package this PEX, and if this is a release, upload the PEX to the GitHub release
(plus docs changes)

thejcannon · 2023-07-11T20:39:55Z

This PR currently depends on changed from #19444, so it's draft until that gets merged and I can update this branch.

It stands on its own though, so please do review.

thejcannon · 2023-07-11T20:40:53Z

src/python/pants/BUILD

+pex_binary(
+    name="pants-pex",
+    dependencies=[":pants-packaged"],
+    script="pants",
+    execution_mode="venv",
+    shebang="/usr/bin/env python",
+    strip_pex_env=False,
+    layout="zipapp",
+)


This was copied from release.py, but without the --platforms flag. As I understand it, this will resolve only for the local platform, so that's exactly what we want!

I also am hardcoding zipapp PEX to guard against __defaults__ accidentally changing that.

I believe that you understand correctly.

thejcannon · 2023-07-11T20:42:18Z

src/python/pants_release/generate_github_workflows.py

+                            "if": "needs.release_info.outputs.is-release == 'true'",
+                            "run": dedent(
+                                """\
+                                LOCAL_TAG=$(PEX_INTERPRETER=1 dist/src.python.pants/pants-pex.pex -c "import sys;major, minor = sys.version_info[:2];import os;uname = os.uname();print(f'cp{major}{minor}-{uname.sysname.lower()}_{uname.machine.lower()}')")


This is my assume-nothing way of generating the tag. Using the PEX itself means we'll land the right IC, and using Python means we'll be confident scie-pants's Python script can produce a similar tag.

The value looks like: cp39_linux_x86_64

thejcannon · 2023-07-11T20:42:58Z

src/python/pants_release/release.py

-# TODO: We should be using `./pants package` and `pex_binary` for this...If Pants is lacking in
-#  capabilities, we should improve Pants. When porting, using `runtime_package_dependencies` to do
-#  the validation.


I wrote this todo, so I'm extra psyched that it's now done!

To be extremely fair, I don't think we can technically build a "universal" PEX still, but also that's hand wavy anyways

thejcannon · 2023-07-13T00:07:01Z

This looks good to review 👍

src/python/pants_release/generate_github_workflows.py

src/python/pants_release/release.py

thejcannon · 2023-07-14T14:03:12Z

(FYI @benjyw @huonw @Eric-Arellano)

I'm going to work with @kaos to give this a deadline of before-he-starts-releasing so we can let this see some runtime, and gives us a chance to stabilize any issues. If anything goes awry we can always run with the code prior to the change, or revert.

We're also not cherry-picking so that we do get this time to stabilize, and so that we can have one consistent "start version" for the change (even though aspects might be uploaded to older releases)

Feel free to review post-mortem and I'll do a follow-up PR.

benjyw

Excellent!

benjyw · 2023-07-14T20:46:58Z

docs/markdown/Contributions/releases/release-process.md

@@ -196,22 +196,8 @@ Then, run:

 This will tag the release with your PGP key, and push the tag to origin, which will kick off a [`Release` job](https://github.com/pantsbuild/pants/actions/workflows/release.yaml) to build the wheels and publish them to PyPI.

-Step 4: Release a Pants PEX


All this red makes me happy!

benjyw · 2023-07-14T20:48:08Z

src/python/pants/BUILD

+pex_binary(
+    name="pants-pex",
+    dependencies=[":pants-packaged"],
+    script="pants",
+    execution_mode="venv",
+    shebang="/usr/bin/env python",
+    strip_pex_env=False,
+    layout="zipapp",
+)


I believe that you understand correctly.

benjyw · 2023-07-14T20:51:22Z

src/python/pants_release/release.py

-# TODO: We should be using `./pants package` and `pex_binary` for this...If Pants is lacking in
-#  capabilities, we should improve Pants. When porting, using `runtime_package_dependencies` to do
-#  the validation.


I wrote this todo, so I'm extra psyched that it's now done!

This updates our docs to refer to the new release process, using GitHub Releases, instead of PyPI. The `pantsbuild.pants` package becomes somewhat of an implementation detail too, so I've adjusted these too. Also, the docs previously referred to the universal PEX we published to GitHub Releases which we no longer do (#19450), and thus references to it are removed too. --------- Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com> Co-authored-by: Andreas Stenius <andreas.stenius@imanage.com>

…ick of #19924) (#19930) This updates our docs to refer to the new release process, using GitHub Releases, instead of PyPI. The `pantsbuild.pants` package becomes somewhat of an implementation detail too, so I've adjusted these too. Also, the docs previously referred to the universal PEX we published to GitHub Releases which we no longer do (#19450), and thus references to it are removed too. Co-authored-by: Huon Wilson <huon@exoflare.io> Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com> Co-authored-by: Andreas Stenius <andreas.stenius@imanage.com>

This updates our docs to refer to the new release process, using GitHub Releases, instead of PyPI. The `pantsbuild.pants` package becomes somewhat of an implementation detail too, so I've adjusted these too. Also, the docs previously referred to the universal PEX we published to GitHub Releases which we no longer do (#19450), and thus references to it are removed too. --------- Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com> Co-authored-by: Andreas Stenius <andreas.stenius@imanage.com>

…ick of #19924) (#19936) This updates our docs to refer to the new release process, using GitHub Releases, instead of PyPI. The `pantsbuild.pants` package becomes somewhat of an implementation detail too, so I've adjusted these too. Also, the docs previously referred to the universal PEX we published to GitHub Releases which we no longer do (#19450), and thus references to it are removed too. Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com> Co-authored-by: Andreas Stenius <andreas.stenius@imanage.com>

thejcannon added 2 commits July 11, 2023 15:29

upload local PEXs to release

1c51513

some tweaks

e37a639

thejcannon added the category:new feature label Jul 11, 2023

thejcannon requested review from stuhood, benjyw, huonw and Eric-Arellano July 11, 2023 20:39

thejcannon commented Jul 11, 2023

View reviewed changes

thejcannon mentioned this pull request Jul 12, 2023

Switch scie-pants to use Pants PEX, when supported pantsbuild/scie-pants#226

Merged

thejcannon added the needs-cherrypick label Jul 12, 2023

thejcannon added this to the 2.15.x milestone Jul 12, 2023

thejcannon removed the needs-cherrypick label Jul 12, 2023

thejcannon removed this from the 2.15.x milestone Jul 12, 2023

thejcannon marked this pull request as ready for review July 13, 2023 00:01

Merge branch 'main' into jcannon/uploadPEXs

176da6e

huonw reviewed Jul 13, 2023

View reviewed changes

src/python/pants_release/generate_github_workflows.py Show resolved Hide resolved

src/python/pants_release/release.py Show resolved Hide resolved

reorder

cd1e8a5

thejcannon mentioned this pull request Jul 13, 2023

initial draft of What's New for v2.17.x #19168

Merged

thejcannon requested a review from huonw July 13, 2023 01:54

kaos approved these changes Jul 14, 2023

View reviewed changes

benjyw approved these changes Jul 14, 2023

View reviewed changes

kaos merged commit de1a361 into main Jul 14, 2023
24 checks passed

kaos deleted the jcannon/uploadPEXs branch July 14, 2023 20:54

huonw mentioned this pull request Sep 22, 2023

Remove references to PyPI/pantsbuild.pants wheels in docs #19924

Merged

WorkerPants mentioned this pull request Sep 24, 2023

Remove references to PyPI/pantsbuild.pants wheels in docs (Cherry-pick of #19924) #19930

Merged

huonw mentioned this pull request Sep 24, 2023

Remove references to PyPI/pantsbuild.pants wheels in docs (Cherry-pick of #19924) #19936

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upload local Pants PEXs to release #19450

Upload local Pants PEXs to release #19450

thejcannon commented Jul 11, 2023 •

edited

Loading

thejcannon commented Jul 11, 2023

thejcannon Jul 11, 2023

thejcannon Jul 11, 2023

benjyw Jul 14, 2023

thejcannon Jul 11, 2023

thejcannon Jul 11, 2023

benjyw Jul 14, 2023

thejcannon Jul 14, 2023

thejcannon commented Jul 13, 2023

thejcannon commented Jul 14, 2023 •

edited

Loading

benjyw left a comment

benjyw Jul 14, 2023

benjyw Jul 14, 2023

benjyw Jul 14, 2023

		@@ -196,22 +196,8 @@ Then, run:

		This will tag the release with your PGP key, and push the tag to origin, which will kick off a [`Release` job](https://github.com/pantsbuild/pants/actions/workflows/release.yaml) to build the wheels and publish them to PyPI.

		Step 4: Release a Pants PEX

Upload local Pants PEXs to release #19450

Upload local Pants PEXs to release #19450

Conversation

thejcannon commented Jul 11, 2023 • edited Loading

thejcannon commented Jul 11, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thejcannon commented Jul 13, 2023

thejcannon commented Jul 14, 2023 • edited Loading

benjyw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thejcannon commented Jul 11, 2023 •

edited

Loading

thejcannon commented Jul 14, 2023 •

edited

Loading