Infra Fixes (#1)

## Summary Various infrastructure fixes for the package. ### Why? Gotta get CICD working. ### How? - Fix typo with OS CICD workflow. - Fix read the docs config, using new docs.txt requirements file. - Move test/test.sh to ./nox.sh to reflect its many uses. - Install github cli in dockerfile. - Use hatch-requirements-txt to dynamically load dependencies. - Pin dependencies for each nox session using new update_requirements session. - Omit protobuf autogen code from coverage tests. ## Checklist Most checks are automated, but a few aren't, so make sure to go through and tick them off, even if they don't apply. This checklist is here to help, not deter you. Remember, "Slow is smooth, and smooth is fast". - [X] **Unit tests** - Every input should have a test for it. - Every potential raised exception should have a test ensuring it is raised. - [X] **Documentation** - New functions/classes/etc. must be added to `docs/api.rst`. - Changed/added classes/methods/functions have appropriate `versionadded`, `versionchanged`, or `deprecated` [directives](http://www.sphinx-doc.org/en/stable/markup/para.html#directive-versionadded). - The appropriate entry in `CHANGELOG.md` has been included in the "Unreleased" section, i.e. "Added", "Changed", "Deprecated", "Removed", "Fixed", or "Security". - [X] **Future work** - Future work should be documented in the contributor guide, i.e. `.github/CONTRIBUTING.md`. If you have any questions not answered by a quick readthrough of the [contributor guide](https://pysparkplug.mattefay.com/en/latest/contributor_guide.html), add them to this PR and submit it.
matteosox · Aug 14, 2023 · b944311 · b944311
1 parent 80a22e6
commit b944311
Show file tree

Hide file tree

Showing 29 changed files with 632 additions and 94 deletions.
diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
@@ -6,67 +6,87 @@ We use Docker as a clean, reproducible development environment within which to b
 
 ## Tests
 
-_TL;DR: Run `test/test.sh` to run the full suite of tests._
+_TL;DR: Run `./nox.sh` to run the full suite of tests._
 
 ### Black Code Formatting
 
-_TL;DR: Run `test/test.sh -s black` to test your code's formatting._
+_TL;DR: Run `./nox.sh -s black` to test your code's formatting._
 
-We use [Black](https://black.readthedocs.io/en/stable/index.html) for code formatting. To format your code, run `test/test.sh -s fix` to get all your spaces in a row. Black configuration can be found in the `pyproject.toml` file at the root of the repo.
+We use [Black](https://black.readthedocs.io/en/stable/index.html) for code formatting. To format your code, run `./nox.sh -s fix` to get all your spaces in a row. Black configuration can be found in the `pyproject.toml` file at the root of the repo.
 
 ### isort Import Ordering
 
-_TL;DR: Run `test/test.sh -s isort` to test your code's imports._
+_TL;DR: Run `./nox.sh -s isort` to test your code's imports._
 
-For import ordering, we use [isort](https://pycqa.github.io/isort/). To get imports ordered correctly, run `test/test.sh -s fix`. isort configuration can be found in the `pyproject.toml` file at the root of the repo.
+For import ordering, we use [isort](https://pycqa.github.io/isort/). To get imports ordered correctly, run `./nox.sh -s fix`. isort configuration can be found in the `pyproject.toml` file at the root of the repo.
 
 ### Pylint Code Linting
 
-_TL;DR: Run `test/test.sh -s pylint` to lint your code._
+_TL;DR: Run `./nox.sh -s pylint` to lint your code._
 
-We use [Pyint](https://pylint.pycqa.org/en/latest/) for Python linting (h/t Itamar Turner-Trauring from his site [pythonspeed](https://pythonspeed.com/articles/pylint/) for inspiration). To lint your code, run `test/test.sh -s pylint`. In addition to showing any linting errors, it will also print out a report. Pylint configuration can be found in the `pylintrc` file at the root of the repo.
+We use [Pyint](https://pylint.pycqa.org/en/latest/) for Python linting (h/t Itamar Turner-Trauring from his site [pythonspeed](https://pythonspeed.com/articles/pylint/) for inspiration). To lint your code, run `./nox.sh -s pylint`. In addition to showing any linting errors, it will also print out a report. Pylint configuration can be found in the `pylintrc` file at the root of the repo.
 
 Pylint is setup to lint the `src`, `test/unit_tests` and `docs` directories, along with `noxfile.py`. To add more modules or packages for linting, edit the `pylint` test found in `noxfile.py`.
 
 ### Mypy Static Type Checking
 
-_TL;DR: Run `test/test.sh -s mypy` to type check your code._
+_TL;DR: Run `./nox.sh -s mypy` to type check your code._
 
-We use [Mypy](https://mypy.readthedocs.io/en/stable/) for static type checking. To type check your code, run `test/test.sh -s mypy`. Mypy configuration can be found in the `pyproject.toml` file at the root of the repo.
+We use [Mypy](https://mypy.readthedocs.io/en/stable/) for static type checking. To type check your code, run `./nox.sh -s mypy`. Mypy configuration can be found in the `pyproject.toml` file at the root of the repo.
 
 Mypy is setup to run on the `src` and`test/unit_tests`, along with `noxfile.py` and `docs/linkcode.py`. To add more modules or packages for type checking, edit the `mypy` test found in `noxfile.py`.
 
 ### Unit Tests
 
-_TL;DR: Run `test/test.sh -s unit_tests-3.10 -- fast` to unit test your code quickly._
+_TL;DR: Run `./nox.sh -s unit_tests-3.10 -- fast` to unit test your code quickly._
 
-While we use [`unittest`](https://docs.python.org/3/library/unittest.html) to write unit tests, we use [`pytest`](https://docs.pytest.org/) for running them. To unit test your code, run `test/test.sh -s unit_tests-3.10 -- fast`. This will run unit tests in Python 3.10 only, without any coverage reporting overhead. To run the tests across all supported versions of Python, run `test/test.sh -s unit_tests`, which will also generate coverage reports which can be aggregated using `test/test.sh -s coverage`.
+While we use [`unittest`](https://docs.python.org/3/library/unittest.html) to write unit tests, we use [`pytest`](https://docs.pytest.org/) for running them. To unit test your code, run `./nox.sh -s unit_tests-3.10 -- fast`. This will run unit tests in Python 3.10 only, without any coverage reporting overhead. To run the tests across all supported versions of Python, run `./nox.sh -s unit_tests`, which will also generate coverage reports which can be aggregated using `./nox.sh -s coverage`.
 
 `pytest` is setup to discover tests in the `test/unit_tests` directory. All test files must match the pattern `test*.py`. `pytest` configuration can be found in the `pyproject.toml` file at the root of the repo. To add more directories for unit test discovery, edit the `testpaths` configuration option.
 
 ### Test Coverage
 
-_TL;DR: Run `test/test.sh -s coverage` after running the unit tests with coverage to test the coverage of the unit test suite._
+_TL;DR: Run `./nox.sh -s coverage` after running the unit tests with coverage to test the coverage of the unit test suite._
 
 We use [Coverage.py](https://coverage.readthedocs.io/en/coverage-5.5/) to test the coverage of the unit test suite. This will print any coverage gaps from the full test suite. Coverage.py configuration can be found in the `pyproject.toml` file at the root of the repo.
 
 ### Documentation Tests
 
-_TL;DR: Run `test/test.sh -s docs` to build and test the documentation._
+_TL;DR: Run `./nox.sh -s docs` to build and test the documentation._
 
 See [below](#documentation) for more info on the documentation build process. In addition to building the documentation, the `test/docs.sh` shell script uses Sphinx's [`doctest`](https://www.sphinx-doc.org/en/master/usage/extensions/doctest.html) builder to ensure the documented output of usage examples is accurate. Note that the `README.md` file's ` ```python` code sections are transformed into `{doctest}` directives by `docs/conf.py` during the documentation build process. This allows the `README.md` to render code with syntax highlighting on Github & [PyPI](https://pypi.org) while still ensuring accuracy using `doctest`.
 
 ### Packaging Tests
 
-_TL;DR: Run `test/test.sh -s packaging` to build and test the package._
+_TL;DR: Run `./nox.sh -s packaging` to build and test the package._
 
-We use [`build`](https://pypa-build.readthedocs.io/en/latest/) to build source distributions and wheels. We then use [`check-wheel-contents`](https://github.com/jwodder/check-wheel-contents) to test for common errors and mistakes found when building Python wheels. Finally, we use [`twine check`](https://twine.readthedocs.io/en/latest/#twine-check) to check whether or not `pysparkplug`'s long description will render correctly on [PyPI](https://pypi.org). To test the package build, run `test/test.sh -s packaging`. While there is no configuration for `build` or `twine`, the configuration for `check-wheel-contents` can be found in the `pyproject.toml` file at the root of the repo.
+We use [`build`](https://pypa-build.readthedocs.io/en/latest/) to build source distributions and wheels. We then use [`check-wheel-contents`](https://github.com/jwodder/check-wheel-contents) to test for common errors and mistakes found when building Python wheels. Finally, we use [`twine check`](https://twine.readthedocs.io/en/latest/#twine-check) to check whether or not `pysparkplug`'s long description will render correctly on [PyPI](https://pypi.org). To test the package build, run `./nox.sh -s packaging`. While there is no configuration for `build` or `twine`, the configuration for `check-wheel-contents` can be found in the `pyproject.toml` file at the root of the repo.
+
+## Requirements
+
+### Package Dependencies
+
+_TL;DR: `pysparkplug`'s dependencies are defined in `requirements/requirements.txt`_
+
+We use the [hatch-requirements-txt](https://github.com/repo-helper/hatch-requirements-txt) Hatch extension to define `pysparkplug`'s dependenices dynamically in a separate file, specifically `requirements/requirements.txt`.
+
+### Nox Session Dependencies
+
+_TL;DR: Run `./nox.sh -s update_requirements` to update the requirements of each nox session._
+
+We use [nox](https://nox.thea.codes/en/stable/) to define developer workflows in Python. Each nox session has its own Python virtual environment and set of pinned requirements associated with it. We want these statically defined so developer workflows are reproducible. To do this, we generate a `requirements/{session_name}.txt` file for each session by running `./nox.sh --session update_requirements`, which uses `pip-compile`.
+
+To control which packages are installed, manually edit `requirements/{session_name}.in`. This gives us both a flexible way to describe dependencies while still achieving reproducible builds. Inspired by [this](https://hynek.me/articles/python-app-deps-2018/) and [this](https://pythonspeed.com/articles/pipenv-docker/).
+
+### Note on Hashes
+
+While using [hashes](https://pip.pypa.io/en/stable/cli/pip_install/#hash-checking-mode) would be nice, different platforms, e.g. Apple's ARM vs Intel's x86, sometimes require different wheels with different hashes. This is true despite ensuring a consistent Linux OS in Docker sadly. In the spirit of enabling a diverse ecosystem of developers with different machines, I've kept hashing off.
 
 ## Documentation
 
-_TL;DR: To build and test the documentation, run `test/test.sh -s docs`._
+_TL;DR: To build and test the documentation, run `./nox.sh -s docs`._
 
-We use [Sphinx](https://www.sphinx-doc.org/en/master/index.html) for documentation site generation. To build the documentation, run `test/test.sh -s docs`. To view it, open `docs/build/html/index.html` in your browser.
+We use [Sphinx](https://www.sphinx-doc.org/en/master/index.html) for documentation site generation. To build the documentation, run `./nox.sh -s docs`. To view it, open `docs/build/html/index.html` in your browser.
 
 Sphinx configuration can be found in `docs/conf.py`. It is setup to generate pages based on what it finds in the `toctree` directive in `docs/index.md`. To add new pages, add them to the table of contents with that directive.
 

diff --git a/.github/workflows/cicd.yaml b/.github/workflows/cicd.yaml
@@ -19,17 +19,17 @@ jobs:
           fetch-depth: 0
 
       - name: Test
-        run: test/test.sh
+        run: ./nox.sh
 
       - name: Create draft Github release
         if: ${{ github.ref == 'refs/heads/main' }}
-        run: test/test.sh -s draft_release
+        run: ./nox.sh -s draft_release
         env:
           GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Publish package to testpypi
         if: ${{ github.ref == 'refs/heads/main' }}
-        run: test/test.sh -s publish -- testpypi
+        run: ./nox.sh -s publish -- testpypi
         env:
           TWINE_USERNAME: __token__
           TWINE_PASSWORD: ${{ secrets.TESTPYPI_TOKEN }}
@@ -60,16 +60,14 @@ jobs:
         pip install coverage[toml] nox==2023.04.22
 
     - name: Run unit tests
-      run: nox --session unit_tests-{{ matrix.python-version }}
+      run: nox --session unit_tests-${{ matrix.python-version }}
 
     - name: Combine coverage reports
       run: |
         coverage combine
         coverage xml --fail-under 0
 
-    - name: Upload to Codecov
-      uses: codecov/codecov-action@v2
-      with:
-        env_vars: OS,PYTHON
-        fail_ci_if_error: true
-        verbose: true
+    - name: Upload coverage reports to Codecov
+      uses: codecov/codecov-action@v3
+      env:
+        CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
diff --git a/.github/workflows/publish.yaml b/.github/workflows/publish.yaml
@@ -14,7 +14,7 @@ jobs:
         uses: actions/checkout@v2
 
       - name: Publish package to pypi
-        run: test/test.sh -s publish -- pypi
+        run: ./nox.sh -s publish -- pypi
         env:
           TWINE_USERNAME: __token__
           TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
diff --git a/.readthedocs.yaml b/.readthedocs.yaml
@@ -14,14 +14,7 @@ python:
   install:
     - method: pip
       path: .
-    - requirements:
-      - furo
-      - myst-parser
-      - packaging
-      - sphinx
-      - sphinx-copybutton
-      - sphinx-notfound-page
-      - sphinxext-opengraph
+    - requirements: requirements/docs.txt
 
 # Build documentation in the docs/source directory with Sphinx
 sphinx:

diff --git a/Dockerfile b/Dockerfile
@@ -22,7 +22,15 @@ RUN --mount=type=cache,target=/var/cache/apt \
     apt-get --yes install --no-install-recommends \
     python3.8 python3.8-distutils \
     python3.9 python3.9-distutils python3.10 python3.10-venv \
-    python3.11 git tini
+    python3.11 git tini curl && \
+    # Install Github CLI
+    curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | \
+    dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg && \
+    chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg && \
+    echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | \
+    tee /etc/apt/sources.list.d/github-cli.list > /dev/null && \
+    apt-get update && \
+    apt-get --yes install --no-install-recommends gh
 
 # Create and activate virtual environment
 ENV VIRTUAL_ENV="/root/.venv"

diff --git a/test/test.sh → nox.sh b/test/test.sh → nox.sh
@@ -2,10 +2,14 @@
 set -o errexit -o nounset -o pipefail
 IFS=$'\n\t'
 
-REPO_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"/..
+REPO_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 cd "$REPO_DIR"
 
-echo "Running tests"
+if [[ "$#" -gt 0 ]]; then
+    echo "Running" "$@" "in nox"
+else
+    echo "Running test suite"
+fi
 
 docker compose run --rm cicd nox "$@"
 

diff --git a/noxfile.py b/noxfile.py
@@ -1,5 +1,6 @@
 """Test/developer workflow automation"""
 
+import pathlib
 from typing import cast
 
 import nox
@@ -28,30 +29,30 @@ def fix(session: nox.Session) -> None:
     session.notify("isort", ["fix"])
 
 
-@nox.session
+@nox.session()
 def black(session: nox.Session) -> None:
     """Black Python formatting tool"""
-    session.install("black")
+    session.install("--requirement", "requirements/black.txt")
     if session.posargs and session.posargs[0] == "fix":
         session.run("black", ".")
     else:
         session.run("black", "--diff", "--check", ".")
 
 
-@nox.session
+@nox.session()
 def isort(session: nox.Session) -> None:
     """ISort Python import formatting tool"""
-    session.install("isort")
+    session.install("--requirement", "requirements/isort.txt")
     if session.posargs and session.posargs[0] == "fix":
         session.run("isort", ".")
     else:
         session.run("isort", "--check-only", ".")
 
 
-@nox.session
+@nox.session()
 def pylint(session: nox.Session) -> None:
     """Pylint Python linting tool"""
-    session.install("pylint", ".", "nox", "packaging")
+    session.install("--requirement", "requirements/pylint.txt", ".")
     session.run(
         "pylint",
         "src",
@@ -61,21 +62,12 @@ def pylint(session: nox.Session) -> None:
     )
 
 
-@nox.session
+@nox.session()
 def mypy(session: nox.Session) -> None:
     """Mypy Python static type checker"""
-    session.install(
-        "mypy",
-        ".",
-        "nox",
-        "packaging",
-        "types-protobuf",
-        "types-paho-mqtt",
-    )
+    session.install("--requirement", "requirements/mypy.txt", ".")
     session.run(
         "mypy",
-        "--install-types",
-        "--non-interactive",
         "src",
         "noxfile.py",
         "test/unit_tests",
@@ -86,12 +78,7 @@ def mypy(session: nox.Session) -> None:
 @nox.session(python=["3.8", "3.9", "3.10", "3.11"])
 def unit_tests(session: nox.Session) -> None:
     """Unit test suite run with coverage tracking"""
-    session.install(
-        ".",
-        "coverage[toml]",
-        "pytest",
-        "packaging",
-    )
+    session.install("--requirement", "requirements/unit_tests.txt", ".")
     if session.posargs and session.posargs[0] == "fast":
         session.run("python", "-m", "pytest")
     else:
@@ -101,7 +88,7 @@ def unit_tests(session: nox.Session) -> None:
 @nox.session()
 def coverage(session: nox.Session) -> None:
     """Report on coverage tracking"""
-    session.install("coverage[toml]")
+    session.install("--requirement", "requirements/coverage.txt")
     try:
         session.run("coverage", "combine")
         session.run("coverage", "report")
@@ -112,16 +99,7 @@ def coverage(session: nox.Session) -> None:
 @nox.session()
 def docs(session: nox.Session) -> None:
     """Generate and test documentation"""
-    session.install(
-        ".",
-        "furo",
-        "myst-parser",
-        "packaging",
-        "sphinx",
-        "sphinx-copybutton",
-        "sphinx-notfound-page",
-        "sphinxext-opengraph",
-    )
+    session.install("--requirement", "requirements/docs.txt", ".")
     session.run(
         "sphinx-build",
         "-T",
@@ -148,10 +126,10 @@ def docs(session: nox.Session) -> None:
     )
 
 
-@nox.session
+@nox.session()
 def packaging(session: nox.Session) -> None:
     """Build and test packaging"""
-    session.install("check-wheel-contents", "twine", "build")
+    session.install("--requirement", "requirements/packaging.txt", ".")
     try:
         session.run("python", "-m", "build")
         session.run("check-wheel-contents", "dist")
@@ -160,10 +138,10 @@ def packaging(session: nox.Session) -> None:
         session.run("rm", "-rf", "dist", external=True)
 
 
-@nox.session
+@nox.session()
 def draft_release(session: nox.Session) -> None:
     """Create a draft Github Release"""
-    session.install(".")
+    session.install("--requirement", "requirements/draft_release.txt", ".")
     version_str = _version(session)
     version_obj = Version(version_str)
     if not version_obj.is_devrelease:
@@ -184,20 +162,12 @@ def draft_release(session: nox.Session) -> None:
     session.run(*cmd, external=True)
 
 
-@nox.session
+@nox.session()
 def publish(session: nox.Session) -> None:
     """Publish package to PyPI and upload build artifacts to Github Release"""
-    session.install(".", "twine", "build")
+    session.install("--requirement", "requirements/publish.txt", ".")
     version_str = _version(session)
     version_obj = Version(version_str)
-    if (
-        version_obj.is_devrelease
-        or version_obj.is_postrelease
-        or version_obj.local is not None
-    ):
-        raise ValueError(
-            f"Package version {version_str} should not be a post or dev release"
-        )
     repository = session.posargs[0]
     if repository == "pypi":
         if (
@@ -257,3 +227,30 @@ def _get_notes() -> str:
             changes_lines.append(line)
 
     return "".join(changes_lines)
+
+
+@nox.session()
+def update_requirements(session: nox.Session) -> None:
+    """Pin requirements files for nox environments.
+
+    NOTE: "'pip-compile' should be run from the same virtual environment as your
+    project so conditional dependencies that require a specific Python
+    version, or other environment markers, resolve relative to your project's
+    environment." This does not do that, and may break in the future as a
+    result, especially the various unit test environments, with their
+    different Python versions.
+    """
+    session.install("pip-tools")
+    for path in pathlib.Path(pathlib.Path.cwd(), "requirements").iterdir():
+        if path.suffix == ".in":
+            session.run(
+                "pip-compile",
+                "--allow-unsafe",
+                "--resolver=backtracking",
+                "--upgrade",
+                "--verbose",
+                "--output-file",
+                f"requirements/{path.stem}.txt",
+                str(path),
+                env={"CUSTOM_COMPILE_COMMAND": "./nox.sh -s update_requirements"},
+            )