[FEATURE] Branching strategy Phase 3 - PyPI and CodeArtifact publish workflows by lowlydba · Pull Request #638 · OvertureMaps/schema

John McCall (lowlydba) · 2026-08-05T16:55:57Z

Summary

Phase 3 of the branching strategy (#509): the publish side of the pipeline Phase 2.B (#557) computed versions for. Internal builds go to CodeArtifact on every push to main; human-cut releases go to public PyPI.

What lands

Internal builds → CodeArtifact

main-publish.yaml, on push to main: detect-affected-packages (new composite action, using tj-actions/changed-files pinned by commit SHA rather than a hand-rolled git diff --name-only, for correct filename quoting and its dir_names-to-package-name folding) diffs the push against packages/** to find packages touched without a version bump (bumped packages release via the path below instead). Each affected package gets a .postN+main.<sha> version stamped in via compute-version, built, and published to CodeArtifact.

The same workflow doubles as a build-only smoke test on PRs touching the composite actions or itself: it discovers every package unconditionally instead of diffing (a PR touching .github/actions/** never touches packages/**, so the diff would always return zero and skip the job it's meant to exercise), builds, and stops short of publishing. Replaces compute-versions-dry-run.yaml.

Releases → PyPI

release-publish.yaml, on release: published: parses <package>-v<version> off the tag, builds, verifies the wheel matches (and, for overture-schema-pyspark, that the generated expressions are in it), and publishes via pypa/gh-action-pypi-publish with OIDC trusted publishing and attestations. The version-bump PR review is the approval gate; no separate environment reviewer step, one human approval per release is enough.

It also takes a workflow_dispatch (package, target: test-pypi|pypi) that always publishes a synthetic <on-disk-version>.dev0, never the real version. This exists to prime a package's PyPI Trusted Publisher from "pending" to "normal" ahead of its real release, working around PyPI's pending-publisher rate limit (~3 at a time, independently on pypi.org and test.pypi.org, see #653). Real-PyPI dispatch reuses the same pypi-release-<package> environment as the automated path; workflow_dispatch already requires repo write access to trigger, which is gate enough for a disposable .dev0. Temporary: #688 tracks removing the pypi target once every package is primed for the v2.0 launch.

release-trigger.yaml creates releases with an overture-release-publisher app installation token instead of GITHUB_TOKEN: GitHub doesn't fire release: published for releases created with the default token, so release-publish.yaml would never see them. Token scoped to contents: write only. App provisioning was tracked on #637 (now done).

A package's first version counts as a bump too: detect_version_bumps.py no longer skips before: null entries, so a brand-new package flows through release-trigger to PyPI on its first version, same as any other bump. detect_affected_packages.py's bumped set was widened to match (any package with a non-null after version), so the same push doesn't also queue it for an internal .postN build.

Mechanical follow-through: diff-package-versions action, deleted workflows, docs

package_versions.py diff was inlined at three call sites (detect-version-bumps, the new detect-affected-packages, and the pre-existing reusable-check-python-package-versions.yaml), each reimplementing the temp-file-then-$GITHUB_OUTPUT plumbing slightly differently. Pulled it into its own diff-package-versions action exposing count/diff outputs; the three consumers now just pipe diff into their own filter logic instead of each knowing the script's path and output convention.

package_versions.py itself moved from .github/workflows/scripts/ into diff-package-versions/, its only remaining caller, so every action in the repo now carries its own script(s) instead of reaching outside its directory.

Deletes compute-versions-dry-run.yaml and publish-python-packages.yaml, both superseded by the two workflows above.

docs/versioning.md gets a workflow-name reference table and the manual-dispatch priming mechanics under the release walkthrough. CONTRIBUTING.md drops the "rolling out in phases" banner (this is the last phase) and corrects two release-flow descriptions that predated the approval gate.

Testing

detect-affected-packages's exclusion logic is tested locally against real repo commit ranges plus synthetic bumped/removed/new-package cases, including through the diff-package-versions wiring and after the script relocation. The uv version <post-release-string> --package <pkg> --frozen --dry-run stamping technique is confirmed to accept PEP 440 local-label and dev-release strings. zizmor is clean on every touched/new workflow and action file. The workflows themselves can't run end to end until the external prerequisites below land.

Still open (tracked on #653, all external/manual)

PyPI Trusted Publisher config per package, batched to work around the pending-publisher rate limit
pypi-release-<package>/test-pypi-<package> GitHub Environments
a resolver smoke test against real CodeArtifact and a real end-to-end PyPI release, once the above are live

vnext dev builds to CodeArtifact moved out of this phase entirely, tracked on Phase 4 (#510) instead: it's blocked on a CodeArtifact repo migration (ops-team#299) with no committed timeline, so Phase 3 ships without waiting on it.

Closes #509
Contributes to #653, #688

…ish workflows Adds the publish side of the versioning pipeline built in Phase 2.B: - main-publish.yaml: on push to main, diffs changed packages/** files (via the new detect-affected-packages action) to find packages touched without a version bump, stamps a .postN+main.<sha> build via compute-version, and publishes to CodeArtifact. Also runs as a build-only smoke test on PRs touching the composite actions or itself, replacing compute-versions-dry-run.yaml. - elease-publish.yaml: on elease: published, parses <package>-v<version> from the tag, builds, and publishes to PyPI via `pypa/gh-action-pypi-publish` (OIDC trusted publishing + attestations), gated by the pypi-release environment's required reviewers. - elease-trigger.yaml: releases now get created with an overture-release-publisher app installation token instead of GITHUB_TOKEN, since GITHUB_TOKEN-created releases don't fire elease: published for other workflows to pick up. App provisioning is tracked separately in #637 and has to happen before this path works. Deletes compute-versions-dry-run.yaml and publish-python-packages.yaml, both superseded by the two workflows above. docs/versioning.md gets a workflow-name reference table and the pypi-release approval-gate mechanics. Still open, all external/manual, tracked on #509: - provisioning overture-release-publisher (#637) - PyPI Trusted Publisher config per package - pypi-release environment + reviewers - p3-dev-builds-ca, blocked on ops-team#299 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>

github-actions · 2026-08-05T16:57:54Z

🗺️ Schema reference docs preview is live!


🌍 Preview	https://staging.overturemaps.org/schema/pr/638/schema/index.html
🕐 Updated	Aug 24, 2026 15:55 UTC
📝 Commit	2e7332d
🔧 env SCHEMA_PREVIEW	`true`

Note

♻️ This preview updates automatically with each push to this PR.

Replaces the hand-rolled `git diff --name-only` + path-splitting with tj-actions/changed-files' dir_names output, pinned by commit SHA. Fixes a real gap in the old parser: git diff --name-only quotes non-ASCII/unusual filenames, which naive line.split("/") didn't account for. Uses �ll_modified_files (ACMRD) rather than �ll_changed_files (ACMR) so file deletions still count as a package change, matching the old diff's behavior. detect_affected_packages.py drops its subprocess/git plumbing entirely and just reads the directory list from CHANGED_DIRS; the bump/removed exclusion logic (a three-way set difference) stays in Python rather than jq/bash, that's a different complexity class than this repo's existing jq usage in enforce-change-type-label.yaml. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>

…scripts run back to back The action wasn't wrong, just awkward to read: python, jump to a JS action, jump back to python. Steps 1 (version diff) and 2 (changed dirs) don't depend on each other, only step 3 does, so nothing stops them running adjacent. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>

package_versions.py diff was inlined at three call sites (detect-version-bumps, detect-affected-packages, and eusable-check-python-package-versions.yaml), each reimplementing the temp-file-then-$GITHUB_OUTPUT plumbing slightly differently (one even wrote to a different temp path and used heredoc-style output). diff-package-versions is now the one place that knows how to run the script and expose it as count/diff outputs. The three consumers just pipe steps.diff.outputs.diff into their own filter logic: detect_version_bumps.py and detect_affected_packages.py are unchanged, they already read JSON from stdin. eusable-check-python-package-versions.yaml's CodeArtifact existence check now reads the diff from an env var instead of a temp file; its changed_packages output is compact JSON now instead of pretty-printed, no consumer depends on the formatting. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>

Last remaining dangling script: package_versions.py lived in .github/workflows/scripts/ and was reached into by relative path. Every other action already keeps its script(s) alongside its own action.yml (detect-version-bumps, detect-affected-packages, create-package-release); this was the one holdout, and after the last refactor diff-package-versions is its only remaining caller anyway. Moved it in, switched the reference to ${GITHUB_ACTION_PATH}, and removed the now-empty scripts/ directory. No behavior change: git commands inside the script still run with the repo root as CWD, only the path used to invoke it changed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>

…lease flow Phase 3 was the last phase (#509); Phase 4 got folded into it instead of staying a separate doc-polish pass (see #490). Drops the "rolling out in phases" banner pointing at the tracking issue, since the flow it describes is now fully implemented rather than in progress. Also corrects two release-flow descriptions that predated the pypi-release approval gate this phase added: a release no longer lands on PyPI "immediately", it starts a maintainer-gated publish first. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>

push never fires for fork PRs (only same-repo branch pushes), so a required check relying solely on push would never run for external contributors, a required status check that can never be satisfied by the PRs GitHub Actions actually needs to gate. Add pull_request with the same path filters; push stays for post-merge validation on main. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>

push never fires for fork PRs, so a required-status-check rule depending on test-schema.yaml would deadlock every fork PR. Adds a pull_request trigger alongside push, and switches the concurrency group's PR fallback to github.event.pull_request.number so per-PR runs don't collide. Cherry-picked from #638's test-schema.yaml hunk, that PR's scope is the unrelated Phase 3 publish workflows and won't merge soon, this piece can't wait on it since it's needed to verify the Build job actually triggers and reports on a fork PR per #647's acceptance criteria. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>

release-trigger.yaml assumes the narrow gha-releaser-secrets-reader OIDC role and fetches the PEM from omf-github-terraform/releaser/pem instead of reading GHA repo secrets, matching the safe-settings and project-manager app pattern. Client ID is inlined (not sensitive) instead of a secret. Companion Terraform PR wires the role and Secrets Manager entry in omf-github-terraform. Fixes #637. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>

…ion (#648) * [CHORE](ci) Add all-Python-checks gate, resolve Build job-name collision check-python-code.yaml's check matrix produces one status per cell with no aggregator, so a required-status-check rule has to track six names and misses any new Python version until updated. test-schema.yaml and schema-pr-preview.yml both name their job Build, ambiguous once test-schema.yaml gets a pull_request trigger and needs pinning itself. Adds an all-python-checks-pass job to check-python-code.yaml (needs: [check], if: always()) using lowlydba/are-we-good to roll the matrix's per-cell results into one pass/fail. Renames schema-pr-preview.yml's build job from Build to Build preview, leaving test-schema.yaml's Build as the stable name #646 already plans to pin. Fixes #647 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org> * [CHORE](ci) Add pull_request trigger to test-schema.yaml push never fires for fork PRs, so a required-status-check rule depending on test-schema.yaml would deadlock every fork PR. Adds a pull_request trigger alongside push, and switches the concurrency group's PR fallback to github.event.pull_request.number so per-PR runs don't collide. Cherry-picked from #638's test-schema.yaml hunk, that PR's scope is the unrelated Phase 3 publish workflows and won't merge soon, this piece can't wait on it since it's needed to verify the Build job actually triggers and reports on a fork PR per #647's acceptance criteria. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org> --------- Signed-off-by: John McCall <john@overturemaps.org>

detect_version_bumps.py no longer skips before:null entries; a brand-new package now flows through release-trigger to PyPI on its first version instead of sitting on CodeArtifact-only .postN builds. detect_affected_packages.py's bumped set is widened to match (any package with a non-null after version) so the same push doesn't also queue an internal build for a package that just got its first release. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>

Signed-off-by: John McCall <john@overturemaps.org>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>

'default' didn't say what it defaulted to. locked names what actually distinguishes it from lowest-direct: it runs against the committed uv.lock. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>

Victor Schappert (vcschapp)

Raised a few minor nittish review comments. Nothing big. I'm excited to merge this, once we do it seems we're in the business of publishing Python packages!

- Downscope main-publish's PR smoke test to the CodeArtifact read-only IAM role instead of the publish role. - Validate the parsed package directory exists for both release-publish arms (release tag and workflow_dispatch), not just workflow_dispatch. - Standardize on the actions/checkout v7.0.1 pin used elsewhere in the repo; main-publish/release-publish had drifted to a mix of v7.0.0/v7.0.1. - Fix wording nit: "in-memory checkout" -> temporary checkout. - detect_affected_packages.py: read version_diff + changed_dirs from a single stdin JSON payload instead of splitting across stdin and an env var, and raise a clear error instead of an IndexError on a malformed changed-directory path. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>

John McCall (lowlydba) · 2026-08-24T14:57:31Z

Copilot resolve the merge conflicts in this pull request

Co-authored-by: lowlydba <16843041+lowlydba@users.noreply.github.com>

Copilot · 2026-08-24T15:02:28Z

Copilot resolve the merge conflicts in this pull request

Resolved by merging origin/main and fixing the conflict state in commit 65bb3bb6 (with a follow-up guard fix in c83e62cc).

… workflow_dispatch PyPI rate-limits pending Trusted Publisher registrations to ~3 at a time, on both pypi.org and test.pypi.org independently (see #653). Priming a publisher past "pending" requires an actual publish, and workflow_dispatch previously only supported Test PyPI, so there was no way to prime real PyPI ahead of a package's first release. workflow_dispatch now takes a target input (test-pypi default, or pypi) and always publishes a synthetic <on-disk-version>.dev0 instead of the real version: valid PEP 440, ignored by default resolvers, and never collides with the eventual human-owned release. Dispatching to real PyPI uses a new pypi-dispatch-<package> environment with its own required-reviewer gate, since it's the one path that publishes to production PyPI without a version-bump PR behind it; pypi-release-<package> (the automated release path) and test-pypi-<package> are unchanged. This is a bootstrapping tool, not a permanent fixture. Once all 12 packages are primed for the v2.0 launch, the pypi target and its dispatch environments should come out. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>

…phase-3' into 509-devops-branching-strategy---phase-3 Signed-off-by: John McCall <john@overturemaps.org>

…ackage>, flag temp blocks for #688 Drops the separate pypi-dispatch-<package> environment: workflow_dispatch already requires repo write access to trigger at all, which is gate enough for a disposable .dev0 priming publish, so a real-PyPI priming dispatch now reuses the same pypi-release-<package> environment the automated release path uses. One environment set per package instead of a third variant. Also marks every pypi-target-specific block with TODO(#688) so the temporary priming path (vs. the permanent test-pypi dispatch) is obvious at the code site, not just in the tracking issue. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>

John McCall (lowlydba) temporarily deployed to staging August 5, 2026 16:57 — with GitHub Actions Inactive

John McCall (lowlydba) temporarily deployed to staging August 5, 2026 17:10 — with GitHub Actions Inactive

John McCall (lowlydba) temporarily deployed to staging August 5, 2026 17:12 — with GitHub Actions Inactive

John McCall (lowlydba) added the automation 🦾 Change/enhance automation of a process label Aug 5, 2026

John McCall (lowlydba) temporarily deployed to staging August 5, 2026 17:21 — with GitHub Actions Inactive

John McCall (lowlydba) temporarily deployed to staging August 5, 2026 17:25 — with GitHub Actions Inactive

John McCall (lowlydba) temporarily deployed to staging August 5, 2026 17:36 — with GitHub Actions Inactive

This was referenced Aug 11, 2026

[Devops] Branching Strategy - Phase 3 #509

Open

[Devops] Branching Strategy - Phase 4 - vnext dev builds to CodeArtifact #510

Open

[Devops] Streamlined branching strategy for schema #490

Open

John McCall (lowlydba) requested review from Seth Fitzsimmons (sethfitz) and Victor Schappert (vcschapp) August 12, 2026 14:18

Merge branch 'main' into 509-devops-branching-strategy---phase-3

1d2c94c

Signed-off-by: John McCall <john@overturemaps.org>

John McCall (lowlydba) temporarily deployed to staging August 12, 2026 14:21 — with GitHub Actions Inactive

John McCall (lowlydba) marked this pull request as ready for review August 12, 2026 15:25

John McCall (lowlydba) added 2 commits August 12, 2026 11:42

Add changelog fragment for pyspark prebuild un-hoisting

f8e3b0e

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>

Expand changelog fragment with CI-only scope note

2f8a302

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>

John McCall (lowlydba) temporarily deployed to staging August 12, 2026 15:45 — with GitHub Actions Inactive

John McCall (lowlydba) temporarily deployed to staging August 12, 2026 15:50 — with GitHub Actions Inactive

Victor Schappert (vcschapp) reviewed Aug 13, 2026

View reviewed changes

John McCall (lowlydba) temporarily deployed to staging August 14, 2026 13:52 — with GitHub Actions Inactive

Merge branch 'main' into 509-devops-branching-strategy---phase-3

3dd502d

John McCall (lowlydba) temporarily deployed to staging August 14, 2026 14:21 — with GitHub Actions Inactive

Victor Schappert (vcschapp) previously approved these changes Aug 19, 2026

View reviewed changes

Seth Fitzsimmons (sethfitz) previously approved these changes Aug 19, 2026

View reviewed changes

Copilot started work on behalf of John McCall (lowlydba) August 24, 2026 14:57 View session

Copilot AI and others added 2 commits August 24, 2026 14:59

Merge origin/main into 509-devops-branching-strategy---phase-3

65bb3bb

Co-authored-by: lowlydba <16843041+lowlydba@users.noreply.github.com>

Fix optional pyspark import guard behavior

c83e62c

Co-authored-by: lowlydba <16843041+lowlydba@users.noreply.github.com>

Copilot AI dismissed stale reviews from Seth Fitzsimmons (sethfitz) and Victor Schappert (vcschapp) via c83e62c August 24, 2026 15:02

Copilot finished work on behalf of John McCall (lowlydba) August 24, 2026 15:02

John McCall (lowlydba) requested review from Seth Fitzsimmons (sethfitz) and Victor Schappert (vcschapp) August 24, 2026 15:07

Copilot AI temporarily deployed to staging August 24, 2026 15:08 Inactive

John McCall (lowlydba) added 2 commits August 24, 2026 11:38

Merge remote-tracking branch 'origin/509-devops-branching-strategy---…

e548436

…phase-3' into 509-devops-branching-strategy---phase-3 Signed-off-by: John McCall <john@overturemaps.org>

John McCall (lowlydba) mentioned this pull request Aug 24, 2026

[Devops] Remove the real-PyPI priming dispatch path once all packages are primed #688

Open

2 tasks

John McCall (lowlydba) temporarily deployed to staging August 24, 2026 15:40 — with GitHub Actions Inactive

John McCall (lowlydba) deployed to staging August 24, 2026 15:54 — with GitHub Actions View deployment

John McCall (lowlydba) mentioned this pull request Aug 24, 2026

[Devops] Configure PyPI Trusted Publisher + pypi-release environment (manual, one-time) #653

Open

9 tasks

Navigation Menu

Uh oh!

[FEATURE] Branching strategy Phase 3 - PyPI and CodeArtifact publish workflows - #638

[FEATURE] Branching strategy Phase 3 - PyPI and CodeArtifact publish workflows#638
John McCall (lowlydba) wants to merge 24 commits into
mainfrom
509-devops-branching-strategy---phase-3

John McCall (lowlydba) commented Aug 5, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 5, 2026 •

edited

Loading

Uh oh!

Victor Schappert (vcschapp) left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

John McCall (lowlydba) commented Aug 24, 2026

Uh oh!

Copilot AI commented Aug 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

John McCall (lowlydba) commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What lands

Internal builds → CodeArtifact

Releases → PyPI

Testing

Still open (tracked on #653, all external/manual)

Uh oh!

github-actions Bot commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🗺️ Schema reference docs preview is live!

Uh oh!

Victor Schappert (vcschapp) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

John McCall (lowlydba) commented Aug 24, 2026

Uh oh!

Copilot AI commented Aug 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

John McCall (lowlydba) commented Aug 5, 2026 •

edited

Loading

github-actions Bot commented Aug 5, 2026 •

edited

Loading