[Devops] Branching Strategy - Phase 3

[lowlydba]

Goal

Build and publish packages to the right destinations using version strings computed by Phase 2. Each workflow has one trigger and one destination. No version logic lives here.

vnext dev builds to CodeArtifact are out of scope here, split to Phase 4 (#510): they're blocked on a CodeArtifact repo that doesn't exist yet (ops-team#299), so this phase can ship and close without waiting on that migration.

Trigger x destination matrix

Trigger	Workflow	Packages	Version	Destination
Push to main, no bump	main-publish.yaml	affected only	<version>.postN+main.<sha>	CodeArtifact
GitHub Release published (manual bump)	release-publish.yaml	released package	<version>	Public PyPI (upstreamed by all internal CA)
Push to vnext	see Phase 4 (#510)	affected only	<version>.postN+vnext.<sha>	⛔ Blocked on ops-team#299

CodeArtifact: domain overture-pypi / account 505071440022 / region us-west-2.

Tasks

Internal builds -> CodeArtifact

• Trigger: push to main with no version bumps.
• Detect affected packages, version via the compute-version action (main context), build + publish to CodeArtifact. Workspace dependency floors are declared statically in each pyproject.toml (dual declaration), so no metadata rewriting is needed at build time.
• Implemented in PR [FEATURE] Branching strategy Phase 3 - PyPI and CodeArtifact publish workflows #638.

Releases -> PyPI

• Trigger: GitHub Release published. release-trigger.yaml creates the release with the overture-release-publisher app's installation token (provisioned via Provision overture-release-publisher GitHub App for Phase 3 release publishing #637) instead of GITHUB_TOKEN, so the release actually fires the release: published event this workflow needs.
• Build the released package at its released version and publish to public PyPI via Trusted Publishing (OIDC) + attestations.
• Human gate: GitHub Environment (pypi-release) approval (required reviewers) on this job.
• Pattern follows the overturemaps-py publish workflow.
• Prereq: PyPI Trusted Publisher OIDC + pypi-release environment configured (manual, one-time; [Devops] Configure PyPI Trusted Publisher + pypi-release environment (manual, one-time) #653).
• Implemented in PR [FEATURE] Branching strategy Phase 3 - PyPI and CodeArtifact publish workflows #638.

Docs

• Update docs/versioning.md with the matrix above and consumer pin guidance.
• Retire compute-versions-dry-run.yaml and the legacy publish-python-packages.yaml once real publish workflows are live.
• Done in PR [FEATURE] Branching strategy Phase 3 - PyPI and CodeArtifact publish workflows #638.

Require the new CI checks

• Now that the publish pipeline and its supporting checks exist, wire them up as required status checks rather than advisory-only. Split into [Devops] Wire schema's CI checks up as required status checks (safe-settings in omf-github-terraform) #646 (ruleset wiring in omf-github-terraform) and [Devops] Stabilize schema CI check names for required-status-check pinning #647 (check-name stability fixes in this repo, a prerequisite for pinning more of [Devops] Wire schema's CI checks up as required status checks (safe-settings in omf-github-terraform) #646's checks).

Consumer guidance

Internal consumers (CDP) pin >=X.Y.Z per package: this resolves the freshest .postN build from CodeArtifact and falls back to the clean release. Never pin >, PEP 440 excludes post-releases from exclusive comparisons. Verified against uv in Phase 2.B.

Definition of done

• Release-event token/dispatch plumbing decided and wired (Provision overture-release-publisher GitHub App for Phase 3 release publishing #637, overture-release-publisher app token in release-trigger.yaml)
• CI checks required on the branch ruleset ([Devops] Wire schema's CI checks up as required status checks (safe-settings in omf-github-terraform) #646, [Devops] Stabilize schema CI check names for required-status-check pinning #647)
• PR [FEATURE] Branching strategy Phase 3 - PyPI and CodeArtifact publish workflows #638 merged and live:
  • main-publish.yaml
  • release-publish.yaml, tested against a real release
  • docs/versioning.md updated; dry-run + legacy publish workflows retired
• PyPI Trusted Publisher OIDC + pypi-release environment configured ([Devops] Configure PyPI Trusted Publisher + pypi-release environment (manual, one-time) #653)
• Resolver smoke test against real CodeArtifact (>= picks .postN)

Original issue body (superseded, kept for history)

Original matrix used +dev.N (vnext) and CI-computed <next-patch> (main), with public PyPI on major/minor bumps only:

Trigger	Version	Destination
Push to vnext	<last-published>+dev.N	CA dev repo
Push to main (no major/minor bump)	<major>.<minor>.<next-patch>	TBD
Major/minor bump on main	<major>.<minor>.0	Public PyPI

The patch-destination decision (Option A: patches to PyPI; Option B: CA only) was superseded by making patch a human-owned bump like any other: deliberate patch bumps release to PyPI, and no-bump merges publish internal-only .postN builds. +dev.N was dropped because local-only labels are not PEP 440 orderable, so >= consumers could never resolve them.

CDP pin guidance was >=<last-published>+dev.0; now plain >=X.Y.Z.

[vcschapp]

Arrived here from #534, via a #wg-schema Slack thread.

What's the best place to get an overall glossary and picture of the process along with reasons for key decisions?

Some questions I have are:

1/ What is a supplemental package?

Bump a supplemental package's <major>/<minor> on main → CA release repo only, never PyPI.

2/ Why do we need, and what is the utility of, having two separate CA repos, one dev and one release?

3/ What is the intention behind publishing every commit to main to the CA release repo?

4/ The only row that indicates Public PyPI, the third one, p3-pypi-publish, is speaking only about overture-schema, but overture-schema depends on a bunch of other packages that need to be in public PyPI, e.g., overture-schema-transportation-theme, overture-schema-common, overture-schema-system, etc. How are these other packages getting in to public PyPI?

5/ It will be common for changes to cut across packages, e.g. a change may be made to the common package and then used in multiple theme packages with the effect then manifesting also in overture-schema. All of these packages will need version changes to stay in sync. How will this work?

6/ The last two rows in the table that indicate "version bump on main" with a version <major>.<minor>.0 suggest to me that a human-generated PR can only produce .0 patch versions, and that only .0 patch versions will go to public PyPI. This isn't likely to be true for typical development: there may be trivial changes, such as bug fixes or docstring updates, that we want to make public via via patch version. How will this work?

[vcschapp]

One thing I meant to mention in the meeting this morning but didn't get a chance to: when doing the auto-publishes to dev CodeArtifact, we'll need to be aware of the dependency graph, which may require cascading version changes.

As an example, one branch of the dependency graph looks like:

overture-schema ← overture-schema-places-theme ← overture-schema-common ← overture-schema-system

If a change is made in overture-schema-common such that it gets a new auto-version, you need to ensure that the version of overture-schema-places-theme depends on the new published version, which necessitates a version bump for overture-schema-places-theme; with a cascading impact down the line to overture-schema.

This may be a non-trivial problem to solve, because the version-controlled pyproject.toml's are referring to the public versioning.

[lowlydba]

Victor Schappert (@vcschapp) I'm going to do some more cleanup, but this issue is updated post today's meeting and hopefully catalogs some of the thinking/decisions.

1/ What is a supplemental package?

Good callout - not a word we should use here. It was being used to refer to essentially non-core packages, but I think after the Schema WG meeting today (5/27) we established we can/should treat all packages the same in the context of versions/releases broadly.

2/ Why do we need, and what is the utility of, having two separate CA repos, one dev and one release?

This was more an artifact of the fact that those CAs will (soon) exist as our general approach for package management, but as we mentioned in todays WG call - not necessarily pertinent for Schema packages. The chart has been updated to remove that target - now we'll only hit Dev CA and PyPI, just need to decide which target receives main patches (i.e. do we release patches often publicly, or only publish major/minor updates publicly that contain lots of patches)

3/ What is the intention behind publishing every commit to main to the CA release repo?

As teased in the previous reply - the initial thought was to align us with true Continuous Deployments. But if there are negative outcomes from this (i.e. we don't want release churn) we could alternatively only make the patches internal, and publish big major/minor changes (see updated top comment)

4/ The only row that indicates Public PyPI, the third one, p3-pypi-publish, is speaking only about overture-schema, but overture-schema depends on a bunch of other packages that need to be in public PyPI, e.g., overture-schema-transportation-theme, overture-schema-common, overture-schema-system, etc. How are these other packages getting in to public PyPI?

Fixed - removed that row, and made the others more generic - we'll talk about packages all together going forward for release/versioning.

5/ It will be common for changes to cut across packages, e.g. a change may be made to the common package and then used in multiple theme packages with the effect then manifesting also in overture-schema. All of these packages will need version changes to stay in sync. How will this work?

See my last comment reply below - we're not pinning dep versions, so this should flow naturally.

6/ The last two rows in the table that indicate "version bump on main" with a version ..0 suggest to me that a human-generated PR can only produce .0 patch versions, and that only .0 patch versions will go to public PyPI. This isn't likely to be true for typical development: there may be trivial changes, such as bug fixes or docstring updates, that we want to make public via via patch version. How will this work?

Depends on the above answers - but we can do whatever mix/match we choose pretty easily.

One thing I meant to mention in the meeting this morning but didn't get a chance to: when doing the auto-publishes to dev CodeArtifact, we'll need to be aware of the dependency graph, which may require cascading version changes.

I actually am not super worried about this for us - primarily because all the non-core package dependencies aren't pinned to versions.

To walk through an example: say someone merges a fix to overture-schema-common on main. CI auto-publishes overture-schema-common 0.1.2. Now a consumer installs overture-schema-places-theme 0.1.1 -- because places-theme declares "overture-schema-common" with no pin, they get 0.1.2 automatically. Same logic flows all the way up to overture-schema.

The only time you'd need to manually bump a downstream is if you're intentionally encoding a floor, like "this release of places-theme requires at least common 0.2.0 because we depend on a new API". But that's a deliberate / bump decision to begin with, not something that should cascade automatically. At that point it needs to be a human choice, and the assumption is the code change author/reviewers are best to set floors as-needed for deps.