Skip to content

feat(secrets): gsm:// / env:// reference resolution in config (Google Secret Manager) - #80

Merged
anaselmhamdi merged 1 commit into
mainfrom
anaselmhamdi/secret-manager-config
Jun 26, 2026
Merged

feat(secrets): gsm:// / env:// reference resolution in config (Google Secret Manager)#80
anaselmhamdi merged 1 commit into
mainfrom
anaselmhamdi/secret-manager-config

Conversation

@anaselmhamdi

Copy link
Copy Markdown
Collaborator

What

Keep secrets out of pipeline YAML by referencing them with a URI scheme instead of plaintext. Resolution runs once over the raw config dict before Pydantic validation, so connectors keep reading plain strings and need no changes — same mechanism as the existing BIZON_ENV_ feature, generalized.

Google Secret Manager is the first provider; the design is factory-extensible (awssm://, vault:// later = one adapter + one registry entry).

Reference syntax

Form Example
Whole value token: gsm://notion-api-token
Pinned version key: gsm://bq-sa-key/versions/3
Full resource path gsm://projects/<p>/secrets/<id>/versions/<n>
Inline (in a larger string) dsn: "postgres://u:${gsm://db-pw}@host/db"
Env var env://MY_TOKEN (also works inline)
  • gsm:// authenticates via Application Default Credentials (workload identity / ambient creds).
  • Optional top-level secrets: block holds provider defaults (e.g. secrets.gsm.project_id).
  • Plain URIs (postgresql://..., https://...) are never mistaken for references (unknown scheme → left untouched).
  • Legacy whole-value BIZON_ENV_FOO keeps working unchanged.

CLI

bizon secrets check <config> dry-runs every reference and reports resolved/failed with masked output (length only), exiting non-zero on failure — catch typos / missing secrets / ADC issues before a real run.

source.authentication.params.token   gsm://notion-api-token   ✓  (40 chars)
engine.backend.config.password       ${gsm://db-pw}           ✗  could not access ...

Changes

  • New module bizon/engine/resolvers/AbstractReferenceResolver, ResolverRegistry (lazy per-scheme build + per-run cache), resolve_references_in_config, collect_references_in_config, and env / gcp.gsm adapters.
  • bizon/engine/engine.py — resolver pass after the legacy env pass.
  • bizon/common/models.py — optional secrets field on BizonConfig.
  • bizon/cli/main.pybizon secrets check.
  • pyproject.toml — new optional extra secretmanager (google-cloud-secret-manager); imported lazily.
  • Also fixes the env walker's list-skipping limitation (references in list fields now resolve).

Testing

  • 21 new unit tests (tests/engine/test_reference_resolution.py) covering whole-value/inline/pinned/full-path gsm, env, caching, list fields, error cases, and GSMResolver path-building (mocked GSM client — no real GCP). All pass.
  • Full tests/engine suite: 65 passed; the only failures are pre-existing my_pg_backend tests that require a live Postgres (unrelated).
  • ruff format + ruff check clean.
  • Smoke-tested the CLI end-to-end (success → exit 0, missing secret → exit 1, values masked).

🤖 Generated with Claude Code

…ecret Manager)

Add scheme-dispatched reference resolution so secrets stay out of YAML. A config
string can reference a managed secret via a URI scheme, resolved once over the raw
config before Pydantic validation — connectors need no changes.

- gsm://<id> (latest), gsm://<id>/versions/<N> (pinned), or full resource path;
  Google Secret Manager via Application Default Credentials
- env://<VAR> built-in scheme (works inline, unlike legacy whole-value BIZON_ENV_)
- inline ${...} tokens inside larger strings (e.g. DSNs), multiple per value
- optional top-level secrets: block for provider defaults (secrets.gsm.project_id)
- bizon secrets check <config> to dry-run/validate references (masked output)
- new optional extra: bizon[secretmanager]
- legacy BIZON_ENV_ references keep working unchanged

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@anaselmhamdi
anaselmhamdi requested a review from aballiet June 26, 2026 16:43
@anaselmhamdi
anaselmhamdi merged commit 61628f0 into main Jun 26, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant