Add backend-neutral runner and Azure ML compute backend

[Joaquín Rivero (jQuinRivero)]

Description

Adds a typed, backend-neutral compute execution contract and Azure Machine Learning command-job adapter while preserving Azure Batch and local Docker behavior.

• Migrates training, inference, embedding, imagery preparation, and artifact packaging to persisted ComputeJobHandle lifecycle operations.
• Adds explicit and deterministic auto backend routing, idempotent submission, legacy Batch-handle compatibility, and server-owned runtime-field protection.
• Makes HASTE_JOB_WORKDIR the provider-neutral container workspace contract while retaining legacy Batch aliases.
• Adds AML configuration, dependency wiring, reference-only Existing mode, optional source-only Create mode, tests, architecture/spec documentation, and changelog coverage.

The initial rollout references operator-provided AML resources with AML_MODE=Existing. No Azure resources were created, changed, or deployed during implementation or validation.

Type of change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that changes existing behavior)
• Documentation update
• Infrastructure / CI change

Checklist

• I have read CONTRIBUTING.md
• My changes follow the project's coding standards
• I have added or updated tests that cover my changes
• Python tests pass locally (cd hastelib && hatch run test:pytest) and the UI lints clean (cd ui && npm run lint)
• I have updated the relevant documentation
• I have added an entry to CHANGELOG.md

Testing

• Consolidated compute models/runners/processors: 663 passed, 5 skipped (Windows-only symlink cases).
• API and queue suites: 90 passed.
• Container workspace suites: 21 passed.
• Batch adapter/idempotency suites: 93 passed.
• Configuration and dependency policy suites: 58 passed.
• Full hastelib suite: 1,103 passed, 5 skipped, with one unchanged Azurite-dependent fixture failure in test_artifacts.py::TestArtifactProcessor::test_zip; the fixture fails while creating its test container before feature code runs because local Azurite is unavailable.
• Feature-file Black, isort, flake8, and detect-secrets hooks pass.
• Bicep compiles locally; infra/main.json is synchronized; static checks confirm Existing mode creates or mutates zero AML resources.
• Wheel build contains every new compute module.
• Final backend validation and pre-landing review found no Critical, High, or Medium blockers.
• The latest GitHub Copilot review findings were addressed with routing, Actions deployment, Create-mode dependency RBAC, storage-networking, and documentation regressions covered by tests.

Live AML CPU/GPU smoke tests, five-workload Batch/AML output-parity tests, mixed-backend execution, cross-deployment access denial, and live deployment validation remain Stage 1 rollout gates. They were not run because cloud operations were explicitly excluded from this implementation pass.

Additional context

The commit history is split into atomic commits covering the spec/ADR, core contract, Batch/local adapters, AML adapter/dependency, workload migration, container workspace contract, API/queue integration, infrastructure wiring, and public documentation.

Supersedes #192, which GitHub closed automatically when the head branch was renamed to follow the repository branch convention.

[Copilot]

🔵 Needs a closer look

It’s a large cross-cutting compute + infra change, and the auto-routing path still lacks fast-fail validation for duplicate backend candidates (can silently skew routing).

Pull request overview

Introduces a backend-neutral compute execution layer (ComputeJobSpec/ComputeJobHandle) and adds an Azure Machine Learning (AML) adapter, while preserving Azure Batch and local Docker behavior. This expands HASTE’s compute backends and updates deployment/configuration, tests, and documentation to support per-job and per-workload backend selection plus deterministic auto routing.

Changes:

• Added a typed compute contract and orchestration components (registry/router/execution service) and migrated workloads to handle-based lifecycle operations.
• Wired AML configuration through IaC and deployment scripts (Disabled/Existing/Create modes) with dependency pinning and log-safety regressions tests.
• Updated container workspace contract to use HASTE_JOB_WORKDIR with legacy AZ_BATCH_* compatibility, plus broad docs/spec/ADR updates.

File summaries

File	Description
THIRD-PARTY-NOTICES.md	Notes Azure ML usage via azure-ai-ml under Azure SDKs.
spec/features/aml-compute-backend/README.md	Feature spec overview for backend-neutral runner + AML backend.
spec/features/aml-compute-backend/impact-analysis.md	Risk/scope/impact analysis for compute refactor + AML.
spec/architecture/overview.md	Architecture diagram/text updated to include AML/local compute.
spec/architecture/decisions/0005-backend-neutral-compute-runner-and-aml-backend.md	New ADR capturing the runner contract change and AML enablement strategy.
README.md	High-level system diagram updated to reflect AML/local compute.
infra/modules/storage.bicep	Allowlists AML compute subnet (Create-mode) in storage firewall rules.
infra/modules/functions.bicep	Adds compute backend default + AML app settings plumbing and outputs queue principal ID.
infra/modules/amlWorkspace.bicep	New Create-mode AML workspace module (keyless/identity-based).
infra/modules/amlRole.bicep	New Create-mode least-privilege AML RBAC assignment module.
infra/modules/amlEnvironment.bicep	New Create-mode immutable AML environment version registration.
infra/modules/amlDatastore.bicep	New Create-mode identity-based AML datastore registration.
infra/modules/amlCompute.bicep	New Create-mode AML compute cluster module.
infra/main.bicepparam	Adds environment-variable mapping for compute default + AML modes/settings.
hastelib/tests/core/test_config.py	Unit tests for Config compute/AML config parsing helpers.
hastelib/tests/core/runners/test_runner_log_safety.py	Regression tests ensuring runner/execution-service logs don’t leak signed URLs.
hastelib/tests/core/processors/test_processors_backend_neutral.py	Enforces processors no longer reference Batch-specific config/vars or legacy runner APIs.
hastelib/tests/core/processors/test_imagery_preprocess_config.py	Updates imagery processor tests to inject execution_service.
hastelib/tests/core/processors/test_imagery_output_fallback.py	Updates output/log fallback tests to use handle-based output reads.
hastelib/tests/core/models/test_projects_compute.py	Tests additive computeJob/computeBackend fields across project/job models.
hastelib/tests/core/models/init.py	Package marker for new models tests.
hastelib/tests/build/test_release_workflows.py	Adds drift/AML setting coverage checks for deploy paths and workflows.
hastelib/tests/build/test_dependency_pins.py	Validates azure-ai-ml==1.34.1 pin placement and installation scope.
hastelib/src/hastegeo/core/runners/unified_runner.py	Deprecates legacy runner factory and improves logging safety around resource file dicts.
hastelib/src/hastegeo/core/runners/router.py	New deterministic auto-router (weighted rendezvous hashing).
hastelib/src/hastegeo/core/runners/registry.py	New lazy adapter registry keyed by backend/profile.
hastelib/src/hastegeo/core/runners/execution_service.py	New submit/lifecycle dispatch service with idempotency and handle-based routing.
hastelib/src/hastegeo/core/models/projects.py	Adds computeJob (server-owned) and computeBackend (request intent) fields.
hastelib/pyproject.toml	Adds azure-ml optional extra and includes it in the test env features.
env.yml	Adds azure-ai-ml==1.34.1 to dev environment dependencies.
docs/configuration.md	Documents compute backend selection and AML backend configuration/modes.
docs/architecture.md	Updates architecture docs to describe compute backends and neutral runner contract.
docker/training/scripts/tests/test_set_dirs.py	Contract tests for HASTE_JOB_WORKDIR + legacy alias behavior in training image.
docker/training/scripts/tests/test_entrypoint_workdir.py	Contract tests for entrypoint workdir resolution semantics.
docker/training/scripts/tests/init.py	Package marker for training script tests.
docker/training/scripts/set_dirs.sh	Implements HASTE_JOB_WORKDIR canonical workdir + legacy export + config substitution.
docker/training/scripts/entrypoint.sh	Uses canonical/legacy workdir resolution for post-command chmod behavior.
docker/training/code/tests/test_run_workflow.py	Tests provider-neutral workdir resolution in run_workflow.py.
docker/training/code/run_workflow.py	Switches runtime workdir usage to canonical/legacy-neutral resolver.
docker/imageryprep/scripts/tests/test_set_dirs.py	Contract tests for HASTE_JOB_WORKDIR + legacy alias in imageryprep image.
docker/imageryprep/scripts/tests/init.py	Package marker for imageryprep script tests.
docker/imageryprep/scripts/set_dirs.sh	Implements canonical workdir resolution and legacy alias export for imageryprep.
CHANGELOG.md	Adds an Unreleased entry describing the compute refactor + AML backend.
api/hastefuncqueues/tests/test_compute_follow_on.py	Tests follow-on backend inheritance policy and queue log sanitization.
api/hastefuncqueues/requirements.txt	Adds azure-ai-ml==1.34.1 to queues app dependency set.
api/hastefuncqueues/function_app.py	Routes follow-on backend selection and replaces raw message-body logging with safe identifiers.
api/hastefuncapi/function_app.py	Adds backend validation and server-owned runtime field protection on launch/edit endpoints.
.gitleaksignore	Adds allowlist entries for synthetic credential-shaped test fixtures.
.github/workflows/deploy-apps.yml	Wires compute/AML settings into deployment workflow environment.
.github/scripts/deploy_apps.sh	Emits compute/AML app settings in the legacy deployment script.
.github/scripts/check_env_drift.py	Documents AML/compute settings that are read dynamically and/or are intentionally opt-in.

Review details

• Files reviewed: 64/87 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

RC artifacts ready

All branch deployment references use the same RC tag:

• hastegeo_version: 1.0.40rc11
• training_image_tag: 1.0.40rc11
• imageprep_image_tag: 1.0.40rc11
• wheel: hastegeo-1.0.40rc11-py3-none-any.whl

[github-actions]

RC artifacts ready

All branch deployment references use the same RC tag:

• hastegeo_version: 1.0.40rc10
• training_image_tag: 1.0.40rc10
• imageprep_image_tag: 1.0.40rc10
• wheel: hastegeo-1.0.40rc10-py3-none-any.whl

[Copilot]

🟡 Changes recommended

Unresolved adapter validation, output-download, identifier-collision, and Create-mode RBAC issues can prevent reliable AML execution.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (6)

Previously missed (5) — in code that hasn't changed since the last review.

hastelib/src/hastegeo/core/models/compute.py:647

• The validator says it restricts IDs to characters every provider accepts, but it permits . and has no length bound. Azure Batch task IDs allow only alphanumeric, hyphen, and underscore and are capped at 64 characters, while the Batch adapter uses executionId verbatim as task_id; therefore a model-valid spec can be deterministically rejected by Batch. Either enforce the cross-provider intersection here or derive a collision-safe Batch-specific task ID.
  hastelib/src/hastegeo/core/runners/azure_ml.py:464
• This truncates the deterministic AML name without preserving a digest. Two valid long executionId values with the same first 200 characters (or a long common experiment prefix) map to the same provider job name; _get_existing_job() then sees a mismatched execution tag and permanently rejects the second legitimate submission. Use the collision-safe truncation helper (prefix plus hash), as the Batch adapter does.

This issue also appears on line 494 of the same file.
hastelib/src/hastegeo/core/runners/azure_ml.py:1123

• Capacity probing skips _validate_config(). If AML is disabled or missing workspace settings but an AML_COMPUTE_* value remains configured, this reaches MLClient construction and can raise an unclassified SDK/argument error, aborting the entire auto candidate pass instead of reporting AML as unavailable. Validate the base AML configuration before constructing the client and map configuration failures to an UNAVAILABLE snapshot.
  hastelib/src/hastegeo/core/runners/router.py:245
• Programmatically supplied auto_weights bypass weights_from_env() validation. A zero or negative integer reaches weighted rendezvous scoring and silently suppresses or reverses a candidate instead of reporting malformed routing configuration. Apply the same positive-weight invariant in resolve() so both configuration entry points behave consistently.
  hastelib/src/hastegeo/core/utils/compute_specs.py:332
• Enabled mode alone does not mean the explicitly requested AML backend is configured. With AML_MODE=Existing but a missing subscription, resource group, workspace, datastore, compute target, environment, or managed-identity ID, this returns success and the HTTP route queues work that can only fail in the worker. This contradicts the feature test plan's IT-003 expectation that an explicit unconfigured backend returns 400; validate the required AML settings for the requested workload here.

hastelib/src/hastegeo/core/runners/azure_ml.py:503

• AML validates that outputs share a destination but never validates their URI scheme. ComputeOutput accepts schemes such as azureml, adl, and s3, yet this adapter passes each URI through Blob-shaped split_destination_uri() and later constructs an Azure ContainerClient from the persisted URI. Such a spec is accepted here, silently mapped to the wrong datastore prefix, and then fails during output reads. Reject output schemes that this Blob-backed mapping cannot translate before splitting them.

• Files reviewed: 64/87 changed files
• Comments generated: 1
• Review effort level: Balanced

[github-actions]

RC artifacts ready

All branch deployment references use the same RC tag:

• hastegeo_version: 1.0.40rc13
• training_image_tag: 1.0.40rc13
• imageprep_image_tag: 1.0.40rc13
• wheel: hastegeo-1.0.40rc13-py3-none-any.whl