Add sqlmesh-airflow skill

[coleheflin]

Summary

Adds a new skill, sqlmesh-airflow, covering how to orchestrate an open-source SQLMesh project from an Airflow DAG.

Why this gap

There's no existing skill for SQLMesh, and it's easy for an agent (or a human) to get this wrong by:

• Confusing sqlmesh plan (deploy-time, interactive/CI) with sqlmesh run (scheduled, unattended) and auto-applying plans on every DAG tick
• Assuming an apache-airflow-providers-sqlmesh package exists (it does not — verified against PyPI) or that SQLMesh's own sqlmesh.schedulers.airflow module is still current (it was removed upstream; only Tobiko Cloud's paid facade remains)
• Pointing SQLMesh's state connection at ephemeral local storage on autoscaling Airflow workers
• Hardcoding warehouse credentials in config.yaml instead of sourcing them from an Airflow connection
• Instantiating a SQLMesh Context at DAG parse time instead of inside a task

What's covered

• plan vs run split and where each belongs
• Sourcing gateway credentials from an Airflow connection via SQLMesh's SQLMESH__GATEWAYS__<GATEWAY>__CONNECTION__<FIELD> env-var override scheme
• Coarse-grained (single sqlmesh run task) vs per-model task granularity, with the tradeoffs specific to SQLMesh (unlike dbt, a single task usually isn't wasteful since SQLMesh already computes incremental intervals)
• Dependency isolation (virtualenv/ExternalPython/KubernetesPodOperator) to avoid polluting the main worker image
• Keeping parse-time free of Context() instantiation

Validation

Beyond the initial research pass (PyPI 404 for the non-existent provider package, GitHub API check confirming sqlmesh/schedulers/ is now an empty stub), I had a second agent independently live-test this skill end-to-end against a real SQLMesh project (sqlmesh init duckdb, sqlmesh==0.236.1):

• Ran sqlmesh plan --auto-apply then sqlmesh run and confirmed the exact behavior the skill describes (auto-apply applies immediately with no further gate; run correctly no-ops until an interval is due)
• Confirmed the gateways.<name>.connection / default_gateway config shape via sqlmesh init
• Confirmed Context(paths=..., gateway=...) and context.run(environment=, start=, end=, ...) signatures via inspect.signature, and executed context.run(...) successfully
• Confirmed sqlmesh-dag-generator is a real, actively-maintained PyPI package (not hallucinated)
• Found one real gap: Step 3 gestured at "environment variables" without naming the actual scheme SQLMesh reads. Fixed by naming SQLMESH__GATEWAYS__<GATEWAY_NAME>__CONNECTION__<FIELD> explicitly (verified live) and wiring it into Step 4's Context() call, which was previously disconnected from the gateway resolved in Step 3.

🤖 Generated by Otto