[WIP] Run package-installed Galaxy through Gravity

[jmchilton]

Status

This is a WIP architecture spike based on #1690. It reuses that PR's checkout-independent configuration work, YAML-tool support, execution fixes, and test infrastructure, but replaces Planemo's ownership of the embedded Galaxy lifecycle with Gravity-managed subprocesses.

The motivation and broader package-installed Galaxy discussion are in #1690. This PR explores the alternative raised there and in #1690 (comment).

The incremental Gravity implementation is currently the final commit, de66cd2. If this direction is accepted, the intended next step is to reshape #1690 around this approach, not permanently expose two competing package-installed lifecycle models.

What differs from #1690

• Adds an opt-in installed_galaxy engine that invokes the galaxy command from Planemo's Python environment.
• Uses Gravity's existing multiprocessing process manager and direct service commands; this prototype required no Gravity source changes.
• Runs Gunicorn and a separate Celery worker instead of uvicorn and Celery inside the Planemo process.
• Uses Gravity's existing virtualenv setting to bind all services to Planemo's environment, avoiding PATH-dependent executable selection.
• Uses temporary, cross-process SQLite Celery broker and result-backend databases rather than in-memory Celery transports.
• Keeps Galaxy's data, database, Gravity state, logs, and Celery state inside Planemo's temporary configuration directory.
• Reuses Planemo's existing HTTP/BioBlend execution paths and bounded process-group cleanup.
• Rejects checkout-specific options that do not make sense for a package-installed Galaxy.

In particular, this removes the need for Planemo to manage Galaxy application globals, uvicorn threads, Celery worker internals, logging restoration, or ordered in-process teardown. Gravity remains the Galaxy service-lifecycle boundary.

Current configuration

The generated Gravity configuration currently uses:

• process_manager: multiprocessing
• service_command_style: direct
• one Gunicorn worker without preload
• one Celery worker using the solo pool and the galaxy.internal and galaxy.external queues
• Celery beat and gx-it-proxy disabled
• Planemo's active Python environment as Gravity's virtualenv

These settings are deliberately small and test-oriented; they are not proposed as new general Gravity defaults.

Evidence so far

• A real YAML Galaxy tool passed end to end under Gravity 1.2.4, including Celery-backed staging and tool execution.
• Foreground planemo serve starts successfully and terminal Ctrl-C shuts down Planemo, Gravity, Gunicorn, and Celery without leaving a listener behind.
• Two managed runs use fresh temporary state and process groups.
• The generated configuration is checkout-free and keeps runtime data out of the installed package/source tree.
• New focused tests: 7 passed.
• Broader focused Planemo suite: 78 passed, 13 skipped.
• Planemo mypy target: clean across 274 source files.
• Black, isort, Ruff, and whitespace checks: clean.

One initial A/B run measured approximately 29.1 seconds for the in-process engine and 32.5 seconds for the Gravity engine. That is only a preliminary data point, but it suggests Gravity preserves most of the startup improvement while retaining normal Galaxy process isolation.

Before this is ready

• Decide whether this should replace the embedded_galaxy implementation in [WIP] Add an embedded Galaxy engine for package-installed Galaxy #1690. That is the current recommendation.
• Rebuild the branch into a reviewable independent history once the architecture is agreed upon.
• Refactor duplicated managed-process plumbing shared with the checkout-backed engine.
• Run the complete XML, YAML, local-workflow, and Tool Shed workflow acceptance matrix against the final Galaxy package set.
• Add Linux CI coverage and package/dependency declarations for the first compatible Galaxy release.
• Run the full Planemo regression suite after the final branch rewrite.