[WIP] Add an embedded Galaxy engine for package-installed Galaxy

[jmchilton]

Requires a release of Galaxy containing #23360.

Opening a WIP PR just because we've done a lot of work not a reflection of it being close to being ready. I really dislike the tests in here and I'm unsure if the Galaxy layer should be proxied through Gravity or not. A lot of the complexity in planemo/galaxy/embedded.py seems not particularly Planemo-specific and I wonder if Gravity or a new galaxy-launcher-library project could absorb the complexity and isolate it from Planemo. Rest of this writeup is agent stuff. -John

Summary

This adds an opt-in --engine embedded_galaxy mode for planemo run, planemo test, and foreground planemo serve.

Instead of cloning or launching a Galaxy checkout, the engine loads a coherent, package-installed Galaxy into Planemo's Python process. Planemo still talks to Galaxy through its existing HTTP/BioBlend interfaces, so tool execution, workflow execution, reporting, profiles, dependency resolution, and Tool Shed installation continue to use the established Planemo paths.

The engine is deliberately opt-in and does not change the existing galaxy, docker_galaxy, external_galaxy, or uvx_galaxy engines.

Motivation

The managed Galaxy engine provides strong isolation but pays for checkout and subprocess startup. Package-installed Galaxy can provide a substantially lighter authoring and testing loop if Planemo owns the complete in-process lifecycle and leaves no global state, worker, server, child process, or temporary configuration behind.

This work also makes YAML Galaxy tools valid GalaxyTool runnables, fixing the existing mismatch where discovery yielded YAML tool sources that for_path() later rejected.

Implementation

• Registers embedded_galaxy in the engine factory and relevant CLI choices, with validation for unsupported checkout, daemon, non-loopback, and interactive-tool options.
• Extracts checkout-independent managed-Galaxy configuration shared by the existing local and embedded engines.
• Builds Galaxy lazily with build_galaxy_web_app(..., register_shutdown_at_exit=False) and restores Galaxy's process-global application reference on every exit path.
• Pre-binds one loopback socket and serves Galaxy's ASGI application on a dedicated uvicorn thread, avoiding the probe/close/bind race.
• Starts one in-process Celery worker with the solo pool and both galaxy.internal and galaxy.external queues, using in-memory transport and the RPC result backend.
• Reuses the existing Galaxy engine's HTTP execution, workflow, profile, reporting, dependency-resolution, PostgreSQL/Singularity, and Tool Shed installation paths.
• Keeps mixed tool/workflow tests inside one Galaxy construction rather than attempting multiple application lifecycles in one process.
• Captures Galaxy, Celery, and uvicorn logs in the generated configuration directory while preserving Planemo's normal and Rich verbose output behavior.
• Performs ordered, failure-tolerant teardown of uvicorn, Celery, Galaxy's fork pool, the Galaxy application, logging/environment state, and temporary configuration.
• Clears Celery's process-global termination flag after a timed-out worker shutdown so a failed worker cannot poison the next in-process invocation.
• Reports live threads and multiprocessing children when cooperative cleanup exceeds 30 seconds. This is diagnostic rather than a hard bound: Pebble currently performs unbounded internal joins after a process pool is stopped.

Scope and limitations

Version 1 supports local tools and workflows, Tool Shed installs, profiles/PostgreSQL, and the existing Conda/container dependency resolvers.

It does not:

• become the default engine;
• embed a Galaxy source checkout;
• support daemon mode or non-loopback binding;
• run gx-it-proxy, interactive tools, Celery beat, or periodic maintenance;
• promise two independent Galaxy application constructions in one Python process; or
• expose Galaxy's application object as a public Planemo API.

Test coverage

Fast lifecycle and configuration coverage includes:

• option validation and lazy optional-runtime imports;
• generated checkout-free Galaxy configuration;
• startup and teardown ordering;
• construction, readiness, uvicorn, partial-worker, and cleanup-action failures;
• first and second Ctrl-C behavior;
• process-global Galaxy and Celery state restoration;
• logging restoration, bounded failure-log replay, and --no_cleanup preservation;
• fork-pool, thread, child-process, socket, and temporary-directory cleanup; and
• one application construction for mixed tool/workflow inputs.

Opt-in tests against Galaxy packages built with #23360 prove:

• XML and YAML tool execution;
• real Celery-backed upload execution;
• local and Tool Shed-installed workflow execution;
• two consecutive planemo run output downloads in fresh subprocesses;
• foreground planemo serve readiness and SIGINT cleanup; and
• no surviving Planemo-owned process group or listening socket after subprocess exit, plus focused restoration checks for global Galaxy/Celery state, logging, threads, and generated configuration.

Latest focused results:

50 passed, 6 skipped
real embedded acceptance suite against Galaxy dev: 3 passed
mixed XML/YAML/upload/workflow/Tool Shed acceptance: passed
real repeated subprocess run and foreground serve acceptance: passed
black, isort, ruff, flake8, and whitespace checks: passed

Before marking ready for review

• Rebase onto master after Planemo Honor Galaxy test timeouts across uploads and workflows #1687 merges and confirm the PR contains only the embedded-engine work.
• Use the first Galaxy release containing #23360 to add planemo[embedded_galaxy] with a one-series upper bound.
• Add optional-dependency containment and resolver checks.
• Add user documentation and Linux per-PR integration coverage against the released package set.
• Record comparable cold and warm startup timings for galaxy and embedded_galaxy.
• Incorporate the separate mypy cleanup and run tox -e mypy cleanly.
• Run the existing-engine and full Planemo regression suites after the final rebase.

Suggested reviewer path

1. Engine registration, option validation, and YAML runnable recognition.
2. Shared managed-Galaxy configuration extraction and parity tests.
3. planemo/galaxy/embedded.py startup, logging, and ordered teardown.
4. Engine reuse for run/test/workflow/Tool Shed execution and the one-application test path.
5. Concrete package-installed acceptance tests and subprocess cleanup assertions.

[jmchilton]

Recommendation on Gravity and ownership

Short answer: I would not route embedded_galaxy through Gravity.

Gravity's current abstraction is process and service orchestration. Its foreground galaxy command selects the multiprocessing process manager; that manager creates an OS process for each configured service, and its executor ultimately replaces those children with the real Gunicorn/Celery commands. Its stop, terminate, shutdown, and related lifecycle methods are currently stubs. That is a useful boundary for managing Galaxy services, but it is not an in-process Galaxy application lifecycle (usage documentation, multiprocessing implementation, process executor).

Putting Gravity underneath embedded_galaxy would therefore either run Galaxy in child processes—changing the semantics and likely giving back some of the startup benefit—or require a substantial new in-process abstraction that broadens Gravity's present scope.

I recommend this ownership split:

• Galaxy core: expose a supported, context-managed embedded runtime API, building on the merged build_galaxy_web_app work. Galaxy should own application startup/shutdown, restoration of its process-global application state, in-process Celery worker cleanup (including global termination state and fork pools), and logging isolation. These details are tightly coupled to Galaxy and Celery versions, so keeping them in Galaxy avoids making Planemo track private internals.
• Planemo: retain the adapter and policy layer: translate CLI options and runnables into ephemeral Galaxy configuration, select sockets/ports and perform HTTP readiness checks, integrate Planemo logging and diagnostics, install tools/workflows, and implement temporary-directory/no_cleanup and engine behavior.
• Gravity: continue owning process definitions and external service orchestration—subprocess launching and production-style process management.

A separate Gravity-backed, package-installed subprocess engine is still worth prototyping. Benchmark its cold and warm startup against the embedded engine. It could be simpler and more robust because of process isolation, and might eventually be preferable if its latency is acceptable. It should be exposed as a distinct engine, however, rather than hidden as an implementation detail of embedded_galaxy.

I would also avoid creating a new galaxy-launcher-library repository for now. Initially it would mostly relocate private Galaxy/Celery coupling while adding another synchronized release boundary. Start with the lifecycle API inside Galaxy; split it into a library only after the boundary is stable and there is a second real consumer.

Authorship: OpenAI Codex — GPT-5 (session date 2026-09-02).

[mvdbeek]

require a substantial new in-process abstraction that broadens Gravity's present scope.

gravity has the subprocess setup, isn't that the right abstraction ? threads are going to be gil bound

[jmchilton]

Why was codex convinced by your comment and not mine - it didn't even really buy GIL as a reason - it thought process isolation was the real benefit. I've told my agents "Look at mvdbeek's comments - he is smarter than me." enough times they've recorded in their memories I bet 😠😆. Anyway this is prototyped in #1691 as a separate engine. I guess it would be better to not have both though.

The best argument against Gravity I think is that the webserver is a stop-gap right? I would love a core driving loop that just handles like actions and doesn't need any sort of web layer. Driving a tool submission -> preparation -> execute -> job finalize - could probably be easily done with some library code. Workflows would really need probably some sort of application loop - but we've done some structuring around this in core and it would be great. I don't think that goal should prevent this from using gravity in this modality though.