-
Notifications
You must be signed in to change notification settings - Fork 2
Serving-engine Coverage: establish vLLM feasibility and integration scope #6
Copy link
Copy link
Open
Labels
area:servingContext Parallel Serving and additional serving integrations.Context Parallel Serving and additional serving integrations.researchEvidence gathering, compatibility investigation, or an exploratory prototype.Evidence gathering, compatibility investigation, or an exploratory prototype.roadmapA direction-level issue with an explicitly bounded initial stage.A direction-level issue with an explicitly bounded initial stage.
Description
Activity
Metadata
Metadata
Assignees
Labels
area:servingContext Parallel Serving and additional serving integrations.Context Parallel Serving and additional serving integrations.researchEvidence gathering, compatibility investigation, or an exploratory prototype.Evidence gathering, compatibility investigation, or an exploratory prototype.roadmapA direction-level issue with an explicitly bounded initial stage.A direction-level issue with an explicitly bounded initial stage.
Outcome and baseline
Support another serving engine while keeping engine-owned types and state out
of core Plans, Registries, Projections, and the native ABI. vLLM is the first
candidate. The existing SGLang integration and
xpool.ops.ffn_shimexpose auseful boundary, while the FfnAgent also has a bounded dependency on SGLang
low-level Expert kernels that needs explicit treatment.
This issue covers compatibility investigation, a representative end-to-end
spike, and the integration target design. It does not establish vLLM support.
Suggested implementation route
weight filtering, TP/DP, eager/graph execution, output ownership, process
lifecycle, and failure propagation. Identify which existing integration
behavior is engine-neutral and which remains SGLang-owned.
Transport/Fabric path. Observe startup, request completion, output
ownership, eager/captured execution, and failure/shutdown. Document the
treatment of low-level SGLang operator dependencies.
bootstrap/configuration ownership, and qualification requirements. Decide
whether public extension seams suffice and justify any narrower dependency.
engine translation at that integration's ownership boundary.
failure for the selected scope while retaining valid SGLang evidence.
Discovery completion evidence
extension surfaces or incompatible execution contracts.
Create a shared provider abstraction only after a second actual integration
demonstrates a shared seam. Missing hardware is an unmet prerequisite. Model
Coverage and Timeline observability can help evaluation; neither implies a
blanket cross-engine model-coverage claim.
References