feat: check request cancellation before backend dispatch

[pskiran1]

A request cancelled while waiting for an execution slot is no longer passed to the backend. The check sits in TritonModelInstance::Schedule, the single point every scheduler routes through on the way to TRITONBACKEND_ModelInstanceExecute...

[greptile-apps]

Greptile Summary

Adds a final cancellation check after scheduler and rate-limiter waiting but before backend dispatch.

• Completes and removes cancelled non-sequence requests before request preparation and backend execution.
• Avoids invoking the backend when cancellation leaves an empty batch.
• Prevents unsuccessful or null responses from being inserted into the dynamic batcher’s response cache.
• Preserves fixed request positions for sequence-batched models.

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding correctness or repository-rule findings.

The current code removes cancelled ordinary requests at the final backend-dispatch boundary, preserves sequence slots, avoids empty backend calls, and excludes unsuccessful responses from cache insertion. The only previous finding was manually resolved after Greptile conceded that fixed sequence positions make the exception intentional.

Important Files Changed

Filename	Overview
src/backend_model_instance.cc	Adds final pre-dispatch cancellation cleanup while preserving sequence-batch slot mappings and skipping empty backend executions.
src/backend_model_instance.h	Declares the cancellation-cleanup helper and documents its sequence-batching exception.
src/dynamic_batch_scheduler.cc	Restricts response-cache insertion to non-null responses with successful status while retaining ordered-response handling.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Scheduler forms payload] --> B[Rate limiter / execution-slot wait]
  B --> C[TritonModelInstance::Schedule]
  C --> D{Sequence batching?}
  D -->|Yes| E[Preserve fixed sequence slots]
  D -->|No| F[Complete and remove cancelled requests]
  F --> G{Requests remain?}
  G -->|No| H[Return success without backend dispatch]
  G -->|Yes| I[Prepare requests]
  E --> I
  I --> J[Backend execute callback]
  J --> K{Successful response?}
  K -->|Yes| L[Eligible for response-cache insertion]
  K -->|No| M[Deliver without caching]

Loading

_{Reviews (6): Last reviewed commit: "Merge branch 'main' of https://github.co..." | Re-trigger Greptile}

[greptile-apps]

Want your agent to iterate on Greptile's feedback? Try greploops.

[whoisj]

lgtm. thanks.