Skip to content

EAI-7650: Set explicit request timeout on SeaweedFS S3 route - #800

Open
nowycondro wants to merge 1 commit into
mainfrom
EAI-7650-fix-seaweed-s3-route-timeout
Open

EAI-7650: Set explicit request timeout on SeaweedFS S3 route#800
nowycondro wants to merge 1 commit into
mainfrom
EAI-7650-fix-seaweed-s3-route-timeout

Conversation

@nowycondro

Copy link
Copy Markdown

Summary

The seaweed HTTPRoute declares no timeouts, so Envoy applies its 15 second default request timeout to the SeaweedFS S3 data path. Any transfer lasting longer than 15s is reset mid-stream.

The limit is duration, not size, which is why it doesn't show up as a body-size setting anywhere. On a slow client link even a modest object fails.

Evidence

Measured against app-dev using the same 26 MB object and the same presigned URL, varying only transfer speed:

Speed Duration Result
unthrottled 5.4s 26,139,600 bytes, complete
2.5 MB/s 9.5s 26,139,600 bytes, complete
1.2 MB/s cut at ~15s truncated at 23,330,816 bytes
800 KB/s cut at ~15s truncated at 14,172,160 bytes

Failures surface as HTTP/2 stream not closed cleanly: INTERNAL_ERROR. Note the response begins as HTTP 200 and the stream resets afterwards, so clients see a truncated object rather than a clean error — a silent corruption risk for anything that doesn't verify length or checksum.

Confirmed from the live Envoy config dump that the route's timeout field is unset, while routes that need longer already set one explicitly (aiwb-api-route runs at 10m/30m, AIM routes at 1800s). This route was simply left at the default.

Change

Sets request and backendRequest to 30m on the S3 route, matching the existing convention for long-running routes on the same gateway. The seaweed-admin route is left untouched.

Test plan

  • helm template sources/seaweedfs-config renders the expected timeouts block
  • kubectl apply --dry-run=server accepts the manifest, validating the 30m duration format against the Gateway API schema
  • After sync, re-run a throttled transfer that exceeds 15s and confirm it completes

Context

This unblocks adapter artifact uploads in AIWB (EAI-7650), where users upload LoRA adapter files directly to storage through a presigned URL. Without this, those uploads fail for anyone whose connection can't finish inside 15 seconds.

Made with Cursor

The seaweed HTTPRoute declared no timeouts, so Envoy applied its 15s default
request timeout to the S3 data path. Transfers longer than 15s were reset
mid-stream, truncating objects regardless of their size.

Co-authored-by: Cursor <cursoragent@cursor.com>
@nowycondro
nowycondro requested a review from a team as a code owner August 6, 2026 08:12
@nowycondro
nowycondro requested a lite review from Copilot August 6, 2026 09:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the SeaweedFS S3 HTTPRoute to explicitly set longer Gateway API timeouts so Envoy doesn’t apply its 15s default request timeout and reset long-running object transfers mid-stream.

Changes:

  • Add timeouts.request and timeouts.backendRequest set to 30m on the seaweed HTTPRoute rule.
  • Document the rationale inline (duration-based timeout causing truncated objects on slow links).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@brownzebra brownzebra left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is 15 sec -> 30 minutes really the amount to change? would 1m not be enough?

Instead of hardcoding in the template, I suggest to parametrize it into values and allow for a gitea override.

@brownzebra

Copy link
Copy Markdown
Contributor

OK, so it is based on the whole transfer... would still be nice to have it templated so it could be tuned in the field.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants