BUNDLE_RETRY is not applied to lazy full-index quick/Marshal gemspec downloads

[avcsabi]

BUNDLE_RETRY is not applied to lazy full-index quick/Marshal gemspec downloads

When Bundler resolves dependencies through the full-index API, it lazily downloads individual gemspecs from paths such as:

/quick/Marshal.4.8/example-gem-1.0.0.gemspec.rz

If one of these requests encounters a transient network failure, such as Gem::Net::ReadTimeout, Bundler immediately aborts the entire installation. The configured BUNDLE_RETRY value is not applied to this request.

This is particularly disruptive for applications without a lockfile, where dependency resolution may retrieve a large number of individual gemspecs. A single temporary repository or network failure can terminate an otherwise long-running resolution.

The apparent call path is:

Bundler::RemoteSpecification
  -> Bundler::Fetcher#fetch_spec
  -> Bundler::Fetcher::Downloader#fetch
  -> HTTP request

Bundler::Fetcher#fetch_spec directly calls:

downloader.fetch(uri).body

without wrapping the operation in Bundler::Retry.

Downloader classifies timeout and connection exceptions as retryable HTTP errors, but only converts them to Bundler::HTTPError; it does not retry the request itself.

The retry used by specs_with_retry protects index-fetching operations, but it does not encompass subsequent lazy downloads of individual gemspecs.

Relevant Bundler 4.0.19 source:

• https://github.com/ruby/rubygems/blob/v4.0.19/bundler/lib/bundler/fetcher.rb
• https://github.com/ruby/rubygems/blob/v4.0.19/bundler/lib/bundler/fetcher/downloader.rb
• https://github.com/ruby/rubygems/blob/v4.0.19/bundler/lib/bundler/remote_specification.rb
• https://github.com/ruby/rubygems/blob/v4.0.19/bundler/lib/bundler/retry.rb

Did you try upgrading rubygems & bundler?

The repeated production CI failures were originally observed with Bundler 2.6.7.

I also checked the latest RubyGems/Bundler 4.0.19 source. The same unprotected fetch_spec path is still present: individual /quick/Marshal.4.8/*.gemspec.rz downloads are not wrapped in Bundler::Retry.

Bundler 4.0.19 has an improved retry implementation with exponential backoff and jitter, but that implementation is only effective where callers explicitly use Bundler::Retry. The lazy fetch_spec path does not currently use it.

Post steps to reproduce the problem

The following deterministic reproduction simulates one transient failure at the same boundary where Downloader normally raises Bundler::HTTPError.

1. Use Ruby 3.4 with RubyGems/Bundler 4.0.19.

2. Create simulate_transient_quick_fetch_failure.rb:

# frozen_string_literal: true

require "bundler/fetcher/downloader"

module FailFirstQuickGemspecFetch
  class << self
    attr_accessor :failed
  end

  def fetch(uri, ...)
    if uri.to_s.include?("/quick/Marshal.4.8/") &&
        !FailFirstQuickGemspecFetch.failed
      FailFirstQuickGemspecFetch.failed = true

      warn "Simulating one transient quick gemspec download failure: #{uri}"
      raise Bundler::HTTPError, "simulated transient network failure"
    end

    super
  end
end

Bundler::Fetcher::Downloader.prepend(FailFirstQuickGemspecFetch)

3. Create this Gemfile:

# frozen_string_literal: true

require_relative "simulate_transient_quick_fetch_failure"

source "https://rubygems.org"

gem "htmlentities", "4.2.4"

4. Make sure there is no existing Gemfile.lock or populated bundle cache.

5. Run Bundler with full-index resolution and retries enabled:

BUNDLE_RETRY=3 BUNDLE_TIMEOUT=10 bundle _4.0.19_ install --full-index --verbose

The injected failure occurs only once. Therefore, if the individual gemspec request honors BUNDLE_RETRY, the next attempt should succeed normally.

The issue can also be observed without instrumentation when an actual full-index gem repository intermittently times out while serving an individual /quick/Marshal.4.8/*.gemspec.rz request.

Which command did you run?

BUNDLE_RETRY=15 BUNDLE_TIMEOUT=10 bundle install

The equivalent latest-version reproduction command is:

BUNDLE_RETRY=3 BUNDLE_TIMEOUT=10 bundle _4.0.19_ install --full-index --verbose

What were you expecting to happen?

A transient failure while downloading an individual gemspec should be retried according to BUNDLE_RETRY.

With BUNDLE_RETRY=3, Bundler should make the initial request and up to three additional attempts. In the deterministic reproduction, the first request should fail and the second request should succeed.

Ideally, the retry should also be visible in verbose output and identify the affected gemspec URL.

What happened instead?

Bundler aborted after the first failed individual gemspec request. It did not retry that request, even though BUNDLE_RETRY was configured.

Examples from anonymized CI failures:

+ bundle install
Fetching source index from https://REDACTED/rubygems-snapshot/
Fetching gem metadata from https://REDACTED/rubygems/..
Resolving dependencies...
Network error while fetching
https://REDACTED/rubygems/quick/Marshal.4.8/activemodel-3.0.17.gemspec.rz
(Gem::Net::ReadTimeout)

Another independent run failed on a different gemspec:

+ bundle install
Fetching gem metadata from https://REDACTED/rubygems/..
Resolving dependencies...
Network error while fetching
https://REDACTED/rubygems/quick/Marshal.4.8/rubocop-ast-1.7.0.gemspec.rz
(Gem::Net::ReadTimeout)

The fact that different gemspecs fail across otherwise identical runs is consistent with intermittent repository/network failures. In every case, the first timed-out individual gemspec request terminates the installation.

If not included with the output of your command, run bundle env and paste the output below

The original CI environment contains private repository addresses and application Gemfiles, so the relevant values are provided in anonymized form:

Ruby: 3.3.x
Bundler: 2.6.7
Platform: x86_64-linux
Installation: containerized CI environment
Lockfile: none
Bundler retry: 15
Bundler timeout: 10
Gem sources:
  - private released-gem repository
  - private snapshot-gem repository
Resolution mode:
  - full-index fallback

The standalone reproduction above targets:

Ruby: 3.4.x
RubyGems: 4.0.19
Bundler: 4.0.19
Source: https://rubygems.org
Resolution option: --full-index
Bundler retry: 3
Bundler timeout: 10

Suggested behavior

Individual gemspec downloads performed by Bundler::Fetcher#fetch_spec should use Bundler's configured network retry policy.

One possible implementation would be to wrap the downloader.fetch(uri) operation with Bundler::Retry, using the same retryable error set and retry configuration used for other fetch operations.

A regression test could make the downloader fail once with a retryable error and then succeed, verifying that:

1. fetch_spec retries the request.
2. The configured retry count is respected.
3. A successful retry returns the gemspec.
4. Exhausting all attempts raises the final error.
5. Verbose output identifies the retried gemspec request.