Add content addressable gems support

[jenshenny]

#9654

TL;DR

Adds RubyGems and Bundler client support for content-addressable ("skinny") binary gems: one artifact per Ruby ABI, named with a SHA-256 prefix.

nokogiri-1.18.9-x86_64-linux.gem # traditional platform gem
nokogiri-1.18.9-78be552b.gem     # content-addressable gem

The branch covers build, discovery, install, display, yank, lockfiles, caching, and bundle install --local. Existing source and platform gems are unchanged.

Why

"Fat" binary gems contain every supported Ruby ABI and keep growing. Skinny binaries are smaller, but builds for the same gem, version, and platform need distinct filenames. A content-derived suffix gives each artifact a unique identity.

A content address must be 8–64 lowercase hexadecimal characters. RubyGems only treats it as one when the gem also has a non-Ruby platform and constrained required_ruby_version, avoiding false matches with ordinary filenames.

Remaining TODOs / known bugs

• gem install with cooldown isn't content addressable aware
• gem commands that should support CA (gem fetch, gem update, gem dependency)

Follow ups

• Update guides

---

gem build

ref: Shopify/rubygems#171.

User behaviour

• gem build nokogiri.gemspec --ruby-abi 3.4 builds a skinny gem named nokogiri-1.18.9-78be552b.gem, where the suffix is the SHA-256 digest of the gem contents.
• Without --ruby-abi, behaviour is unchanged.

Details

• --ruby-abi validates the ABI format (X.Y), requires a non-Ruby platform to be set, and constrains required_ruby_version: if unset it defaults to ~> X.Y.0; a mismatched existing requirement is rejected.
• The gem is built in memory and validated against the requested ABI; the passed-in spec is only updated once the build succeeds, so a failed build leaves the spec untouched.
• The hash is deterministic for a given set of gem contents.

gem install

ref: Shopify/rubygems#172 (local) and #173 (remote).

Local

• Reads the archive, derives the hash from its filename, verifies it against the file's SHA-256, and preserves both hash and platform. The installed directory and gemspec use the hash. A mismatch fails; reinstalling is idempotent.
• gem install --local GEMNAME is content-addressable aware.

Every install of a CA gem writes a gemspec stub whose # stub: suffix is the hash (so the name-version-<sha> directory resolves). The real platform rides on a separate # stub-target: line that older RubyGems ignore — backwards compatible, while current RubyGems recover both. File: specifications/mygem-1.0-78be552b.gemspec:

# -*- encoding: utf-8 -*-
# stub: mygem 1.0 78be552b lib
# stub-target: platform=x86_64-linux
# stub: ext/mygem/extconf.rb

Gem::Specification.new do |s|
  s.name = 'mygem'
  s.version = Gem::Version.new '1.0'
end

Remote

• Decodes the compact-index hash and platform:= metadata separately. Distinct hashes remain distinct candidates.

For example, a server info/nokogiri response with two skinny variants (different Ruby ABIs) plus a platform fallback:

---
1.18.9-78be552b |checksum:…,ruby:~> 3.4.0,platform:= x86_64-linux
1.18.9-a1b2c3d4 |checksum:…,ruby:~> 3.3.0,platform:= x86_64-linux
1.18.9-x86_64-linux |checksum:…

The hash is carried in the version token (1.18.9-78be552b); the real platform and Ruby requirement travel in the platform:= / ruby: metadata. The two hashes stay distinct resolver candidates, and 1.18.9-x86_64-linux is the platform fallback.

• Prefers a compatible skinny gem; resolution falls back to a traditional platform gem, then a source gem.
• Downloads the SHA-named archive and the SHA-named gemspec for development dependencies.

gem push

ref: Shopify/rubygems#174.

User behaviour

• gem push name-*.gem --platform x86_64-linux --ruby-abi 3.4 reads the specs of the SHA-named files and pushes the single matching artifact.
• No match or multiple matches raise instead of guessing. Passing one exact filename remains unchanged.

Details

• Adds --platform and --ruby-abi selectors. Given multiple SHA-named files, RubyGems reads each specification and selects the one whose platform and required_ruby_version satisfy both selectors.
• ruby_matches? does not check platform (a RUBY-platform gem with a matching ~> X.Y.0 can be selected by --ruby-abi); this is documented in the tests rather than special-cased.

gem yank

ref: Shopify/rubygems#176.

User behaviour

• gem yank mygem -v 1.0.0 --platform x86_64-linux --ruby-abi 3.4 sends gem name, version, platform, and ABI so the server can select one skinny variant.
• Existing yank behaviour is unchanged without --ruby-abi.

Remote queries (gem list / search / info -r)

ref: Shopify/rubygems#175.

User behaviour

• Shows the real platform and Ruby ABI, not the hash. Versions and platforms are separated; ABIs for the same version/platform are grouped. Examples:

$ gem list -r mygem        # single CA gem on the server
mygem (1.0 Platform: x86_64-linux, Ruby ABI: 3.3)

$ gem search -r mygem      # same version, different platforms/ABIs
mygem (1.0 Platform: arm64-darwin, Ruby ABI: 3.4
         1.0 Platform: x86_64-linux, Ruby ABI: 3.3)

$ gem search -r mygem      # same version+platform, multiple ABIs grouped
mygem (1.0 Platform: x86_64-linux, Ruby ABI: 3.3, 3.4)

$ gem list -r mygem        # multiple versions, each on its own line
mygem (3.0 Platform: arm64-darwin, Ruby ABI: 3.4
         2.0 Platform: x86_64-linux, Ruby ABI: 3.4
         1.0 Platform: x86_64-linux, Ruby ABI: 3.3)

$ gem list -r mygem        # mixed CA, platform (1.0), and source gems
mygem (3.0 Platform: arm64-darwin, Ruby ABI: 3.4
         2.0 Platform: x86_64-linux, Ruby ABI: 3.3
         1.0 Platform: x86_64-linux)

$ gem info -r mygem        # info uses a Platforms: section, ABIs grouped per platform
mygem (1.0)
    Author: A User
    Homepage: http://example.com

    this is a summary

Platforms:
    x86_64-linux Ruby ABI: 3.3, 3.4

• Unscoped remote queries skip CA metadata (no platform/ABI rows injected).

bundle install (lockfile + local cache)

ref: Shopify/rubygems#177 (remote) and #178 (lockfile + local cache).

Remote

• Separates the compact-index hash from the real platform, includes the hash in candidate identity, selects an ABI-compatible skinny gem, and falls back to platform/source gems when needed.

Lockfile and local cache

• Writes platform and hash to Gemfile.lock and parses both on the next run.
• Installs the SHA-named artifact from vendor/cache with bundle install --local. Remote and local paths produce the same installed directory; checksums remain keyed by the platform lock name. A CA gem locks with the hash in the version and the real platform beside it.
  • The content address in the version token identifies the artifact, but says nothing about which platform it targets. The trailing platform is what Bundler uses for platform matching on the next run and to preserve the real platform identity when materializing from vendor/cache

GEM
  remote: https://rubygems.org/
  specs:
    content_addressable_test (0.1.0-a56253e0) arm64-darwin

PLATFORMS
  arm64-darwin

DEPENDENCIES
  content_addressable_test

CHECKSUMS
  content_addressable_test (0.1.0-a56253e0) sha256=a56253e023bdae7b291f44ff21a02dd12f2609418eb60ad22327a7ae5cd4f353

BUNDLED WITH
   4.1.0.dev

[tenderlove]

I didn't review the specs super closely, I trust they are covering useful scenarios.

It looks like we've got a lot of array / hash manipulation going on in this PR. Should we be thinking about making real, named objects?

The direction looks good here IMO

[tenderlove]

No impact on this PR, but this is a bad hash and we should fix it upstream.

We should be doing [@set, @name, ...].hash

[OughtPuts]

Fixed!

[tenderlove]

👍

[tenderlove]

This is because it's now a hash of hashes? What is the structure of platforms?

[OughtPuts]

platforms comes from line 213 in output_versions. It's a one-level hash

platforms = Hash.new {|h,version| h[version] = [] }

In reality it would look something like this:

  {
    Gem::Version.new("1.0.0") => [
      Gem::Platform::RUBY,
      Gem::Platform.new("x86_64-linux"),
      Gem::Platform.new("x86_64-linux")
    ],
    Gem::Version.new("0.9.0") => [
      Gem::Platform::RUBY
    ]
  }

And then `transform_values(&:uniq) would remove any dups in the values.

[tenderlove]

This logic is because we're using this same loop with the CA and non-CA RubyGems endpoints?

[OughtPuts]

Yep. The compact-index /versions response uses the same suffix field for both formats, legacy entries contain a platform, while CA entries contain a content-address token. This handles both, so it detects CA suffixes and stores them as content_address. The actual platform is decoded later from the gem’s /info metadata!

[tenderlove]

It feels like we should make a real object here. Just spitballing but like:

class GemInfo < Struct.new(:version, :suffix, :platform, :ruby_abi)
  def hash; suffix; end
  def eql?(other); other.version == version && other.suffix == suffix; end
end

Though now that I type this out, it seems very similar to NameTuple? It feels like we could be doing more simple code here with set intersections. e.g. wanted_rows.map { make_obj(_1) } & compact_index_info_rows(name).map { make_object(_1) }

[OughtPuts]

@tenderlove have implemented a solution in this commit, interested in your thoughts!