Line-aligned emission: source-true backtraces and debuggers by construction

.github/workflows/ci.yml

@@ -22,7 +22,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        ruby: ['3.2', '3.3', '4.0']
+        ruby: ['3.3', '3.4', '4.0']
     steps:
       - uses: actions/checkout@v4
       - uses: ruby/setup-ruby@v1

.gitignore

@@ -50,4 +50,7 @@ build-iPhoneSimulator/
 .rvmrc
 
 # RubyMine
-/.idea
+/.idea
+
+# Generated per-machine by `dev up` (d3mlabs dev tool); never committed
+/.shadowenv.d/

.ruby-version

@@ -0,0 +1 @@
+4.0.6

CHANGELOG.md

@@ -4,6 +4,22 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [3.0.0] - Unreleased
+### Added
+- Line-aligned emission: transformed code is emitted with every loc-carrying statement on its original source line, making backtraces, breakpoints, and debugger display correct by construction (`LineAlignedEmitter`).
+- Authoring toolkit in `TransformationHelper`: `s_at` (loc-anchored node construction), `thunk` (a single invariant-checked `Thunk` node spliced at the execution point; the lowering derives the hidden proc's textual placement from the body's source locations), and `run_after` (sequence-level execution reordering that preserves textual/source order). Thunks lower to a non-lambda proc, so `return` still returns from the enclosing method, and locals assigned by thunked statements are pre-declared to stay method-scope. Reusing one thunk node executes its body from several points.
+- `ASTTransform::Node.register`: type-routed construction of custom IR node classes through `s`, with an emitter postcondition (`LineAlignedEmitter::UnloweredNodeTypeError`) rejecting custom types that were not lowered before emission.
+- `ast_transform/testing/assertions` (test-only): `assert_line_aligned` and `assert_backtrace_lines` for transform authors' suites.
+- Error types, each owned by its producer: `TransformationHelper::MissingLocationError`, `ThunkLowering::PlacementError`, `LineAlignedEmitter::UnloweredNodeTypeError`. Thunk construction invariants raise plain `ArgumentError`.
+
+### Removed
+- **Breaking:** `ASTTransform::SourceMap` and source-map registration. Line-aligned emission makes raw VM line numbers the source line numbers, so there is nothing left to map at display time.
+
+### Changed
+- **Breaking:** `Transformer#transform` and `#transform_file_source` emit line-aligned output (source-anchored layout, always newline-terminated) instead of Unparser's re-normalized formatting.
+- **Breaking:** dropped Ruby 3.2 support (EOL since March 2026); `required_ruby_version` is now `>= 3.3`.
+- Dependency floors now reflect reality: `unparser >= 0.8` (the emitter uses `static_local_variables:`, a 0.7 interface, and 0.8's prism-based round-trip verification is required for Ruby >= 3.4 syntax) and `parser >= 3.3` (unparser's own floor; the declared `>= 3.0` could never resolve lower).
+
 ## [0.1.4] 2019-06-20
 ### Fixed
 - Source mapping for transformations wrapping source nodes into virtual nodes now work.

Gemfile

@@ -7,6 +7,11 @@ git_source(:github) { |repo_name| "https://github.com/#{repo_name}" }
 # Specify your gem's dependencies in ast_transform.gemspec
 gemspec
 
-# Transitive dependency of rubocop: parallel >= 2 requires Ruby >= 3.3, but we
-# still support 3.2. Drop this pin when our Ruby floor moves to 3.3.
-gem "parallel", "< 2", require: false
+# Development dependencies
+gem "bundler", ">= 2.1"
+gem "minitest", "~> 5.14"
+gem "minitest-reporters", "~> 1.4"
+gem "pry", ">= 0.14"
+gem "rake", "~> 13.0"
+gem "rubocop-shopify", "~> 3.0", require: false
+gem "simplecov", "~> 0.22"

Gemfile.lock

@@ -1,33 +1,33 @@
 PATH
   remote: .
   specs:
-    ast_transform (2.1.4)
-      parser (>= 3.0)
+    ast_transform (3.0.0)
+      parser (>= 3.3)
       prism (>= 1.5)
-      unparser (>= 0.6)
+      unparser (>= 0.8)
 
 GEM
   remote: https://rubygems.org/
   specs:
-    ansi (1.5.0)
+    ansi (1.6.0)
     ast (2.4.3)
     builder (3.3.0)
     coderay (1.1.3)
-    diff-lcs (1.6.2)
+    diff-lcs (2.0.0)
     docile (1.4.1)
     io-console (0.8.2)
     json (2.21.1)
     language_server-protocol (3.17.0.6)
     lint_roller (1.1.0)
     method_source (1.1.0)
     minitest (5.27.0)
-    minitest-reporters (1.7.1)
+    minitest-reporters (1.8.0)
       ansi
       builder
-      minitest (>= 5.0)
+      minitest (>= 5.0, < 7)
       ruby-progressbar
-    parallel (1.28.0)
-    parser (3.3.10.2)
+    parallel (2.1.0)
+    parser (3.3.12.0)
       ast (~> 2.4.1)
       racc
     prism (1.9.0)
@@ -37,7 +37,7 @@ GEM
       reline (>= 0.6.0)
     racc (1.8.1)
     rainbow (3.1.1)
-    rake (13.3.1)
+    rake (13.4.2)
     regexp_parser (2.12.0)
     reline (0.6.3)
       io-console (~> 0.5)
@@ -55,8 +55,9 @@ GEM
     rubocop-ast (1.50.0)
       parser (>= 3.3.7.2)
       prism (~> 1.7)
-    rubocop-shopify (2.18.0)
-      rubocop (~> 1.62)
+    rubocop-shopify (3.0.1)
+      lint_roller
+      rubocop (~> 1.72, >= 1.72.1)
     ruby-progressbar (1.13.0)
     simplecov (0.22.0)
       docile (~> 1.1)
@@ -67,8 +68,8 @@ GEM
     unicode-display_width (3.2.0)
       unicode-emoji (~> 4.1)
     unicode-emoji (4.2.0)
-    unparser (0.8.1)
-      diff-lcs (~> 1.6)
+    unparser (0.9.0)
+      diff-lcs (>= 1.6, < 3)
       parser (>= 3.3.0)
       prism (>= 1.5.1)
 
@@ -81,10 +82,9 @@ DEPENDENCIES
   bundler (>= 2.1)
   minitest (~> 5.14)
   minitest-reporters (~> 1.4)
-  parallel (< 2)
   pry (>= 0.14)
   rake (~> 13.0)
-  rubocop-shopify (~> 2.18)
+  rubocop-shopify (~> 3.0)
   simplecov (~> 0.22)
 
 BUNDLED WITH

README.md

@@ -5,6 +5,8 @@
 
 ASTTransform is an Abstract Syntax Tree (AST) transformation framework. It hooks into the compilation process and allows to perform AST transformations using an annotation: `transform!`.
 
+Transformed code is emitted **line-aligned**: every statement carrying a source location is placed on its original source line. Backtraces, failure messages, `break file:line` breakpoints, and debugger display are therefore correct by construction — no source maps, no backtrace filtering, no debugger integration required.
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -160,6 +162,54 @@ In the above, `node#updated` allows updating the node, either its type or its ch
 
 The [ast gem](https://github.com/whitequark/ast) uses a pattern in which a Transformation may implement a method matching a node type, i.e. `on_class`, `on_send`, `on_lvar`, etc... This is very useful when transformations should process all nodes of this type.
 
+### Line-aligned emission and the authoring contract
+
+ASTTransform owns text and lines; transform authors own semantics and execution order. The contract:
+
+* A node **with** a source location is emitted at that location's line (the emitter pads with blank lines to reach it, and packs with `;` when a line is already occupied).
+* A node **without** a source location is synthetic: it packs onto the current line and inherits its neighbors' line number.
+* Textual order is source order. If your transform needs code to *execute* in a different order than it *appears*, use a thunk (below) instead of moving nodes.
+
+`ASTTransform::TransformationHelper` (included by `AbstractTransformation`) provides the authoring toolkit:
+
+* `s(type, *children)` — builds a loc-less (synthetic) node. Registered custom types (see below) construct their registered class.
+* `s_at(anchor, type, *children)` — builds a node anchored at `anchor`'s source location, so it is emitted at `anchor`'s line. Raises `TransformationHelper::MissingLocationError` if the anchor has no location.
+* `thunk(*statements)` — wraps statements in a single `Thunk` node: splice it wherever the statements must *run*, in statement position or composed inside an expression (e.g. an `assert_raises` block body). The wrapped statements keep their own locations, and the lowering derives the hidden proc's textual placement from them — the body still emits on its source lines even though execution waits. Reuse the same node to execute one body from several points. Thunk construction is invariant-checked (`ArgumentError` on a missing id or empty body); a body whose source lines fall after its execution point fails lowering with `ThunkLowering::PlacementError` (a thunk can only delay execution, never text).
+* `run_after(statements, run:, after:)` — the paved road over `thunk`: returns a reordered copy of `statements` where the contiguous `run` executes after `after`, while remaining at its source position textually. Elements are matched by object identity.
+
+Thunked statements keep their original meaning as far as Ruby's closure semantics allow:
+
+* `return` still returns from the enclosing method — the hidden closure is a non-lambda proc, and the proc and its call always share one method activation (placements never cross a `def` boundary).
+* Locals assigned by the thunked statements stay method-scope: the lowering pre-declares each one (`result = result`) before the proc, so code after the execution point can read them. Before the thunk runs they are `nil` — exactly what an unexecuted assignment yields.
+* Jump keywords whose owner lies *outside* the thunked statements keep Ruby's native behavior: `break`/`retry` raise `LocalJumpError` at the jump's own source line, while `next`/`redo` silently end or restart the thunk body. ASTTransform does not validate this — what a transform surface allows users to thunk is the transform author's call.
+
+#### Gotcha: location-only rewrites are silently dropped
+
+`Parser::AST::Node#updated` returns `self` when the new children compare `==` to the old ones — and `AST::Node#==` ignores source locations. A `Processor` pass that replaces a node with an equal-valued one (e.g. the same call rebuilt loc-less, hoping to change its emitted line) is a no-op: every `node.updated(nil, process_all(node))` up the tree discards the replacement. Location is part of a node's *emission*, not its *value* — to change where a node emits, change what it is (`s_at` an anchored rebuild with different children), or restructure the parent explicitly rather than relying on `updated`.
+
+#### Custom node types (intermediate representation)
+
+Transformations that parse a DSL can build their own IR by registering node classes:
+
+```ruby
+class InteractionNode < ASTTransform::Node
+  register :my_interaction
+
+  def cardinality = children[0]
+end
+
+s(:my_interaction, ...) # => InteractionNode, with domain accessors
+```
+
+Custom node types are IR **between stages that understand them** — the stage that owns a type must lower it to plain Ruby nodes before emission. The emitter enforces this: any registered or `ast_`-prefixed type reaching emission raises `LineAlignedEmitter::UnloweredNodeTypeError`.
+
+#### Testing your transformation
+
+`require 'ast_transform/testing/assertions'` (test-only) provides `ASTTransform::Testing::Assertions`, a Minitest-flavored module to include in your test class:
+
+* `assert_line_aligned(source, *transformations)` — transforms `source` through the real pipeline and asserts every surviving statement is emitted at its source line.
+* `assert_backtrace_lines(source, path:, raise_at:)` — compiles and executes `source`, asserting the raw first backtrace frame cites `path:raise_at` with no filtering.
+
 ### Parameterizable transformations
 
 If you want your transformation to be customizable, accept the parameters in the constructor. The annotation can the be changed accordingly:
@@ -183,6 +233,8 @@ end
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
+If you use the [d3mlabs dev tool](https://github.com/d3mlabs/dev), `dev up` provisions the pinned Ruby (see `.ruby-version`) with a per-project shadowenv, and `dev test` runs the suite — plain Bundler as above works just as well.
+
 To install this gem onto your local machine, run `bundle exec rake install`.
 
 ## Releasing a New Version

ast_transform.gemspec

@@ -20,20 +20,16 @@ Gem::Specification.new do |spec|
   spec.bindir        = "exe"
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.require_paths = ["lib"]
-  spec.required_ruby_version = ">= 3.2"
+  spec.required_ruby_version = '>= 3.3'
 
-  # Development dependencies
-  spec.add_development_dependency("bundler", ">= 2.1")
-  spec.add_development_dependency("minitest", "~> 5.14")
-  spec.add_development_dependency("minitest-reporters", "~> 1.4")
-  spec.add_development_dependency("pry", ">= 0.14")
-  spec.add_development_dependency("rake", "~> 13.0")
-  # rubocop-shopify >= 3.0 requires Ruby >= 3.3; bump alongside our own floor.
-  spec.add_development_dependency("rubocop-shopify", "~> 2.18")
-  spec.add_development_dependency("simplecov", "~> 0.22")
+  # Development dependencies live in the Gemfile (Gemspec/DevelopmentDependencies).
 
   # Runtime dependencies
-  spec.add_runtime_dependency("parser", ">= 3.0")
-  spec.add_runtime_dependency("prism", ">= 1.5")
-  spec.add_runtime_dependency("unparser", ">= 0.6")
+  # parser provides the runtime AST vocabulary (Parser::AST::Node/Processor,
+  # Source::Buffer/Map); parsing itself goes through prism's translation layer.
+  spec.add_runtime_dependency "parser", ">= 3.3"
+  spec.add_runtime_dependency "prism", ">= 1.5"
+  # unparser >= 0.8: static_local_variables: (0.7 interface) + the prism-based
+  # round-trip verification parser required for Ruby >= 3.4 syntax.
+  spec.add_runtime_dependency "unparser", ">= 0.8"
 end

dependencies.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+# Toolchain-only manifest for d3mlabs' dev tool: it provisions this exact
+# Ruby (rbenv + shadowenv) for `dev` commands. Gems stay bundler-managed
+# through the hand-written gemspec/Gemfile; contributors without dev can
+# ignore this file and use .ruby-version.
+require "dev/deps"
+
+Dev::Deps.define do
+  ruby "4.0.6"
+end

lib/ast_transform/abstract_transformation.rb

@@ -23,6 +23,14 @@ def process(node)
       process_node(node)
     end
 
+    # Thunks are framework-owned IR: descend into the body so passes that don't know about thunks still process the
+    # wrapped statements. Without this, Processor's handler_missing default would pass the node through opaquely,
+    # hiding the body from every later transformation. The token (first child) is not a node and passes through
+    # untouched.
+    def on_ast_thunk(node)
+      node.updated(nil, [node.children[0], *process_all(node.children.drop(1))])
+    end
+
     private
 
     # Processes the given +node+.

lib/ast_transform/kwargs_builder.rb

@@ -3,14 +3,16 @@
 require "prism/translation/parser"
 
 module ASTTransform
-  # Extends the default Prism parser builder to distinguish keyword arguments
-  # from hash literals in the AST.
+  # Extends the default Prism parser builder to distinguish keyword arguments from hash literals in the AST.
   #
-  # The upstream builder always emits :hash nodes for both `foo(bar: 1)` and
-  # `foo({ bar: 1 })`. Unparser uses the node type to decide whether to emit
-  # braces: :hash gets `{}`, :kwargs does not. Since Ruby 3.0+ treats these as
-  # semantically different (strict keyword/positional separation), we need the
-  # AST to preserve the distinction.
+  # The upstream builder always emits :hash nodes for both `foo(bar: 1)` and `foo({ bar: 1 })`. Unparser uses the
+  # node type to decide whether to emit braces: :hash gets `{}`, :kwargs does not. Since Ruby 3.0+ treats these as
+  # semantically different (strict keyword/positional separation), we need the AST to preserve the distinction.
+  #
+  # NOTE: parsed nodes deliberately stay plain Parser::AST::Node. Custom node classes exist only for registered
+  # custom types (see ASTTransform::Node), which are IR and never reach Unparser: AST::Node#eql? compares class, and
+  # Unparser verifies dynamic-string emission by re-parsing and comparing eql? against the freshly parsed
+  # (plain-class) node — custom-class nodes of standard types would fail that verification.
   class KwargsBuilder < Prism::Translation::Parser::Builder
     def associate(begin_t, pairs, end_t)
       node = super

lib/ast_transform/layout.rb

@@ -0,0 +1,64 @@
+# frozen_string_literal: true
+
+module ASTTransform
+  # Line-addressed output: text is placed at absolute line numbers, top to bottom, and the cursor never rewinds.
+  # When a placement's target line is already behind the cursor, the text is packed (`; `) onto the current line
+  # instead — Ruby lets statements share a physical line, so alignment degrades locally and the next placement
+  # whose target is still ahead re-anchors. Knows nothing about Ruby structure or ASTs; callers decide WHAT goes
+  # on WHICH line, the layout owns the pad-or-pack mechanics.
+  class Layout
+    def initialize
+      @lines = []
+    end
+
+    # The line number currently being written; the next fresh line would be +cursor + 1+.
+    def cursor
+      @lines.size
+    end
+
+    # Places +text+ at +target_line+ when the cursor hasn't passed it; otherwise packs onto the current line.
+    # Multi-line text advances the cursor by its height. When opening a fresh line, the first line is indented to
+    # +column+ — cosmetic only (leading whitespace is never significant in emitted code), but it keeps the artifact
+    # visually close to the source. Packed text ignores the column, as do continuation lines (they keep their own
+    # relative indentation).
+    def place(target_line, text, column: nil)
+      first, *rest = text.split("\n")
+
+      if target_line && target_line > @lines.size
+        @lines << '' while @lines.size < target_line
+        @lines[-1] = indented(first, column)
+      else
+        pack(first)
+      end
+
+      @lines.concat(rest)
+    end
+
+    # Appends +text+ on a new line unconditionally — for text that must never be `;`-packed after a statement
+    # (e.g. keywords).
+    def place_on_fresh_line(text)
+      @lines << text
+    end
+
+    # Appends +text+ to the current line with a `; ` separator. The last line is never blank here: padding blanks
+    # are only created inside +place+, which immediately overwrites the padded line.
+    def pack(text)
+      if @lines.empty?
+        @lines << text
+      else
+        @lines[-1] = "#{@lines.last}; #{text}"
+      end
+    end
+
+    # @return [String] the laid-out text, with a trailing newline.
+    def to_source
+      "#{@lines.join("\n")}\n"
+    end
+
+    private
+
+    def indented(text, column)
+      column && column.positive? ? "#{' ' * column}#{text}" : text
+    end
+  end
+end