Release v0.6 - #233
Open
conradbzura wants to merge 12 commits into
Open
Conversation
Table validated coordinate_system and interval_type in __post_init__ but was a plain dataclass, so assigning either field afterwards installed a value the constructor rejects. The value is not inert: it selects the coordinate translation the emitted SQL performs. Setting it to a third string produced an ON predicate that is neither the 0-based form nor the 1-based one but a half-shifted hybrid matching no coordinate system, and transpile raised nothing. Freezing makes __post_init__ the single way in, so validation holds for the object's whole lifetime rather than only at construction. Callers vary a config with dataclasses.replace, which re-runs that validation. Immutability also restores __hash__, which the generated __eq__ had set to None. Separately, _build_tables duck-typed every non-str entry for a .name attribute. An arbitrary object therefore reached pass 1 and failed there with an AttributeError naming an internal column attribute, telling the caller nothing about which argument was wrong. It now rejects the entry with a TypeError naming the offending type. Adds tests for the two field validations, which had none.
Target, Capabilities, GenericTarget, DuckDBTarget and DataFusionTarget are all exported from the package root and autodocumented, yet the only public function that consumes a target rejected every one of them and reported the object's repr as though it were a misspelled name. Selecting a target by name stays the documented default and is the right seam for the plugin-distribution case, where a package ships a target and users select it without importing it. What that does not cover is the one-off: making a bespoke Target selectable meant mutating the process-global registry, and the registration outlived the call. The object path removes that side effect and is purely additive. GenericTarget is now accepted as an instance while the name "generic" still raises. That asymmetry is deliberate: None remains the one public spelling for the generic target, but an instance is unambiguous. Also drops the three transpile overloads. All three returned str and the widest admitted a bare str, so the set collapsed to the implementation signature and taught a type checker nothing. Their stated purpose was editor completion of the built-in dialect names, which the DialectName literal alias in the signature preserves without three public typing artifacts.
DuckDB's IE_JOIN is INNER-only, so a LEFT or RIGHT outer join carrying a column-to-column INTERSECTS fell through to the naive predicate: a hash join on chrom with the position inequalities as a residual filter, which is quadratic when the chromosome key has low cardinality. The query is now rewritten as a UNION ALL of an INNER half for the matched pairs, a NOT EXISTS half for the preserved side's unmatched rows, and a third branch for its NULL-chromosome rows. The first two reach the fast operator; the third is a filtered scan, and it is load-bearing rather than defensive, because both partitions come from SELECT DISTINCT chrom where a NULL renders as a NULL literal that string_agg skips. RIGHT is served by swapping the FROM and joined tables so one LEFT-shaped path covers both. Shapes the rewrite cannot express decline as one unit. Two properties of the emission decide whether it is worth taking, and both are settled by execution rather than by plan inspection, which reports IE_JOIN either way. The first is contig cardinality. One UNION ALL branch is emitted per distinct chromosome, so cost tracks that count while the plain predicate's does not: measured at 262,144 rows per side, the partitioned form runs 0.79s against 4.53s naive at 24 contigs and 57.2s against 0.11s at 3,000. The partition's cardinality is a property of the data, so the choice is made at execution time by a CASE over the partition's own row count, above which the same query binds with the chromosome equality inlined instead. The unmatched half also partitions on the chromosome INTERSECT rather than the preserved side's distinct chromosomes, carrying its left-only chromosomes in one non-partitioned branch: those rows cannot match, so a branch apiece scans both tables to prove an emptiness the partition already knows. The second is session state. Each half declares a DuckDB session variable, and the emitted script cannot release them because the final statement has to be the SELECT. Names are therefore a digest of the variable's own rendered value, which bounds a session's variable set by the number of distinct query shapes rather than the number of calls; naming them per call retained 26 to 84 MB after 50 to 200 queries. The naive-predicate fallback resolves through the registry rather than calling the built-in directly, so a user expander registered on (GenericTarget, Intersects) reaches the shapes this target declines instead of applying under dialect=None alone.
The bedtools lane declared its integration marker in conftest.py, where pytest does not honour pytestmark, leaving all of its tests unmarked. Both documented selection commands therefore inverted: running with the integration marker skipped the whole lane, and running without it pulled in the lane and its bedtools and pybedtools dependencies. CI was unaffected because it runs the suite with no marker filter, so nothing surfaced it. Each module now declares the marker itself, matching the datafusion lane's working convention. The cross-target oracle asserted only that the three targets agree, which it would continue to do if duckdb silently stopped decomposing. It now also pins which plan duckdb took, through a target-to-SQL map the oracle fixture exposes. Adds bedtools oracles for the RIGHT outer join, expressed as the left outer join with the operands swapped, and for duplicate input rows, the multiplicity axis a UNION ALL rewrite is most likely to break and which the property lane could not reach because it draws from unique inputs.
The dialect parameter promised that an unqualified projection raises at transpile time. That holds for the INNER, SEMI and ANTI shapes and not for the outer joins, which decline silently to the naive predicate instead, so one stated rule covered two behaviours with no way for a caller to tell which applied. The same paragraph carried an inline list of declined shapes that had drifted behind the guide it points at, so the list is gone and the pointer stays: a second copy of an enumeration only drifts again. Returns said the result was a SQL query. Under duckdb an accelerated join returns a multi-statement script, and a driver that splits statements or forwards only the last drops the variable the SELECT reads and yields empty results. The partition-count ceiling decides at execution time whether a query takes the per-chromosome form at all, and it is the difference between winning and losing by orders of magnitude on a scaffold-level assembly. The performance guide now carries the measured crossover and the reasoning behind the bound, rather than generalising from a single low-contig measurement to a claim of large speedups at scale. Two further claims are restated from measurements rather than intuition. The NULL-chromosome branch, described as costing a linear scan of the preserved table, is pruned by null statistics to a quarter of a percent of runtime on a base table. The per-chromosome LEFT JOIN comparison asserted an inflection near 1e5 rows that the recorded figures do not support; it now states the two points that were actually measured. Also documents what the session-variable token is, now that it addresses the variable's content rather than being random, and gives transform_to_sql its own contract in place of a description of its sibling: the script shape, the ValueError it raises, and that the query it is handed is never mutated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Auto-generated by the cut release workflow.