Skip to content

Export alignment spectrum provenance links - #791

Open
htsugawa wants to merge 3 commits into
masterfrom
feature/spectrum-provenance-export
Open

Export alignment spectrum provenance links#791
htsugawa wants to merge 3 commits into
masterfrom
feature/spectrum-provenance-export

Conversation

@htsugawa

@htsugawa htsugawa commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Adds two Console text exports that make an alignment spot traceable back to the per-file peak it came from: AlignResult-*.mdpeakid.tsv, a compact matrix from each MasterAlignmentID and sample to the source MasterPeakID, and the optional AlignResult-*.mdprovenance.tsv audit sidecar behind a new Detailed alignment provenance config key. Without them there is no way to say which .mdpeak row a given aligned value came from, which is the link the companion catalog needs in order to attach an annotation claim to the evidence behind it.

This branch has been sitting unmerged since 2026-09-01 and had fallen ten commits behind, so it is rebased onto current master (f36aea416).

The rebase

One textual conflict, in ConfigParser.cs, where both sides added a new reader method at the same position: master added ReadLbmAnnotatorPriority (#783), the branch adds ReadDetailedAlignmentProvenance. Both are needed, so both are kept side by side, each verbatim as its own side wrote it. LcmsProcess.cs auto-merged because the two sides touch different regions — master the annotator setup, the branch the export block and the ExecuteAsync signature.

ConfigParser.cs uses method.ToLower() nine times and ToLowerInvariant() once, the once being master's new method. Each side is left as written: a rebase should not smuggle in a normalization neither side authored. The nine ToLower() sites are a real latent bug — under a Turkish or Azeri culture 'I'.ToLower() gives dotless ı, so a config key silently stops matching and the run proceeds with the default — but they belong in their own change.

The rebase changes no existing output

Measured, not reasoned. Same demo, same method file, this branch against plain master:

Output Result
7 per-file .mdpeak, 7 per-file .mdmsp byte-identical
.mdalign, alignment .mdmsp, .qa.tsv byte-identical
.mzTab differs only in the MTD mzTab-ID line, which is the run timestamp
Extra files .mdpeakid.tsv and .mdprovenance.tsv, nothing else

The alignment spot count on this demo moved from 2512 to 2553 relative to the older checked-in reference output. That is master's, not this branch's: #773 made sample max / blank average also set FoldChangeForBlankFiltering, which changes blank filtering, which runs at the alignment stage. Peak-detection columns are unchanged in every row — Peak ID, RT (min), Precursor m/z, Height, Area, S/N, Scan all show zero differing rows — and the .mdpeak differences are confined to the annotation block, from #783, #784, #785 and #789.

Three things the sidecar asserted without measuring

Found by review of the rebase and fixed in 7a7b8c28d. All three were invisible to the original tests, because both asserted substrings of a row that had a real source peak and stopped before the mz column.

mz was backwards in every row. The in-memory overload read ChromXsTop.Mz, the light overload read Mass. On an aligned peak ChromXsTop carries the chromatogram axis and the gap filler resets it, so the column reported a sentinel for every member that did have a source peak and a real value only for the gap-filled ones. On the FastLC demo: 7879 of 7879 has_source_peak=true rows carried mz=-1, and 9992 of 9992 gap-filled rows carried a plausible m/z. Both overloads now read Mass, which is what the .mdalign MZ column uses.

Gap-filled rows published a false raw-scan pointer. The gap filler resets the peak ids but leaves the spectrum ids at their default 0, and 0 is a real scan index. Every gap-filled row therefore claimed ms1_raw_spectrum_id=0, ms1_raw_spectrum_id_top=0 and ms2_raw_spectrum_id=0. In a file whose purpose is provenance that points an auditor at a spectrum belonging to a different peak. All five raw-spectrum columns are now withheld when there is no source peak, in both overloads. The light overload had additionally gated ms2_raw_spectrum_ids on MS2RawSpectrumID >= 0, which is true for that same 0.

The two files disagreed about the same cell. The sidecar wrote -2 for a gap-filled peak id where the matrix wrote -1. They now agree on -1, and the distinction is preserved by name in a new peak_origin column valued detected, gap_filled or absent. Same lesson as #785: name the state instead of encoding it in a magic number.

One behavioural change to existing output, in alignment-light mode only

The QA-matrix block's six accessors move from an unconditional new LegacyQuantValueAccessor(...) to a factory that returns AlignmentLightQuantValueAccessor when the light store is in use. This is a bug fix rather than part of the export: in light mode each spot holds only its representative peak, so LegacyQuantValueAccessor, which iterates AlignedPeakProperties, emitted one column's worth of values for the whole matrix. Flagging it explicitly because the commit subject does not suggest the QA matrix would change.

That switch also exposed a divergence in the light-mode Reference matched column, fixed here: it derived from RepresentativeLibraryID >= 0, but a suggested annotation carries a library id too, so a precursor-only or low-score suggestion counted as a reference match in light mode while normal mode reported false through AnnotationName.IsReferenceMatched, the predicate #784 centralised. The store now persists the representative's own IsReferenceMatched verdict. RecordSize goes 102 to 103; the file is a per-run temporary spill from Path.GetTempFileName, so the layout carries no cross-version obligation.

Verification

  • Build: Release, net48, 0 errors.
  • MsdialCoreTests: 316/316 (311 before, plus five new covering gap-filled and unwritten-light rows, the mz-from-Mass rule, and the reference-matched verdict).
  • MsdialCoreTestAppTests: 13/13, including a new ReadDetailedAlignmentProvenance test next to master's ReadLbmAnnotatorPriority test. That is the project that actually compiles the rebased ConfigParser, and running it was the one piece of evidence the original green result did not cover.
  • End to end on the FastLC demo, ingested into the companion catalog: every member with a source peak carries a usable m/z, no member without one carries a scan index, and peak_origin splits 17871 members into 7879 detected and 9992 gap_filled.

Two guards so this class of defect cannot return quietly: WriteMember now goes through a WriteRow that refuses a field count the header does not declare, since the header and the two value lists are three hand-maintained arrays; and the tests assert whole lines rather than substrings.

Deliberately not in this PR

  • Gating .mdpeakid.tsv behind a config flag. It is currently unconditional, and in light mode it adds an O(spots x files) seek-per-cell pass — which is exactly the regime light mode exists for. Worth a follow-up; the matrix is the authoritative join key, so defaulting it off would need thought.
  • Converting the nine method.ToLower() sites to the invariant form.
  • Splitting the QA-accessor fix into its own commit.

🤖 Generated with Claude Code

DESKTOP-382ETUR\Hiroshi Tsugawa and others added 3 commits September 4, 2026 23:36
Review of the rebased branch found the audit sidecar reporting three things that
were not measurements, all of them invisible to the existing tests because both
tests asserted substrings of a row that had a real source peak, and stopped
before the mz column.

The mz column was read from ChromXsTop.Mz in the in-memory overload and from
Mass in the light overload. On an aligned peak ChromXsTop carries the
chromatogram axis, and the gap filler resets it outright, so the column reported
a sentinel for every member that DID have a source peak and a real value only
for the gap-filled ones. Measured on the FastLC demo: 7879 of 7879 rows with
has_source_peak=true carried mz=-1, and 9992 of 9992 gap-filled rows carried a
plausible m/z. Exactly backwards, in every row. Both overloads now read Mass,
which is also what the .mdalign MZ column uses.

A member with no source peak has no source spectrum either, but the gap filler
leaves the spectrum ids at their default 0 while resetting only the peak ids, so
every gap-filled row published ms1_raw_spectrum_id=0, ms1_raw_spectrum_id_top=0
and ms2_raw_spectrum_id=0. A 0 is a real scan index. In a file whose whole
purpose is provenance, that points an auditor at a spectrum belonging to some
other peak. All five raw-spectrum columns are now withheld when there is no
source peak, in both overloads; the light overload had additionally gated
ms2_raw_spectrum_ids on MS2RawSpectrumID >= 0, which is true for that same 0.

The peak-id columns published -2 for a gap-filled cell while the compact peak-ID
matrix published -1 for the same cell, so two files shipped by one feature
disagreed numerically. They now agree on -1, and the distinction the raw values
carried is preserved by name in a new peak_origin column, valued detected,
gap_filled or absent. That is the same lesson as PR #785: name the state rather
than encode it in a magic number, so a reader does not need the gap filler's
internals to interpret a cell.

Also fix the light-mode QA matrix's "Reference matched" column. The switch to
the file-backed quant accessor made light mode derive it from
RepresentativeLibraryID >= 0, but a suggested annotation carries a library id
too, so a precursor-only or low-score suggestion counted as a reference match in
light mode while normal mode reported false. Normal mode resolves the column
through AnnotationName.IsReferenceMatched, the predicate PR #784 centralised.
The store now persists the representative's own IsReferenceMatched verdict and
the accessor reads that. RecordSize goes 102 to 103; the file is a per-run
temporary spill created by Path.GetTempFileName, so the layout carries no
cross-version obligation.

Two guards so this class of defect cannot return silently. WriteMember now goes
through a WriteRow that refuses a field count the header does not declare, since
the header and the two value lists are three hand-maintained arrays. And the
tests assert whole lines rather than substrings, and cover a gap-filled row and
an unwritten light row in both overloads, which is the case that hid all three
divergences.

Build: Release net48, 0 errors. Tests: MsdialCoreTests 316/316 (311 before, plus
five new), MsdialCoreTestAppTests 13/13 including a new
ReadDetailedAlignmentProvenance test alongside master's ReadLbmAnnotatorPriority
test in the project that actually compiles the rebased ConfigParser.

Verified end to end on the FastLC demo, ingesting the output into the companion
msdial_spectrum_catalog: every member with a source peak now carries a usable
m/z, no member without one carries a scan index, and peak_origin splits the
17871 members into 7879 detected and 9992 gap_filled.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant