Export alignment spectrum provenance links - #791
Open
htsugawa wants to merge 3 commits into
Open
Conversation
Review of the rebased branch found the audit sidecar reporting three things that were not measurements, all of them invisible to the existing tests because both tests asserted substrings of a row that had a real source peak, and stopped before the mz column. The mz column was read from ChromXsTop.Mz in the in-memory overload and from Mass in the light overload. On an aligned peak ChromXsTop carries the chromatogram axis, and the gap filler resets it outright, so the column reported a sentinel for every member that DID have a source peak and a real value only for the gap-filled ones. Measured on the FastLC demo: 7879 of 7879 rows with has_source_peak=true carried mz=-1, and 9992 of 9992 gap-filled rows carried a plausible m/z. Exactly backwards, in every row. Both overloads now read Mass, which is also what the .mdalign MZ column uses. A member with no source peak has no source spectrum either, but the gap filler leaves the spectrum ids at their default 0 while resetting only the peak ids, so every gap-filled row published ms1_raw_spectrum_id=0, ms1_raw_spectrum_id_top=0 and ms2_raw_spectrum_id=0. A 0 is a real scan index. In a file whose whole purpose is provenance, that points an auditor at a spectrum belonging to some other peak. All five raw-spectrum columns are now withheld when there is no source peak, in both overloads; the light overload had additionally gated ms2_raw_spectrum_ids on MS2RawSpectrumID >= 0, which is true for that same 0. The peak-id columns published -2 for a gap-filled cell while the compact peak-ID matrix published -1 for the same cell, so two files shipped by one feature disagreed numerically. They now agree on -1, and the distinction the raw values carried is preserved by name in a new peak_origin column, valued detected, gap_filled or absent. That is the same lesson as PR #785: name the state rather than encode it in a magic number, so a reader does not need the gap filler's internals to interpret a cell. Also fix the light-mode QA matrix's "Reference matched" column. The switch to the file-backed quant accessor made light mode derive it from RepresentativeLibraryID >= 0, but a suggested annotation carries a library id too, so a precursor-only or low-score suggestion counted as a reference match in light mode while normal mode reported false. Normal mode resolves the column through AnnotationName.IsReferenceMatched, the predicate PR #784 centralised. The store now persists the representative's own IsReferenceMatched verdict and the accessor reads that. RecordSize goes 102 to 103; the file is a per-run temporary spill created by Path.GetTempFileName, so the layout carries no cross-version obligation. Two guards so this class of defect cannot return silently. WriteMember now goes through a WriteRow that refuses a field count the header does not declare, since the header and the two value lists are three hand-maintained arrays. And the tests assert whole lines rather than substrings, and cover a gap-filled row and an unwritten light row in both overloads, which is the case that hid all three divergences. Build: Release net48, 0 errors. Tests: MsdialCoreTests 316/316 (311 before, plus five new), MsdialCoreTestAppTests 13/13 including a new ReadDetailedAlignmentProvenance test alongside master's ReadLbmAnnotatorPriority test in the project that actually compiles the rebased ConfigParser. Verified end to end on the FastLC demo, ingesting the output into the companion msdial_spectrum_catalog: every member with a source peak now carries a usable m/z, no member without one carries a scan index, and peak_origin splits the 17871 members into 7879 detected and 9992 gap_filled. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two Console text exports that make an alignment spot traceable back to the per-file peak it came from:
AlignResult-*.mdpeakid.tsv, a compact matrix from eachMasterAlignmentIDand sample to the sourceMasterPeakID, and the optionalAlignResult-*.mdprovenance.tsvaudit sidecar behind a newDetailed alignment provenanceconfig key. Without them there is no way to say which.mdpeakrow a given aligned value came from, which is the link the companion catalog needs in order to attach an annotation claim to the evidence behind it.This branch has been sitting unmerged since 2026-09-01 and had fallen ten commits behind, so it is rebased onto current
master(f36aea416).The rebase
One textual conflict, in
ConfigParser.cs, where both sides added a new reader method at the same position:masteraddedReadLbmAnnotatorPriority(#783), the branch addsReadDetailedAlignmentProvenance. Both are needed, so both are kept side by side, each verbatim as its own side wrote it.LcmsProcess.csauto-merged because the two sides touch different regions —masterthe annotator setup, the branch the export block and theExecuteAsyncsignature.ConfigParser.csusesmethod.ToLower()nine times andToLowerInvariant()once, the once beingmaster's new method. Each side is left as written: a rebase should not smuggle in a normalization neither side authored. The nineToLower()sites are a real latent bug — under a Turkish or Azeri culture'I'.ToLower()gives dotlessı, so a config key silently stops matching and the run proceeds with the default — but they belong in their own change.The rebase changes no existing output
Measured, not reasoned. Same demo, same method file, this branch against plain
master:.mdpeak, 7 per-file.mdmsp.mdalign, alignment.mdmsp,.qa.tsv.mzTabMTD mzTab-IDline, which is the run timestamp.mdpeakid.tsvand.mdprovenance.tsv, nothing elseThe alignment spot count on this demo moved from 2512 to 2553 relative to the older checked-in reference output. That is
master's, not this branch's: #773 madesample max / blank averagealso setFoldChangeForBlankFiltering, which changes blank filtering, which runs at the alignment stage. Peak-detection columns are unchanged in every row —Peak ID,RT (min),Precursor m/z,Height,Area,S/N,Scanall show zero differing rows — and the.mdpeakdifferences are confined to the annotation block, from #783, #784, #785 and #789.Three things the sidecar asserted without measuring
Found by review of the rebase and fixed in
7a7b8c28d. All three were invisible to the original tests, because both asserted substrings of a row that had a real source peak and stopped before themzcolumn.mzwas backwards in every row. The in-memory overload readChromXsTop.Mz, the light overload readMass. On an aligned peakChromXsTopcarries the chromatogram axis and the gap filler resets it, so the column reported a sentinel for every member that did have a source peak and a real value only for the gap-filled ones. On the FastLC demo: 7879 of 7879has_source_peak=truerows carriedmz=-1, and 9992 of 9992 gap-filled rows carried a plausible m/z. Both overloads now readMass, which is what the.mdalignMZ column uses.Gap-filled rows published a false raw-scan pointer. The gap filler resets the peak ids but leaves the spectrum ids at their default 0, and 0 is a real scan index. Every gap-filled row therefore claimed
ms1_raw_spectrum_id=0,ms1_raw_spectrum_id_top=0andms2_raw_spectrum_id=0. In a file whose purpose is provenance that points an auditor at a spectrum belonging to a different peak. All five raw-spectrum columns are now withheld when there is no source peak, in both overloads. The light overload had additionally gatedms2_raw_spectrum_idsonMS2RawSpectrumID >= 0, which is true for that same 0.The two files disagreed about the same cell. The sidecar wrote
-2for a gap-filled peak id where the matrix wrote-1. They now agree on-1, and the distinction is preserved by name in a newpeak_origincolumn valueddetected,gap_filledorabsent. Same lesson as #785: name the state instead of encoding it in a magic number.One behavioural change to existing output, in alignment-light mode only
The QA-matrix block's six accessors move from an unconditional
new LegacyQuantValueAccessor(...)to a factory that returnsAlignmentLightQuantValueAccessorwhen the light store is in use. This is a bug fix rather than part of the export: in light mode each spot holds only its representative peak, soLegacyQuantValueAccessor, which iteratesAlignedPeakProperties, emitted one column's worth of values for the whole matrix. Flagging it explicitly because the commit subject does not suggest the QA matrix would change.That switch also exposed a divergence in the light-mode
Reference matchedcolumn, fixed here: it derived fromRepresentativeLibraryID >= 0, but a suggested annotation carries a library id too, so a precursor-only or low-score suggestion counted as a reference match in light mode while normal mode reported false throughAnnotationName.IsReferenceMatched, the predicate #784 centralised. The store now persists the representative's ownIsReferenceMatchedverdict.RecordSizegoes 102 to 103; the file is a per-run temporary spill fromPath.GetTempFileName, so the layout carries no cross-version obligation.Verification
Release,net48, 0 errors.MsdialCoreTests: 316/316 (311 before, plus five new covering gap-filled and unwritten-light rows, themz-from-Massrule, and the reference-matched verdict).MsdialCoreTestAppTests: 13/13, including a newReadDetailedAlignmentProvenancetest next tomaster'sReadLbmAnnotatorPrioritytest. That is the project that actually compiles the rebasedConfigParser, and running it was the one piece of evidence the original green result did not cover.peak_originsplits 17871 members into 7879detectedand 9992gap_filled.Two guards so this class of defect cannot return quietly:
WriteMembernow goes through aWriteRowthat refuses a field count the header does not declare, since the header and the two value lists are three hand-maintained arrays; and the tests assert whole lines rather than substrings.Deliberately not in this PR
.mdpeakid.tsvbehind a config flag. It is currently unconditional, and in light mode it adds anO(spots x files)seek-per-cell pass — which is exactly the regime light mode exists for. Worth a follow-up; the matrix is the authoritative join key, so defaulting it off would need thought.method.ToLower()sites to the invariant form.🤖 Generated with Claude Code