Stop losing downstream scores to NA, and let splatter/symsim fail properly - #20
Closed
rcannood wants to merge 4 commits into
Closed
Stop losing downstream scores to NA, and let splatter/symsim fail properly#20rcannood wants to merge 4 commits into
rcannood wants to merge 4 commits into
Conversation
* generate_cosine(): filter the Moran's I values rather than the objects holding them. `real[!is.na(real) & !is.na(sim)]` operates on a list, so it never dropped anything and one NaN took the whole metric with it -- 32 of 99 runs came out NA, while generate_mantel() passes na.rm and lost none. * calculate_precision(): report 0 when the simulation has no spatially variable genes at all. That is a bad simulation rather than a missing result, and reporting NA left both negative controls without a score on any dataset -- 8 of 30 expected control scores -- so nothing anchored the bottom of the scale. * calculate_recall(): left as NA on an empty denominator, and said why. There it is the real dataset that has no spatially variable genes, so there is nothing for any simulator to recover.
* Drop the `try()` wrapping the per-cluster loop body in both. A cluster that failed left `simulated_result` a cluster short and the loop carried on; the script only fell over later, on a length mismatch, and `res` was never looked at anyway. * symsim: sample gene lengths with replacement. `gene_len_pool` is finite, so `replace = FALSE` errors outright on any dataset with more genes than the pool holds.
file_dataset_sp.yaml declared `logcounts` as integer. Log-transformed counts are doubles, and the shipped datasets store them as such.
Member
Author
|
Same here -- splitting into one PR per fix. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes
Stacked on #18 -- please merge that one first. The
utils.Rchanges here sit on top of thecompute_logcounts()helper it adds.Validity fixes for the
downstreammetrics and two methods. None of these lose or corrupt scores the way #17 and #18 did; they mostly turn results that should have been numbers into numbers.generate_cosine()was filtering the wrong thing.real[!is.na(real) & !is.na(sim)]operates on the list returned byBivariateMoransI(), not on the Moran's I values inside it --is.na()on a list of matrices isFALSEfor every element, so it never dropped anything. One NaN anywhere then took the whole metric with it:crosscor_cosinecame out NA for 32 of 99 runs, whilegenerate_mantel()passesna.rm = TRUEon the same inputs and lost none. Now filters the values:A simulation with no spatially variable genes now scores 0, not NA.
calculate_precision()returned NA whentp + fp == 0, which is exactly what a simulator producing nothing spatially variable does -- the worst possible outcome, recorded as a missing result. Both negative controls were NA on all 10 datasets, so QC reported 8 of 30 expected control scores and nothing anchored the bottom of the scale.calculate_recall()deliberately keeps NA in the same situation, and I added a comment saying why: there the empty denominator is a property of the real dataset (brain and hindlimbmuscle have no significant SVGs at all), so there is nothing for any simulator to recover and scoring it 0 would penalise all of them for the dataset.splatterandsymsimnow fail properly. Both wrapped their per-cluster loop body intry()and never looked at the result, so a failed cluster leftsimulated_resulta cluster short and the loop carried on; it only fell over later on a length mismatch. Andsymsimsampled gene lengths withreplace = FALSEfrom the finitegene_len_pool, which errors outright on any dataset with more genes than the pool holds.logcountsis a double.file_dataset_sp.yamldeclared it integer.Not included
precompute_downstreamwritesvarm$spatial_variable_genesandvarm$spatial_autocorrelation, andsc_featureswritesuns$L_statsand friends, but the metrics recompute all of it (the reads are commented out indownstream/script.R, with a TODO). I left it alone because wiring it up is not the small change it looks like: the autocorrelation precompute would have to go through the samecompute_logcounts()that Normalise the simulated data the same way as the real data #18 introduces or it reintroduces exactly the mismatch that PR fixes, andgenerate_svg_sparkx()filters mitochondrial genes so its table is notn_vars-aligned. Worth its own PR, or deleting the components. Happy to do either.--base tissueis still advertised for methods that don't implement it. Onlyscdesign3_nb,scdesign3_poissonandsrtsimsupport it; the other fivestop(). I had documented that incomp_method.yamlbut the edit got reverted underneath me, so I've left the file untouched rather than push it back.scdesign3_poissonwas killed at 8h 3m on breast (exit 140), which is exactly whathightimeallows -- but as you said, a method that can't finish in 8h doesn't want a longer wall clock, it wants looking at.Checklist before requesting a review
I have performed a self-review of my code
Check the correct box. Does this PR contain:
Proposed changes are described in the CHANGELOG.md
CI Tests succeed and look good!