Skip to content

Stop losing downstream scores to NA, and let splatter/symsim fail properly - #20

Closed
rcannood wants to merge 4 commits into
fix/metric-normalisationfrom
fix/metric-validity
Closed

Stop losing downstream scores to NA, and let splatter/symsim fail properly#20
rcannood wants to merge 4 commits into
fix/metric-normalisationfrom
fix/metric-validity

Conversation

@rcannood

Copy link
Copy Markdown
Member

Describe your changes

Stacked on #18 -- please merge that one first. The utils.R changes here sit on top of the compute_logcounts() helper it adds.

Validity fixes for the downstream metrics and two methods. None of these lose or corrupt scores the way #17 and #18 did; they mostly turn results that should have been numbers into numbers.

generate_cosine() was filtering the wrong thing. real[!is.na(real) & !is.na(sim)] operates on the list returned by BivariateMoransI(), not on the Moran's I values inside it -- is.na() on a list of matrices is FALSE for every element, so it never dropped anything. One NaN anywhere then took the whole metric with it: crosscor_cosine came out NA for 32 of 99 runs, while generate_mantel() passes na.rm = TRUE on the same inputs and lost none. Now filters the values:

identical inputs  -> 1        (want 1)
one NaN in sim    -> 1        (was NaN before)
unrelated inputs  -> 0.8618
all NaN           -> NA       (want NA)

A simulation with no spatially variable genes now scores 0, not NA. calculate_precision() returned NA when tp + fp == 0, which is exactly what a simulator producing nothing spatially variable does -- the worst possible outcome, recorded as a missing result. Both negative controls were NA on all 10 datasets, so QC reported 8 of 30 expected control scores and nothing anchored the bottom of the scale.

calculate_recall() deliberately keeps NA in the same situation, and I added a comment saying why: there the empty denominator is a property of the real dataset (brain and hindlimbmuscle have no significant SVGs at all), so there is nothing for any simulator to recover and scoring it 0 would penalise all of them for the dataset.

splatter and symsim now fail properly. Both wrapped their per-cluster loop body in try() and never looked at the result, so a failed cluster left simulated_result a cluster short and the loop carried on; it only fell over later on a length mismatch. And symsim sampled gene lengths with replace = FALSE from the finite gene_len_pool, which errors outright on any dataset with more genes than the pool holds.

logcounts is a double. file_dataset_sp.yaml declared it integer.

Not included

  • The precompute components are still dead weight. precompute_downstream writes varm$spatial_variable_genes and varm$spatial_autocorrelation, and sc_features writes uns$L_stats and friends, but the metrics recompute all of it (the reads are commented out in downstream/script.R, with a TODO). I left it alone because wiring it up is not the small change it looks like: the autocorrelation precompute would have to go through the same compute_logcounts() that Normalise the simulated data the same way as the real data #18 introduces or it reintroduces exactly the mismatch that PR fixes, and generate_svg_sparkx() filters mitochondrial genes so its table is not n_vars-aligned. Worth its own PR, or deleting the components. Happy to do either.
  • --base tissue is still advertised for methods that don't implement it. Only scdesign3_nb, scdesign3_poisson and srtsim support it; the other five stop(). I had documented that in comp_method.yaml but the edit got reverted underneath me, so I've left the file untouched rather than push it back.
  • scdesign3 resource labels. scdesign3_poisson was killed at 8h 3m on breast (exit 140), which is exactly what hightime allows -- but as you said, a method that can't finish in 8h doesn't want a longer wall clock, it wants looking at.

Checklist before requesting a review

  • I have performed a self-review of my code

  • Check the correct box. Does this PR contain:

    • Breaking changes
    • New functionality
    • Major changes
    • Minor changes
    • Bug fixes
  • Proposed changes are described in the CHANGELOG.md

  • CI Tests succeed and look good!

rcannood added 4 commits July 28, 2026 13:57
* generate_cosine(): filter the Moran's I values rather than the objects
  holding them. `real[!is.na(real) & !is.na(sim)]` operates on a list, so it
  never dropped anything and one NaN took the whole metric with it -- 32 of 99
  runs came out NA, while generate_mantel() passes na.rm and lost none.
* calculate_precision(): report 0 when the simulation has no spatially
  variable genes at all. That is a bad simulation rather than a missing
  result, and reporting NA left both negative controls without a score on any
  dataset -- 8 of 30 expected control scores -- so nothing anchored the bottom
  of the scale.
* calculate_recall(): left as NA on an empty denominator, and said why. There
  it is the real dataset that has no spatially variable genes, so there is
  nothing for any simulator to recover.
* Drop the `try()` wrapping the per-cluster loop body in both. A cluster that
  failed left `simulated_result` a cluster short and the loop carried on; the
  script only fell over later, on a length mismatch, and `res` was never
  looked at anyway.
* symsim: sample gene lengths with replacement. `gene_len_pool` is finite, so
  `replace = FALSE` errors outright on any dataset with more genes than the
  pool holds.
file_dataset_sp.yaml declared `logcounts` as integer. Log-transformed counts
are doubles, and the shipped datasets store them as such.
@rcannood

Copy link
Copy Markdown
Member Author

Same here -- splitting into one PR per fix.

@rcannood rcannood closed this Jul 28, 2026
@rcannood
rcannood deleted the fix/metric-validity branch July 28, 2026 12:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant