feat: add drug_mean and drug_tissue_mean response transformations (conditional-mean residualization)

[jfrog64]

PR Checklist for all PRs

• This comment contains a description of changes (with reason)
• Referenced issue is linked — no existing issue, came up while benchmarking baselines with response transformations
• If you've fixed a bug or added code that should be tested, add tests!
• Documentation in docs is updated — new module added to docs/API.rst, new options documented in docs/usage.rst

Changes

New features

--response_transformation currently offers standard, minmax and robust. All three are global monotone
rescalings of the response. That means they cannot change any of the metrics DrEval reports: Spearman and Kendall
are rank-based and therefore invariant under a monotone map, and the Normalized * metrics subtract the
NaiveMeanEffectsPredictor baseline, which rescales along with everything else. Pearson/R² are invariant under an
affine map. So today the flag exists, but for the reported scores it is close to a no-op — the only effect is on the
optimizer's conditioning.

This adds two transformations that actually change what the model has to learn: drug_mean and drug_tissue_mean.

drevalpy/response_transformation.py (new) contains GroupMeanCenterer, a transformer that subtracts a
conditional mean — estimated on the training fold only — instead of a global constant:

class GroupMeanCenterer(BaseEstimator, TransformerMixin):
    requires_groups = True

    def __init__(self, group_fields: tuple[str, ...] = ("drug_ids",)): ...
    def fit(self, X, y=None, groups=None) -> "GroupMeanCenterer": ...
    def transform(self, X, groups=None) -> np.ndarray: ...
    def inverse_transform(self, X, groups=None) -> np.ndarray: ...

The model is then trained on the residuals rather than on the main effect, which is where the Normalized * metrics
look anyway. inverse_transform adds the group mean back, so predictions are written out on the original response
scale and every downstream evaluation is unaffected.

group_fields names the columns of DrugResponseDataset the group key is built from:

• drug_mean → ("drug_ids",): removes the drug main effect, i.e. how sensitive cell lines are to this drug on
  average.
• drug_tissue_mean → ("drug_ids", "tissue"): also removes the tissue-specific sensitivity of a drug, so the
  model is left with the within-tissue drug × cell line structure.

The means are nested, not flat. fit estimates a mean for every prefix of group_fields, and the lookup walks
them from specific to coarse: a (drug, tissue) combination that did not occur in the training fold falls back to the
mean of its drug, an unknown drug to the global training mean. Without that, drug_tissue_mean would be useless in
LTO, where every test tissue is unseen by construction and a flat fallback would push every test row onto the global
mean; with it, LTO simply degenerates to drug_mean.

Because the transformer needs to know which row belongs to which group, DrugResponseDataset supplies the columns —
and only for transformations that ask for them:

def _transformation_kwargs(self, response_transformation: TransformerMixin) -> dict:
    if not getattr(response_transformation, "requires_groups", False):
        return {}

    fields = getattr(response_transformation, "group_fields", ("drug_ids",))
    columns = []
    for field in fields:
        values = getattr(self, field, None)
        if values is None:
            raise ValueError(...)
        columns.append(np.asarray(values).astype(str))
    return {"groups": np.stack(columns, axis=1)}

transform, fit_transform and inverse_transform forward these kwargs. For StandardScaler, MinMaxScaler and
RobustScaler the dict is empty, so those calls are byte-for-byte the ones that ran before — no existing
behaviour changes, and --response_transformation keeps its current default.

Details that matter for the eval modes DrEval cares about:

• fit only ever sees the training fold, so no test-fold information leaks into the offsets.
• Unseen groups never raise, they fall back one level at a time (LDO, LTO, cross-study). The lookup joins the fields
  with the ASCII unit separator and uses searchsorted per level, so this stays one vectorized pass per level
  instead of a per-row dict lookup.
• tissue is optional on DrugResponseDataset. If a dataset has none, drug_tissue_mean raises with a message
  naming the missing field rather than silently behaving like drug_mean.

Wiring: get_response_transformation returns the transformer for both names, check_arguments accepts them, and the
typer help text plus docs/usage.rst list them.

Tests: tests/test_response_transformation.py (22 tests) covers

• requires_groups is set on GroupMeanCenterer and absent on plain sklearn scalers
• fit records the global mean and the per-group means, and both nesting levels for two group fields
• transform centers every group on zero, and inverse_transform round-trips to the original values
• shapes are preserved for both (n,) and (n, 1) input, since the dataset calls it with column vectors
• unseen groups sorting before and after all fitted keys fall back to the global mean (this is the searchsorted
  clipping)
• an unseen tissue falls back to the drug mean, an unseen drug to the global mean
• the joined group keys cannot collide (("A", "B_C") vs ("A_B", "C"))
• handing over fewer group fields than at fit time raises instead of centering on the wrong mean
• without groups it degenerates to plain mean-centering, and an empty training fold does not raise
• DrugResponseDataset.fit_transform/inverse_transform hand over drug ids and tissues, and transform the
  predictions with the same groups
• the LTO situation end to end through the dataset API: fitted on one tissue, applied to another
• a dataset without tissues raises for drug_tissue_mean
• a group-unaware scaler is still called without a groups argument
• get_response_transformation and check_arguments accept both names and still reject nonsense

Bug fixes

Maintenance

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.72%. Comparing base (7d24cd6) to head (22cf65a).
⚠️ Report is 117 commits behind head on development.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@               Coverage Diff               @@
##           development     #465      +/-   ##
===============================================
+ Coverage        80.34%   83.72%   +3.38%     
===============================================
  Files              101      133      +32     
  Lines             8171    10885    +2714     
===============================================
+ Hits              6565     9114    +2549     
- Misses            1606     1771     +165     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[PascalIversen]

Nice! Thanks! I actually think I once started something similar, but didn't push on because I got distracted.

Did you try it out yet?

The only thing I am wondering is if NaiveMeanEffectsPredictor should be exempt? Do you know if it makes any difference on its results?

[jfrog64]

Yes, we tried the drug_mean transformation — 16 pairs on CTRPv2/LCO, 5 folds, identical splits,
each pair differing only by --response_transformation drug_mean, all normalized against the same
untransformed NaiveMeanEffectsPredictor.

Linear and weak models gained a lot: ElasticNet +0.13 … +0.15 nPCC (though from near zero,
0.01 → 0.15), SVR +0.11, SimpleNeuralNetwork and MultiViewLightGBM +0.05, DIPK +0.02. The strong
tree-based models lost: MultiViewXGBoost −0.03 and RandomForest −0.06 / −0.07 (our best RF arm
drops from rank 5 to 22 in a 48-arm field). Precily (−0.013) and PharmaFormer (−0.010) were slightly
negative. So it clearly helps models that don't capture the drug main effect themselves, but it is
not a free win for everyone.

And yes, I think NaiveMeanEffectsPredictor should be exempt from the transformation. For
standard/minmax/robust this is a no-op (globally affine, means are affine-equivariant), but not
for the conditional means added here: the same model arm scores nPCC 0.3083 instead of 0.3106.
Small, but the NME is the reference for every Normalized * metric — if it moves along, two runs
with different response_transformation are silently no longer on the same scale.

I'll push another commit for that, plus a regression test that NME predictions are independent of
response_transformation.