Skip to content

Categorical has poorly defiend behavior when some events are 0 probability #459

Description

@YeungOnion

Was looking to have more function implement try_inverse_cdf instead of wrapping the panicking version with Ok, found this.

Categorical::locate (used by both inverse_cdf and try_inverse_cdf) does:

    self.norm_cdf.binary_search_by(|v| v.partial_cmp(&x))

But this won't match the standard quantile definition inf{x | F(x) >= p} if this cdf isn't monotone because there's not an ordering if some categories have 0 weight - a weird but not outright unreasonable case.

Reproducer:

let norm_cdf = vec![1.0, 1.0, 1.0]; // categories [1.0, 0.0, 0.0]
norm_cdf.binary_search_by(|v| v.partial_cmp(&1.0).unwrap()); // Ok(1), want 0

let norm_cdf = vec![0.0, 0.0, 1.0]; // categories [0.0, 0.0, 1.0]
norm_cdf.binary_search_by(|v| v.partial_cmp(&0.0).unwrap()); // Ok(1), want 0,

let norm_cdf = vec![0.25, 0.5, 0.5, 0.5, 1.0]; // interior tie
norm_cdf.binary_search_by(|v| v.partial_cmp(&0.5).unwrap()); // Ok(2), want 1

inverse_cdf/try_inverse_cdf currently reject x == 0.0 and x == 1.0 outright, which happens to route around the two boundary cases above, but does nothing for interior ties (the third example).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions