Skip to content

yapc.score threshold is silently collapsed to a 0/1 test by an implicit FieldElement bool conversion (atl03x and legacy endpoints); yapc v1-3 on atl03x silently return all-zero scores #648

Description

@dshean

Server + client v5.5.0, public cluster. The ICESat-2 module docs describe
yapc.score as "the minimum yapc classification score of a photon to be
used in the processing request" and photon-selection parameters as applied
together (user_guide/icesat2.md §1.2). Observed: the threshold value
is ignored everywhere — score only acts as a boolean:

  1. atl03x, yapc {version:0, score:N}, photon output: identical rows
    for N = 150 / 3300; photons with yapc_score far below N are returned
    (65 photons < 150 on a 3555-photon segment; min returned score 1).
  2. Same + fit {}: h_mean and n_fit_photons identical for
    N = 0 / 150 / 1000 / 3300 / 8000 and with no yapc block at all.
  3. atl06p (legacy), yapc {version:0, score:N}: identical for
    N = 0…8000 and for no yapc block.
  4. atl06p, yapc {version:3, score:N}: any N > 0 changes the
    selection once relative to N = 0 (≡ no-yapc), then identical for all
    N > 0 (50 / 150 / 255 tested).
  5. The boundary case shows the mechanism: on a segment containing 44
    photons with yapc_score == 0 (R007 assigns weight_ph = 0 to
    quality_ph-flagged photons), score: 0 returns 4299 photons while
    score: 1, 150, and 3300 all return the identical 4255 — any
    N > 0 behaves exactly as N = 1.

Root cause (v5.5.0 source): FieldElement<T> defines operator bool()
but no operator T() (packages/core/package/FieldElement.h:92), and the
score comparisons omit .value:

  • datasets/icesat2/package/Atl03DataFrame.cpp:681,689
    if(yapc_score < parms.yapc.score)
  • datasets/icesat2/package/Atl03Reader.cpp:1397,1405,1413
    if(yapc_score < parms->yapc.score)

so the threshold converts through operator bool() + integral promotion
and the test is effectively yapc_score < (score != 0 ? 1 : 0). All the
observations above follow. BathyDataFrame.cpp:582 uses the intended
pattern (parms.yapc.score.value). The v0 check dates to f25834f4
("support for 007 yapc added", v4.20.0), so the documented v0 threshold
has never worked; the legacy computed-score comparison has the same form.

Two documentation notes, verified against returned data: (1) with
version: 0 the returned yapc_score is ATL03 R007 weight_ph verbatim
(bit-identical to atl03_ph_fields: ["weight_ph"]), i.e. uint16 on the
0–65535 saturation-normalized DDA scale (values to ~24k observed; NSIDC
ATL03 v007 data dictionary), not the "0 - 255" stated at
user_guide/icesat2.md lines 49 and 534 — that scale belongs to the
server-computed v1–3 scores (the record type changed uint8 → uint16 in
f25834f4). (2) yapc {version: 3} on atl03x returns yapc_score = 0
for every photon with no error or warning; the docs do say v1–3 are "not
supported by atl03x" (custom yapc was removed from atl03x in the #463
fix), but a warning or rejection would make the failure visible instead
of silent.

Ask: restore the documented threshold by using .value in the five
comparisons above (Atl03DataFrame.cpp ×2, Atl03Reader.cpp ×3), update the
two "0 - 255" doc lines for the v0 scale, and warn (or reject) on
version: 1–3 with atl03x instead of returning silent zeros.

Reproducer (public cluster, no auth): atl03x with
{"poly": [{"lon": -115.62233160383393, "lat": 43.95583666543543}, {"lon": -115.6218973550836, "lat": 43.958531466699206}, {"lon": -115.62251709325558, "lat": 43.958583917795764}, {"lon": -115.6229513420059, "lat": 43.955889116531985}], "t0": "2023-02-15T19:42:02Z", "t1": "2023-02-16T19:42:02Z", "beams": ["gt1l"], "srt": 0, "cnf": -2, "quality_ph": [0,1,2,3], "yapc": {"version": 0, "score": N}}
→ N=0: 4299 photons (min score 0); N=1/150/3300: identical 4255 (min
score 1; 43 photons with score < 150 still present at N=150). Happy to
provide the full probe script. (I have a fix prepared and will open a PR
referencing this issue.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions