Skip to content

Fix: market classifier never classified any market - #10

Open
mpont91 wants to merge 1 commit into
ent0n29:mainfrom
mpont91:fix/market-classifier-is-null
Open

Fix: market classifier never classified any market#10
mpont91 wants to merge 1 commit into
ent0n29:mainfrom
mpont91:fix/market-classifier-is-null

Conversation

@mpont91

@mpont91 mpont91 commented Aug 27, 2026

Copy link
Copy Markdown

The job that populates aware_market_classifications looks for pending markets with:

LEFT JOIN polybot.aware_market_classifications c FINAL
    ON t.market_slug = c.market_slug
WHERE t.market_slug != ""
  AND c.market_slug IS NULL

In ClickHouse an unmatched LEFT JOIN row yields the column type's default rather than NULL unless join_use_nulls is set. market_slug is LowCardinality(String), so an unclassified market comes back as the empty string and IS NULL never matches. get_unclassified_markets() returns an empty list on every run and the job logs "No markets to classify".

The same predicate appears in get_stats(), which therefore always reports zero unclassified markets.

Effect

aware_market_classifications stays empty forever. Because the category views join through it, that leaves:

  • market_category blank on every row of aware_global_trades
  • aware_trader_category_distribution returning an empty category for every trader
  • aware_category_stats reporting everything as UNCLASSIFIED
  • the sectorial PSI indices (PSI-CRYPTO, PSI-POLITICS, PSI-SPORTS) with no eligible members, since they filter by category

Fix

Compare against the empty string instead.

Verified

Against a database with 8,243 distinct traded markets:

before: SELECT count() ... AND c.market_slug IS NULL  ->     0
after:  SELECT count() ... AND c.market_slug = ""     ->  8243

A run of the fixed job classified all 8,243:

CRYPTO 1677 | POLITICS 640 | SPORTS 263 | ENTERTAINMENT 71
ECONOMICS 50 | SCIENCE 48 | NEWS 21 | OTHER 5473

and aware_trader_category_distribution now returns real categories (5,003 traders with CRYPTO exposure, 665 POLITICS, 318 SPORTS).

The job looked for pending markets with `c.market_slug IS NULL`. An
unmatched LEFT JOIN in ClickHouse yields the column type's default, not
NULL, so for LowCardinality(String) the predicate never matched and the
job reported "No markets to classify" on every run.

aware_market_classifications stayed empty, which left market_category
blank across aware_global_trades and aware_trader_category_distribution,
and left the sectorial PSI indices (CRYPTO, POLITICS, SPORTS) with no
eligible members.

After the fix, a local run classified 8243 markets: 1677 CRYPTO,
640 POLITICS, 263 SPORTS, 5473 OTHER.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant