Fix: market classifier never classified any market - #10
Open
mpont91 wants to merge 1 commit into
Open
Conversation
The job looked for pending markets with `c.market_slug IS NULL`. An unmatched LEFT JOIN in ClickHouse yields the column type's default, not NULL, so for LowCardinality(String) the predicate never matched and the job reported "No markets to classify" on every run. aware_market_classifications stayed empty, which left market_category blank across aware_global_trades and aware_trader_category_distribution, and left the sectorial PSI indices (CRYPTO, POLITICS, SPORTS) with no eligible members. After the fix, a local run classified 8243 markets: 1677 CRYPTO, 640 POLITICS, 263 SPORTS, 5473 OTHER.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The job that populates
aware_market_classificationslooks for pending markets with:In ClickHouse an unmatched LEFT JOIN row yields the column type's default rather than NULL unless
join_use_nullsis set.market_slugisLowCardinality(String), so an unclassified market comes back as the empty string andIS NULLnever matches.get_unclassified_markets()returns an empty list on every run and the job logs "No markets to classify".The same predicate appears in
get_stats(), which therefore always reports zero unclassified markets.Effect
aware_market_classificationsstays empty forever. Because the category views join through it, that leaves:market_categoryblank on every row ofaware_global_tradesaware_trader_category_distributionreturning an empty category for every traderaware_category_statsreporting everything as UNCLASSIFIEDFix
Compare against the empty string instead.
Verified
Against a database with 8,243 distinct traded markets:
A run of the fixed job classified all 8,243:
and
aware_trader_category_distributionnow returns real categories (5,003 traders with CRYPTO exposure, 665 POLITICS, 318 SPORTS).