Expand expiry-close model comparisons with verified provenance

README.md

@@ -98,9 +98,9 @@ A ticker with no options shows `Options are not available for {ticker}`. A price
 
 Use **Watch** on a desktop or mobile chain row, then open `/watchlist`. Watches track contracts, not positions or trades; the app does not store holdings, premiums, or assignment decisions. Any supported Nasdaq-listed ticker may be watched. The server checks each submitted watch key against the current chain and Nasdaq universe and deduplicates the exact ticker, option root, call/put side, expiration, and strike. Only ordinary contracts whose parsed root and terms match the chain row are eligible. Adjusted or ambiguous series have a disabled Watch control with a reason; saved watches show **“Assuming standard 100-share terms.”**
 
-Before expiry, the chain and watchlist lead with a dated **stock forecast** showing ITM and OTM odds for the real-world expiry-session close. The event is `close > strike` for a call and `close < strike` for a put. Equality remains a separate outcome in the model and API, so the two displayed percentages can total less than 100%. Completed Yahoo `Close` history gives the dated physical forecast. The default is a zero-drift lognormal distribution with 60-session EWMA volatility. Compare models shows separate EWMA, volatility-scaled empirical, Student-t EWMA, GJR-GARCH Student-t, and intraday results. A user-selected physical model drives the compact odds and hypothetical risk; no model is automatically chosen from past scores. The empirical, Student-t, and GJR methods support 1–25 sessions, while EWMA supports up to one year. The latest completed split-safe bar and verified cache are required. Each result shows its availability, input and fit diagnostics, and separately measured accuracy when matured outcomes exist. The comparison reports independent empirical history blocks, a maximum 95% Monte Carlo error for simulated odds, and the spread between comparable completed-close methods. Those describe sampling or method sensitivity, not predictive accuracy. Missing evidence is N/A, not forecast confidence.
+Before expiry, the chain and watchlist lead with a dated **stock forecast** showing ITM and OTM odds for the real-world expiry-session close. The event is `close > strike` for a call and `close < strike` for a put. Equality remains a separate outcome in the model and API, so the two displayed percentages can total less than 100%. Completed Yahoo `Close` history gives the dated physical forecast. The default is a zero-drift lognormal distribution with 60-session EWMA volatility. Compare models also shows volatility-scaled empirical, Student-t EWMA, GJR-GARCH Student-t, daily-OHLC range/HAR proxy, skewed-t EWMA, EGARCH skewed-t, two-regime switching variance, pooled NGBoost, earnings-jump, IV-informed, and intraday results. A user-selected physical model drives the compact odds and hypothetical risk; no model is automatically chosen from past scores. The new historical methods support 1–25 sessions, while EWMA supports up to one year. The latest completed split-safe bar and verified cache are required. Rights-dependent methods remain visibly unavailable until their input provenance qualifies. Each result shows its availability, input and fit diagnostics, and separately measured accuracy when matured outcomes exist. The comparison reports independent empirical history blocks, a maximum 95% Monte Carlo error for simulated odds, and the spread between comparable completed-close methods. Those describe sampling or method sensitivity, not predictive accuracy. Missing evidence is N/A, not forecast confidence.
 
-**Market-implied odds** are displayed separately as risk-neutral option-price context, not as a substitute for the physical forecast. The `regimelib` estimator fits one two-state distribution to validated call quotes using the Treasury rate for each expiry, then prices a cash digital. A separate per-expiry decreasing, convex call-price curve is also shown when its quote and one-tick stability checks pass. Wide bounds remain visible; quote tightness and held-out quote fit describe market-input robustness, not realized forecast accuracy. Sparse or contradictory strips cannot provide a precise market probability. American exercise, dividends, and missing individual quote timestamps still limit interpretation. The physical and risk-neutral probabilities are never combined into a recommendation score.
+**Market-implied odds** are displayed separately as risk-neutral option-price context, not as a substitute for the physical forecast. The `regimelib` estimator fits one two-state distribution to validated call quotes using the Treasury rate for each expiry, then prices a cash digital. A separate per-expiry decreasing, convex call-price curve is also shown when its quote and one-tick stability checks pass. SSVI is listed separately but remains unavailable until coherent, rights-cleared multi-expiry quotes and defensible American-exercise and dividend treatment exist. Wide bounds remain visible; quote tightness and held-out quote fit describe market-input robustness, not realized forecast accuracy. Sparse or contradictory strips cannot provide a precise market probability. American exercise, dividends, and missing individual quote timestamps still limit interpretation. The physical and risk-neutral probabilities are never combined into a recommendation score.
 
 Visible chain pages refresh odds about every five minutes during the regular trading session; a visible watchlist polls for updated cached odds. If a later watchlist poll fails, the last loaded list stays visible with a dated warning and Retry action. After hours, quote-implied valuation uses the latest completed session's official daily close and its dated rate. A watched contract's last valid market value from the latest completed session remains visible as dated context across refreshes and restarts, then disappears when a newer session completes. The compact stock summary shows its quote time, and market-session labels describe the source state at fetch. The fetch time is a snapshot time, not a claim about each option's quote timestamp; fetched and retrieved times include the browser's local timezone, while market quote time is labeled ET. Legacy strategy forecast snapshots remain in the local database for historical continuity but are never served as current odds. **Check expiry results** requests a separate background close-based outcome update.
 
@@ -129,7 +129,9 @@ Watch jobs, watched contracts, close-based observations, append-only forecast is
 
 From `backend/`, `uv run --group research python scripts/evaluate_predictive.py IREN --refresh` refreshes public Yahoo closes and prints a retrospective rolling-origin report for the older per-origin EWMA/empirical selection policy, alongside its EWMA baseline comparator. That policy is research-only and does not choose the app's default forecast. Without `--refresh`, the command reads only the verified cache; `--as-of YYYY-MM-DD` evaluates a past completed session. To freeze the deterministic 50-stock convenience sample from eligible, verified cached Nasdaq stocks, run `uv run --group research python scripts/freeze_audit_cohort.py`. Then screen a challenger with `uv run --group research python scripts/evaluate_predictive.py --replay-cohort --candidate student_t_ewma --period screen`. These immutable snapshots contain current-vintage history, so replay is retrospective screening, never as-issued evidence.
 
-Live forecast attempts for each method are recorded in an append-only DuckDB ledger, including unavailable attempts. The app's per-model evidence uses only issuances carrying the currently displayed model version. Its candidate coverage is conditional on recorded current-version candidate cells; cells without EWMA count as failures. Older generic-version attempts and missed origins remain outside this rate and are not silently counted as current-version failures. Once exact expiry-session closes have matured and passed the Nasdaq/Yahoo cross-check, `uv run --group research python scripts/evaluate_predictive.py --ledger-contest --candidate student_t_ewma --period holdout --holdout-start YYYY-MM-DD` reports paired as-issued Brier score, log loss, full-distribution score, calibration, availability, rejection reasons, and latency by horizon, moneyness, and volatility regime. Contract sides and strikes sharing one stock close are averaged into one ticker-origin-horizon unit; overlapping intervals are purged and calendar-date blocks are bootstrapped with all tickers together. These reports measure accuracy on matched observations and never change the selected model. An older `forecast_champions.json` file, if present, is left untouched and ignored.
+Live forecast attempts for each method are recorded in an append-only DuckDB ledger, including unavailable attempts. Expected capture windows are recorded separately so missed windows are visible. The app's per-model evidence uses only issuances carrying the currently displayed model version. Its candidate coverage is conditional on recorded current-version candidate cells; cells without EWMA count as failures. Older generic-version attempts and missed origins remain outside this rate and are not silently counted as current-version failures. Once exact expiry-session closes have matured and passed the Nasdaq/Yahoo cross-check, `uv run --group research python scripts/evaluate_predictive.py --ledger-contest --candidate student_t_ewma --period holdout --holdout-start YYYY-MM-DD` reports paired as-issued Brier score, log loss, full-distribution score, calibration, availability, rejection reasons, and latency by horizon, moneyness, and volatility regime. Contract sides and strikes sharing one stock close are averaged into one ticker-origin-horizon unit; overlapping intervals are purged and calendar-date blocks are bootstrapped with all tickers together. The comparison dialog shows a calendar-block interval adjusted for the ten predeclared physical-model comparisons **within each horizon band** when independent support exists; comparisons across bands remain exploratory. These reports measure accuracy on matched observations and never change the selected model. An older `forecast_champions.json` file, if present, is left untouched and ignored.
+
+The [source-rights audit](docs/source-rights.md) explains why this release does not expand Yahoo downloads or store Nasdaq option quotes for training. Pooled NGBoost also needs its exact trained-artifact hash recorded with each issued forecast before live numbers can be shown. Earnings-jump, IV-informed physical, and SSVI market estimates require qualified inputs; their unavailable reasons remain visible in the comparison dialog.
 
 `uv run --group research python scripts/evaluate_intraday_open.py IREN` screens the intraday method on historical Opens; prospectively timestamped 10:00, 13:00, and 15:30 ET snapshots are recorded for separate accuracy measurement. `uv run --group research python scripts/evaluate_intraday_prospective.py` scores matched, matured as-issued snapshots against both the dated-close forecast and simple quote reanchor, with window coverage and latency. `uv run --group research python scripts/evaluate_market_curve.py` reports market-curve quote fit, coverage, rejections, and latency. Market-implied odds are judged on quote consistency rather than realized-outcome Brier score. These reports describe stock-close forecasts and option-quote fits; they do not claim retrospective option-trading performance.
 

backend/pyproject.toml

@@ -16,6 +16,7 @@ dependencies = [
     "pydantic>=2.13",
     "regimelib==0.1.0",
     "scipy>=1.18,<2",
+    "statsmodels==0.15.0",
     "uvicorn[standard]>=0.32",
     "yfinance>=1.7",
 ]
@@ -28,7 +29,10 @@ dev = [
     "httpx2>=2.9",
     "ruff>=0.15",
 ]
-research = []
+research = [
+    "ngboost==0.5.11",
+    "scikit-learn==1.6.1",
+]
 
 [tool.ruff]
 target-version = "py312"

backend/scripts/capture_sec_events.py

@@ -0,0 +1,33 @@
+"""Capture explicit earnings dates from a bounded set of one issuer's recent 8-Ks."""
+
+from __future__ import annotations
+
+import argparse
+import os
+from pathlib import Path
+
+from stocksweeper.config import load_settings
+from stocksweeper.forecast.sec_events import capture_recent_sec_schedules
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("ticker", help="Exact SEC-listed ticker, e.g. AAPL")
+    parser.add_argument("cik", type=int, help="SEC CIK for that ticker")
+    parser.add_argument("--user-agent", help="Declared SEC contact (or use environment variable)")
+    parser.add_argument("--data-dir", type=Path, help="override STOCKSWEEPER_DATA_DIR")
+    args = parser.parse_args()
+    user_agent = args.user_agent or os.environ.get("HYPEROPTIONS_SEC_USER_AGENT")
+    if not user_agent:
+        parser.error("set HYPEROPTIONS_SEC_USER_AGENT to a declared name and contact email")
+    added = capture_recent_sec_schedules(
+        args.data_dir or load_settings().resolved_data_dir(),
+        args.ticker.upper(),
+        args.cik,
+        user_agent,
+    )
+    print(f"forward schedules added: {added}; actual releases are recorded separately")
+
+
+if __name__ == "__main__":
+    main()

backend/scripts/evaluate_predictive.py

@@ -226,7 +226,10 @@ def main() -> None:
         help="atomically save a replay report for the local app comparison view",
     )
     parser.add_argument(
-        "--candidate", choices=("empirical_scaled", "student_t_ewma", "gjr_garch_t")
+        "--candidate", choices=(
+            "empirical_scaled", "student_t_ewma", "gjr_garch_t", "ohlc_har",
+            "skew_t_ewma", "egarch_skew_t", "markov_switching", "ngboost_pooled",
+        )
     )
     parser.add_argument(
         "--provenance", choices=("as_issued", "immutable_replay"), default="as_issued"

backend/src/options_api/live_quant.py

@@ -33,6 +33,8 @@
 from stocksweeper.forecast.ledger import ForecastIssuance
 from stocksweeper.forecast.predictive import PredictiveDistribution
 
+SSVI_VERSION = "ssvi-market-v1"
+
 
 @dataclass(frozen=True)
 class LiveQuant:
@@ -160,10 +162,7 @@ def quant_for_contract(
     if physical_shadow is not None:
         model_views = []
         quote_for_model = market_odds.underlying_quote(ticker)
-        for method in (
-            "lognormal_ewma", "empirical_scaled", "student_t_ewma", "gjr_garch_t",
-            "intraday_shadow",
-        ):
+        for method in MODEL_VERSIONS:
             if predictive.status == "unavailable":
                 view = PredictiveOddsView(
                     method=method, status="unavailable", reason=predictive.reason,
@@ -248,8 +247,15 @@ def quant_for_contract(
         "bid_ask_fit": None,
     }})
     market_models = (
-        (market, market_odds.lookup_curve(ticker, side, expiry_text, strike, root))
-        if physical_shadow is not None else (market,)
+        (
+            market,
+            market_odds.lookup_curve(ticker, side, expiry_text, strike, root),
+            MarketOddsView(
+                method="ssvi", status="unavailable",
+                reason="rights_cleared_option_history_unavailable",
+                model_version=SSVI_VERSION,
+            ),
+        ) if physical_shadow is not None else (market,)
     )
     last_good = (
         market_odds.lookup_last_good(ticker, side, expiry_text, strike, root)