Backtest Autoresearch

Autonomous trading strategy optimization inspired by Karpathy's autoresearch.

An LLM iteratively generates Python trading strategies, each one is backtested against 10+ years of historical stock market data, and only CAGR-improving strategies (subject to a Sharpe floor) advance. The ratchet only moves forward.

How It Works

   Seed strategy (autoresearch/strategies/seed_strategy.py)
        |
        v
   LLM generates candidate (via OpenAI-compatible API)
        |
        v
   python run_autoresearch.py runs the backtest
        |
        v
   Sharpe improved?
    /          \
  yes           no
   |             |
  KEEP        DISCARD
   |          (leader unchanged)
   +------+------+
          |
          v
       REPEAT

Each iteration the LLM sees the last 15 experiment results and the last 5 failed strategies' full code (including crash tracebacks), so it can diagnose mistakes and build on what worked.

The Ratchet

Component	Karpathy (ML)	This Project (Trading)
Editable file	train.py	autoresearch/strategies/strategy_NNNN.py
Fixed harness	prepare.py	autoresearch/backtest_engine.py
Scalar metric	val_bpb (lower=better)	CAGR (higher=better), Sharpe ≥ 1.5 floor
Time budget	5 min GPU training	~90s per 10-year backtest
Agent prompt	program.md	autoresearch/researcher.py:SYSTEM_PROMPT

Setup

# 1. Install dependencies
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# 2. Configure database + LLM access
cp .env.example .env
# Edit .env with your PostgreSQL credentials and LLM_API_KEY / LLM_BASE_URL / LLM_MODEL

# 3. Run the loop
python3 run_autoresearch.py -n 50

It auto-resumes. The loop reads autoresearch/results/history.jsonl to know where it left off.

Usage

source .venv/bin/activate

# Run 50 iterations (resumes from last run if history exists)
python3 run_autoresearch.py -n 50

# Run with a custom seed strategy
python3 run_autoresearch.py -n 50 --seed path/to/strategy.py

# Use a local LLM endpoint (llama.cpp, LM Studio, vLLM, etc.)
python3 run_autoresearch.py --local

Database

PostgreSQL with ~10,000 US stocks (2011–present) in the stock schema:

analytics — 19.5M rows. Daily OHLCV + technical indicators:

• Moving Averages: sma_21, sma_50, sma_150, sma_200, fast_ema, slow_ema
• Momentum: rsi, macd, macd_signal, macd_hist, adx
• Volatility: atr, bb_upper, bb_middle, bb_lower, ttm_max, ttm_min
• Relative Strength: rs_value, rs_grade
• Returns: returns_day, returns_cumulative, returns_year
• Risk: alpha, beta, corr, slope, r_squared
• Benchmarks: SPY, QQQ, IWM, DIA, VTI in the same table

symbols_info — Fundamentals for ~10,600 stocks:

• market_cap, trailing_pe, forward_pe, price_to_book
• profit_margins, return_on_equity, gross_margins
• total_debt, current_ratio, debt_to_equity
• target_mean_price, recommendation_mean

market_exposure (VIEW) — Daily market regime:

• exposure_tier: "Short 100%" through "Long 100%"
• bar_rank: 0–99 percentile

explosive_motifs / get_live_breakouts() — DTW geometric pattern matches.

Execution Model

• Signals at market close → execution at next day's open
• 0.1% commission + progressive slippage (scales with volume impact)
• 3% daily volume cap per order
• $100,000 starting capital
• Max 20 concurrent positions (long + short combined)
• Signal values: 1=buy long, -1=exit long, -2=open short, 2=cover short, 0=hold

Metrics

Primary: CAGR (maximized, subject to Sharpe ≥ 1.5 floor). Also reported: total return, Sharpe, max drawdown, win rate, profit factor, Sortino ratio.

Key Files

• autoresearch/strategies/seed_strategy.py — starting baseline
• autoresearch/strategies/best_strategy.py — current leader (auto-updated)
• autoresearch/results/leaderboard.json — best metrics
• autoresearch/results/history.jsonl — full log of all attempts
• autoresearch/results/iteration_NNNN.json — per-iteration results with strategy descriptions