TIPS

Author: Yu-Chen Den*, Kuan-Yu Chen*, Kendro Vincent, Tien-Hao Chang (*Equal Contribution)

Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD)

Official code implementation of KDD 2026 paper: Integrating Inductive Biases in Transformers via Distillation for Financial Time Series Forecasting (Will change the official repo directory soon)

Setup

Install uv for package management

pip install uv && uv sync

The data/, model/ and output/ directories are created automatically by the scripts below.

Preparing dataset

bash scripts/prepare_data.sh

Configuration

Both stages are configured with Hydra. configs/teacher.yaml and configs/student.yaml pull in configs/model_config.yaml under model_conf, and any field can be overridden on the command line:

python -m src.teacher_main seeds=[0] markets=[csi300] device=cuda:0 wandb=false

To inspect the composed config without training anything:

python -m src.student_main --cfg job

Run both stages

bash scripts/run_pipeline.sh

Arguments are forwarded to both stages, so only pass keys that exist in both configs. For stage-specific keys, run the two scripts separately.

bash scripts/run_pipeline.sh seeds=[0] markets=[csi300] device=cuda:0

Stage 1 — Train Transformer teachers with different attention masks/biases

bash scripts/train_teachers.sh      # or: python -m src.teacher_main

This trains one teacher per (attention bias, market, seed) and writes, for every run, checkpoints to model/tips_teacher-{bias}_{market}/ and predictions to output/tips_teacher-{bias}_{market}/. The train-split predictions (pred_train_s{seed}.npy) are what stage 2 distills from, so stage 1 must be finished for all biases before running stage 2.

Stage 2 — Distill teachers into a single student

bash scripts/distill_student.sh     # or: python -m src.student_main

The teacher predictions on the train split are averaged into a pseudo-label (over teachers first, then over seeds) and used as the only training signal for a single vanilla Transformer, optimized with a listwise ranking loss (ListNet) and reported as a stochastic weight average of the tail epochs. Student checkpoints land in model/tips_student_{market}/ and test predictions in output/tips_student_{market}/.

Stage 2 must use the same seq_len, target_range and date splits as stage 1; the pseudo-labels are checked against the student's feature shape and the run fails fast if they disagree.

Citation

If you use this code, please cite the following:

@article{den2026integrating,
  title={Integrating Inductive Biases in Transformers via Distillation for Financial Time Series Forecasting},
  author={Den, Yu-Chen and Chen, Kuan-Yu and Vincent, Kendro and Chang, Darby Tien-Hao},
  journal={Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2},
  year={2026}
}