This is my collection of hands-on experiments in post-training and aligning language models — from reinforcement learning for reasoning, to preference optimization, to teaching vision-language models to use tools.
I built each project as a self-contained, annotated notebook. My goal isn't to chase benchmark numbers on expensive hardware — it's to understand the techniques from the inside out by building them on small models that fit on a free T4.
📝 Companion article: What Happens When You Train a CNN to Think like an LLM — my deep dive into the DPO experiment in this repo (project
02-vision-dpo).
| # | Project | Technique | Model | Dataset | What it demonstrates |
|---|---|---|---|---|---|
| 01 | GRPO for Medical Reasoning | GRPO (RL) | Gemma 3 1B | medical-o1-reasoning-SFT | Reward-function design for step-by-step reasoning |
| 02 | Vision DPO on MNIST | DPO | CNN (from scratch) | Noisy MNIST | DPO stripped to its mathematical core — on images, not text |
| 03 | Reasoning + Conversational SFT | SFT (data mixing) | Qwen3 0.6B | OpenMathReasoning + FineTome-100k | Balancing reasoning and chat ability via mix ratio |
| 04 | Multimodal + Tool Use | VLM SFT | Qwen2-VL 2B | Custom multimodal + tool-use set | Teaching a vision model to reason over images and call tools |
I intentionally spread these projects across the modern alignment toolkit:
- Reinforcement learning for reasoning (
01) — Designed reward functions from scratch that score both format (did the model actually reason in<think>tags?) and correctness. The reward design is where all the interesting decisions are. - Preference optimization, demystified (
02) — DPO is usually buried in transformer machinery. I applied it to a plain CNN classifying digits instead, so the algorithm itself is the only exotic thing in the room. - Capability balancing (
03) — Most reasoning models forget how to chat, and most chat models can't reason. This explores the data-mix ratio as a tunable knob between the two. - Multimodal + agentic (
04) — Pushing a small vision-language model to do structured medical-image reasoning and emit tool calls in a single fine-tuning pass.
- Unsloth — 2x faster, memory-efficient fine-tuning
- TRL —
GRPOTrainer,SFTTrainer,DPOTrainer - PEFT / LoRA — parameter-efficient fine-tuning
- PyTorch — the DPO-on-CNN experiment is hand-rolled
- Designed to run on a single free T4 (Colab / Kaggle)
Each notebook is independent — just clone the repo and open whichever one interests you:
git clone https://github.com/<your-username>/reasoning-and-alignment.git
cd reasoning-and-alignmentOpen any notebook in Jupyter, Colab, or VS Code. Most install their own dependencies in the first cell (pip install unsloth). You'll need a GPU for the fine-tuning notebooks; 02-vision-dpo is light enough to follow on CPU.
Note on outputs: I kept inference results and data previews so you can see what each notebook produces without running it yourself. Training progress bars and widget noise have been stripped.
I built these to experiment with post-training and alignment. If you found something useful, spotted something wrong, or want to talk through any of it — feel free to reach out. The companion article goes deeper on the ideas behind project 02.