BioLLM Lite

Research prototype / reproduction-style experiment. BioLLM Lite is a small, honest attempt to reproduce the implicit-bias neuron (IBNN) of Mohedano et al., Updating the standard neuron model in artificial neural networks (docs/updating-standard-neuron-model.pdf), inside a tiny pretrained transformer (SmolLM2-135M). It patches selected MLP layers at runtime with a differentiable fixed-point IBNN transformation and benchmarks the effect. It is a study, not a product, and not a claim that BioMLP is a better language model.

Results are mixed, and reported as such:

• Adaptation quality: neutral-to-slightly-worse. On final held-out eval loss the patch does not beat the baseline (TinyStories: ~+0.007 worse). An earlier "5/5 seed wins" claim was an artifact of a biased delta metric and has been corrected.
• Robustness: a small, paper-aligned positive signal. With the paper-faithful all-layer pre_act + uniform patch, the model degrades less than the baseline under input-embedding noise, and the advantage grows with noise (+0.0072 at noise_std=0.3, +0.0101 at 0.5). This is the property the paper actually claims, but the effect is small and single-seed.

See docs/results.md for the full, honest breakdown.

Results

Current experiment results, interpretation, and generated charts are in docs/results.md.

Plain-English takeaway: BioMLP is not a better adapter here — on final held-out eval loss it is neutral-to-slightly-worse, and the only meaningful positive is a small robustness edge under input noise with the all-layer paper-faithful patch. Treat that as a promising-but-unconfirmed signal that needs more seeds and models, not as evidence that BioMLP wins.

Regenerate result charts:

python scripts/plot_results.py

Quickstart

Create and activate the local environment, then install the project:

source .venv/bin/activate
python -m pip install -e .

Download the tiny model into the repo-local cache:

python experiments/download_model.py

Run tests:

python -m pytest

Run a local smoke test:

python experiments/smoke_test.py --local-files-only --cache-dir models/hf
python experiments/smoke_test.py --patch --patch-layers 0 --mode pre_down_proj --local-files-only --cache-dir models/hf

Demo

Launch the local Gradio timed side-by-side demo:

python demo/app.py --local-files-only --cache-dir models/hf

The demo compares runtime behavior and reports generation time, token counts, and tokens/sec. It can also compare saved fine-tuned baseline and BioMLP runtime checkpoints:

python demo/app.py --local-files-only --cache-dir models/hf \
  --baseline-checkpoint checkpoints/tiny_finetune/baseline_... \
  --biomlp-checkpoint checkpoints/tiny_finetune/biomlp_...

See docs/demo.md for details.

Fine-Tuning And Checkpoints

Run a tiny baseline adaptation and save a standard Hugging Face checkpoint:

python experiments/tiny_finetune.py --local-files-only --cache-dir models/hf --max-steps 20 --save-checkpoint

Run a tiny BioMLP adaptation and save a PyTorch runtime checkpoint:

python experiments/tiny_finetune.py --patch --patch-layers 0 --mode pre_down_proj --local-files-only --cache-dir models/hf --max-steps 20 --save-checkpoint

Load a saved checkpoint and generate:

python experiments/load_checkpoint_demo.py --checkpoint checkpoints/tiny_finetune/baseline_... --prompt "Explain neural networks in one sentence."
python experiments/load_checkpoint_demo.py --checkpoint checkpoints/tiny_finetune/biomlp_... --prompt "Explain neural networks in one sentence."

Baseline checkpoints are normal Hugging Face checkpoints. BioMLP checkpoints are project-specific PyTorch runtime state dicts with patch metadata; they are not Ollama/GGUF-compatible artifacts.

Key Experiment Commands

Fixed-eval comparison:

python experiments/run_tiny_comparison.py --local-files-only --cache-dir models/hf --max-steps 25

Dataset learning curves:

python experiments/compare_learning_curves.py --train-samples 64 --eval-samples 32 --max-steps 30 --eval-every 10 --local-files-only --cache-dir models/hf
python scripts/plot_learning_curves.py

Sweep patch modes:

python experiments/sweep_tiny.py --local-files-only --cache-dir models/hf --max-steps 25

Multi-seed repeatability:

python experiments/multiseed_compare.py --local-files-only --cache-dir models/hf --max-steps 50 --seeds 1,2,3,4,5 --shuffle-train --train-repeat 3

Ablation controls:

python experiments/ablation_compare.py --local-files-only --cache-dir models/hf --max-steps 50 --seeds 1,2,3,4,5 --shuffle-train --train-repeat 3

Paper-faithful robustness benchmark (input-noise degradation, lower is better):

python experiments/robustness_compare.py --local-files-only --cache-dir models/hf --eval-samples 32 --mode pre_act --neighbor-mode uniform --patch-layers 0
python scripts/plot_robustness.py

--mode pre_act inserts IBNN on the gate pre-activation and --neighbor-mode uniform uses the paper's all-units interaction. Use --patch-layers all to patch every decoder layer.

Limitations

This is an experimental PyTorch runtime patch. The current evidence is from a small synthetic benchmark and tiny fine-tuning runs. It does not prove general language-model quality, and BioMLP runtime checkpoints are not directly compatible with Ollama, GGUF, or standard Hugging Face model loading.

Model weights and generated artifacts are ignored by git:

models/
outputs/
checkpoints/
runs/
*.safetensors
*.bin
*.gguf
*.pt
*.pth

Detailed implementation history lives in docs/development_log.md.