Protocol Effects in Text Classification: Seeds, Masking, and Baselines

An empirical study of what determines test accuracy on the 25k IMDB reviews when the model family is fixed at "frozen 100-dimensional GloVe embedding + shallow neural head or linear classifier". Across three neural architectures (SNN, CNN, LSTM), three input sequence lengths (100, 200, 300 tokens), 3–8 seeds per condition, and a TF-IDF + linear baseline, the ordering of test accuracy is set less by the neural architecture than by three protocol variables the choice of architecture is usually reported alongside: whether padding is masked out of the recurrence, how the stopping epoch is chosen, and whether a non-neural baseline is measured at all. A two-line TF-IDF + logistic regression pipeline reaches 0.9020 test accuracy — 2.23 percentage points above the best neural condition in the sweep.

This study began as a reanalysis of a 2022 undergraduate internship project (AiQuantum, supervised by Dr. Shyam Lal, ECE NITK); the original artifacts are preserved unmodified in legacy/ and the full methodological comparison is in report/correction_note.md.

Key findings

Numbers below are the primary metric of the study: accuracy at threshold 0.5 on the 10,000-review held-out test partition, reported as mean ± std across seeds. Statistical tests use paired McNemar per seed. Full tables at results/tables/by_condition.md and results/tables/mcnemar.md.

1. A TF-IDF + linear baseline beats every neural condition in the sweep. Logistic regression on unigram + bigram TF-IDF (min_df=2, sublinear_tf=true, max_features=50,000) reaches 0.9020; LinearSVC reaches 0.9014. The best-configured neural condition — masked LSTM at maxlen=300, or the trainable-embedding LSTM at maxlen=100 — is 2.23 pp below. The linear pipeline is deterministic given the fixed 32k/8k/10k split, so its n=1 is the complete outcome distribution rather than a single draw.

2. Masking padding out of the LSTM recurrence is neutral at short sequence lengths and decisive at long ones. At maxlen=100, unmasked LSTM (0.8662 ± 0.0033, n=5) is within noise of masked LSTM (0.8638 ± 0.0027, n=5) — a 0.24-pp gap, sign opposite the expected direction, well inside one seed's std. At maxlen=300 the same architectural difference is worth 26.73 pp: the masked LSTM reaches 0.8797 ± 0.0033 while the unmasked LSTM hits chance ≤ 0.55 on 6 of 8 seeds (75%). The failure mode is a never-started training run, not a converged-then-degraded one — on all 7 failing seeds val_loss reaches its minimum by epoch 3 and EarlyStopping fires shortly after; the one seed that trained normally (seed 107, acc 0.8691) reached its best val_loss at epoch 21 of 24 epochs run.

3. Sequence truncation at 100 tokens costs measurable accuracy on the LSTM. IMDB training reviews after preprocessing have median 88 tokens, p90 232, p99 461, max 1401. At maxlen=100, 42.1% of training reviews are truncated (default truncating='pre' throws away the start); at maxlen=300, only 5.2% are. On the masked LSTM the 100 → 300 accuracy gain is 1.59 pp — comparable to the CNN's own maxlen improvement, and larger than most effects the neural architectures differ by at fixed maxlen.

4. Fixed 6-epoch training misallocates capacity in three different directions. Under EarlyStopping (patience=3, restore_best_weights, cap 30), selected epochs are SNN 1.8 ± 0.4 (val_loss rises from epoch 1 — the SNN overfits from the start), CNN 5.4 ± 0.5, masked LSTM 6.4 ± 1.3 (val_loss still falling at 5). Unfreezing the embedding on the masked LSTM at maxlen=100 shifts the best epoch to 2.3 ± 0.6 and adds 1.03 pp of accuracy — a change that a fixed 6-epoch protocol would have run four to five epochs past and reported as a regression.

5. F1-argmax threshold tuning on the validation set does not transfer to test at these sample sizes. Of 13 multi-seed conditions with a per-seed acc@tuned − acc@0.5 paired delta on TEST, 11 have a 95% CI that straddles zero, 2 lie entirely below zero (both SNN conditions), and 0 lie entirely above. No neural condition reliably benefits from the tuning step and two reliably suffer. Accuracy at threshold 0.5 is therefore the primary metric throughout this study; the tuned column is presented as a documented but unused artefact.

6. Off-the-shelf FLOP profilers silently omit the recurrent block. keras_flops.get_flops() on the LSTM returns 257 FLOPs — exactly the cost of the final Dense(1) head alone. The analytical cost for the same architecture is 23,705,861 FLOPs, ~5 orders of magnitude larger. The CNN and SNN figures the tool returns match analytical values; only recurrent layers are dropped. This shows up in the original 2022 write-up as a table entry that implied the LSTM was cheaper than the SNN — an artifact of the profiler, not the architecture.

Results

Condensed table (full sweep with 95% CI half-widths and per-seed failure counts at results/tables/by_condition.md):

Condition	maxlen	n	acc@0.5 (mean ± std)	AUC	selected epoch
TF-IDF + Logistic Regression	—	1	0.9020	0.9658	n/a
TF-IDF + LinearSVC	—	1	0.9014	0.9661	n/a
LSTM, masked, frozen embedding	300	3	0.8797 ± 0.0033	0.9488	8.7 ± 1.2
LSTM, masked, trainable embedding	100	3	0.8741 ± 0.0104	0.9469	2.3 ± 0.6
LSTM, masked, frozen	200	3	0.8692 ± 0.0069	0.9431	5.7 ± 2.1
LSTM, unmasked, frozen	100	5	0.8662 ± 0.0033	0.9417	7.4 ± 1.5
LSTM, masked, frozen	100	5	0.8638 ± 0.0027	0.9402	6.4 ± 1.3
CNN, frozen	300	3	0.8679 ± 0.0024	0.9416	5.0 ± 0.0
CNN, frozen	200	3	0.8650 ± 0.0034	0.9393	4.7 ± 0.6
CNN, frozen	100	5	0.8546 ± 0.0019	0.9323	5.4 ± 0.5
SNN, frozen	100	5	0.7570 ± 0.0030	0.8323	1.8 ± 0.4
LSTM, unmasked, frozen	200	3	0.8069 ± 0.1029 ⚠	0.8659	11.0 ± 7.2
LSTM, unmasked, frozen	300	8	0.6124 ± 0.1237 ⚠ (6/8 chance)	0.6735	4.2 ± 6.8

⚠ marks rows where at least one seed lands below the working-arch floor; the mean±std of such rows is not a location parameter with noise but a mixture of trained and never-started runs. See results/tables/lstm_unmask_per_seed.md for per-seed mechanism evidence.

Per-seed strip plot and per-seed ROC curves for the L300 unmasked LSTM (the row that motivates the outcome-band reporting):

Threshold-transfer failure (finding 5), one point per multi-seed condition:

Reported vs analytical FLOPs (finding 6), log scale:

Token-length distribution and truncation fractions (finding 3):

Method

Data and split. IMDB 50k labeled reviews. Fixed train/test split with test_size=0.20 and split_random_state=42 → 40,000 / 10,000; within TRAIN a further TRAIN_VAL_RANDOM_STATE=123 split produces 32,000 train / 8,000 val. The test partition is touched once, at final evaluation. The class counts on the 10k test partition are 4961 negative / 5039 positive and serve as the reproduction-identity gate for src/data.py.

Seeds. Five pre-specified seeds (13, 29, 47, 71, 101) for the primary maxlen=100 conditions; three seeds for the maxlen ∈ {200, 300} and trainable-embedding sweeps. The unmasked-LSTM/maxlen=300 row was extended to n=8 with the next five primes (103, 107, 109, 113, 127) against a bimodality criterion pre-registered in the docstring of scripts/run_grid.py:EXTENDED_LSTM_L300_SEEDS; the extension retracted the earlier bimodality label — see §8 of the correction note. Mean ± std with Student-t 95% CI half-width (df = n−1). All headline paired significance tests use seed 13 (first in the pre-specified list).

Convergence. EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True) with an epoch ceiling of 30. Best epoch (1-indexed) is reported per condition.

Metrics. Accuracy at threshold 0.5 is the primary metric (see finding 5). AUC-ROC computed on raw probabilities. Accuracy at the F1-argmax threshold chosen on VAL and applied to TEST is retained in every table for the tuning-transfer analysis but not used as a headline.

Significance. McNemar's test with continuity correction on discordant pairs of the paired test-set predictions, computed per seed — never pooled across seeds, because pooling identically-labeled test items across trials violates the independence assumption McNemar's χ² relies on. Reported alongside: the Haldane–Anscombe odds ratio (adds 0.5 to both discordant cells so the effect direction is defined when either cell is empty). Full contingency tables at results/tables/mcnemar.md.

Baselines. TF-IDF (unigram + bigram, min_df=2, sublinear_tf=true, max_features=50,000) fed to scikit-learn Logistic Regression (lbfgs solver, deterministic) and LinearSVC (fixed random_state). Fit on the same 32k train partition, tuned on the same 8k val, evaluated on the same 10k test.

Failure classification. Seeds are reported by outcome, not only averaged. src/analysis.py uses chance = acc ≤ 0.55 (coin-flip noise band on 10k samples), poor = 0.55 < acc ≤ 0.70 (below any working-arch floor), normal = acc > 0.70. The pre-registered vocabulary in scripts/run_grid.py uses chance ≤ 0.55, intermediate in (0.55, 0.80), trained ≥ 0.80 — a different upper boundary. On the current L300 data no seed lands in [0.70, 0.80), so both partitions yield identical counts; the two files disagree by design and the note documents the disagreement rather than papering over it (§5 of the note).

Repository layout

• src/ — corrected pipeline: data.py, models.py, train.py, train_grid.py, flops.py, analysis.py, tfidf_baselines.py, config.py.
• scripts/ — orchestration: run_grid.py, run_all.sh (overnight driver, tees to results/run_YYYYMMDD_HHMMSS.log), measure_token_lengths.py, measure_vocab_coverage.py, measure_negation_removal.py, build_figures.py, build_correction_note.py, download_glove.sh.
• results/raw/ — one JSON per (condition, arch, maxlen, seed) trial plus raw probability .npy arrays for paired significance tests. Superseded runs are retained rather than overwritten so the extension history is auditable.
• results/tables/ — aggregated by_condition.{json,csv,md}, mcnemar.{json,md}, threshold_effect.{json,md}, lstm_unmask_per_seed.md, token_length_distribution.*, vocab_coverage.*, negation_removal.json, highlights.md.
• results/figures/ — seven figures as PNG + SVG, produced by scripts/build_figures.py from the artefacts under results/.
• report/ — correction_note.md (rendered from correction_note.tpl.md) and the placeholder audit trail _numbers.md.
• notebooks/01_results_analysis.ipynb — executed in place by make notebook; reads only from results/.
• legacy/ — the 2022 notebook and write-up preserved unmodified. It does not run on the pinned environment: the 2022 code imports keras.layers.core, a private path removed in Keras 2.12–2.15. Reproducing it end-to-end requires TF ≤ 2.10; src/ reimplements the equivalent public-API code paths on the pinned TF 2.15 stack.
• tests/ — pytest guardrails on src/data.py, src/flops.py, and src/analysis.py.

Setup and reproduction

Environment: Python 3.11.15, TensorFlow 2.15.0, Keras 2.15.0, NumPy 1.26.4, scikit-learn 1.4.2. All pinned in requirements.txt. Python 3.11 specifically — TensorFlow 2.15 does not publish wheels for 3.12, and the pinned Keras 2.15 uses import paths that Keras 3 (the default from TF 2.16) removed.

# 1. Install and fetch embeddings
make setup                              # pip install -r requirements.txt + downloads GloVe 6B into data/

# 2. Smoke test — verifies the pipeline end-to-end on 2000 samples, 1 epoch (~1 min CPU)
make smoke

# 3. Guardrail tests
make test                               # pytest on src/data.py, src/flops.py, src/analysis.py

# 4. Full grid — 3 archs × 3 maxlens × 3–8 seeds + TF-IDF baselines
make experiments                        # overnight on CPU; tees a timestamped log into results/

# 5. Regenerate tables, figures, note, and notebook (idempotent, ~2 min after step 4)
make report                             # analysis → build-figures → build-note → nbconvert

Runtimes on the reference CPU (16-core, no GPU): make smoke ~1 min, one seed of the maxlen=100 grid ~15 min, the full grid + baselines ~6–8 h. GPU determinism is not guaranteed on this stack — the study runs multiple seeds explicitly so a single reproducible trial is not required.

Individual conditions can also be run directly via environment variables that src/train.py reads:

CONDITION=mask_zero_lstm SEED=13 MAXLEN=300 python3 -m src.train

Reproducibility

Every number in this README, in report/correction_note.md, and in every table is generated from a file under results/ by scripts/build_correction_note.py. Templates live at README.tpl.md and report/correction_note.tpl.md; rendered files are overwritten on every make report. The full audit trail — every placeholder name, its resolved value, the source file it came from, and which template consumed it — is at report/_numbers.md. Editing a rendered file will not survive the next build; edit the template or the extractor.

Raw per-trial artefacts (results/raw/*.json and the .npy probability arrays used for paired McNemar) are retained under their original filenames, including for the earlier n=3 unmasked-LSTM/maxlen=300 runs that were superseded by the n=8 extension. Nothing under results/raw/ is overwritten by the aggregation step, so the extension history is inspectable and any downstream number can be traced back to the specific trial that produced it. scripts/build_correction_note.py hashes extractor inputs so a silent drift between template and source cannot recur (see §8 of the correction note for the specific incident this defence responds to).

Limitations

• Single dataset (IMDB). Any architecture-ranking claim should be tested on at least one out-of-domain sentiment corpus (SST-2, Yelp, Amazon) before being generalised beyond this study.
• Frozen 100-dimensional GloVe as the sole embedding source. The trainable-embedding ablation is only run on the masked LSTM at maxlen=100.
• No transformer comparison — deliberate scope: the study characterises the shallow-neural / linear-baseline regime, not the state-of-the-art regime.
• Asymmetric seed counts at maxlen=300 (unmasked-LSTM extended to n=8; masked-LSTM and CNN remain at n=3), so cross-condition CI comparisons at that maxlen are asymmetric.

Full limitations, retractions, and open threads: §8 of report/correction_note.md.

Citation and further reading

The full methodological comparison against the 2022 original, including the reproduction gate, per-error diagnosis, and retracted bimodality claim, is in report/correction_note.md. If you use this repository:

@misc{ajith_imdb_correction_2026,
  author  = {Ajith, Siddharth},
  title   = {IMDB sentiment classification: architecture, sequence handling, and evaluation protocol},
  year    = {2026},
  note    = {Reanalysis of a 2022 AiQuantum internship project. Corrected pipeline, five-seed evaluation, TF-IDF baseline.},
  url     = {https://github.com/siddddd17/protocol-effects-text-classification}
}