Shivank Samir Prabhudessai
Independent researcher working on training small language models efficiently on free compute. My focus is careful, reproducible pretraining: resumable tranche-based runs on free 2× T4 GPUs, evaluation gates frozen before training, matched baselines with confidence intervals, and honest reporting, including the results that fall short.
Highlighted: Stasis 1
ShivankXD/stasis-1 is a 109.5M-parameter decoder-only LM (12 layers, RoPE, SwiGLU, RMSNorm, 2048 context) trained from scratch on FineWeb-Edu.
- ~2.95B training tokens so far (current checkpoint: step 36,420), all on free 2× T4 compute
- Latest run (step 23,612 → 36,420, 12,808 updates): all 8 pre-registered held-out gates passed (−1.39% CE on a fresh primary panel vs step 23,612)
- Matched GPT-2 small comparison at step 36,420 (same frozen metrics as at step 12,269): still behind, not at parity. WikiText-2 test BPB 1.226 vs 1.057 (16.0% worse, was 20.6% at step 12,269); fixed WikiText-2 panel 10.9% worse (was 14.1%); LAMBADA acc 20.0% vs 33.7%; HellaSwag acc_norm 29.4% vs 31.3%; PIQA and ARC-Easy statistically inconclusive
- Released: weights (safetensors), modeling code, training logs, frozen metric definitions, and per-item benchmark rows
Datasets
Released alongside Stasis (CC-BY-4.0, generated by my own Python template scripts; not used in base pretraining):
Roadmap / planned future work
- Continued pretraining on free compute: further resumable tranches on free 2× T4 GPUs, each judged by evaluation gates frozen before the run.
- Matched GPT-2 re-evaluations after each run: the same frozen-metric comparison against GPT-2 small, with confidence intervals, published whether the gap closes or not.
- Later: instruction/chat fine-tuning, then a public chat demo Space. This is planned and not built yet. The current model is a base checkpoint with no chat or instruction-following ability.
Links