Alexey Borsky · 2026-09-19 · notable
mini-AGI — a model that keeps learning on an 8GB laptop GPU
mini-AGI is an MIT-licensed byte-level model that trains from scratch on a single 8GB GPU and keeps learning from a stream of text without forgetting earlier subjects. Its weights live on disk and are paged into VRAM as needed.
mini-AGI trains a byte-level model from scratch on one 8GB laptop GPU and keeps learning without forgetting what it already read.
Key specs
| Parameters | 540.1M |
|---|---|
| Vram needed | 8 GB |
What is it?
mini-AGI is a continual-learning experiment published on 19 September 2026: a byte-level language model that trains from scratch on a single 8GB VRAM GPU and keeps improving as new text arrives. Its weights are stored as individual files on disk and paged into VRAM on demand, so the model's size is bounded by disk space rather than by the graphics card. Alexey Borsky, the author, calls it a toy-level model and a proof of concept, not a frontier system.
How does it work?
Characters pass through two dense prelude blocks and then a recurrent block applied up to 24 times, each pass picking its own top-8 experts from a shared pool. A PonderNet halting head lets easy characters exit early and gives harder ones more depth. Forgetting is held off by a split learning rate: the trunk — embeddings, routers and halting head — trains at a tenth of the experts' rate. Experts grow by recombining hidden units from existing ones, and are pruned on how often they are used rather than on their gate values, which the author found to be anti-predictive.
Why does it matter?
Continual learning usually breaks because new data overwrites old: read chess long enough and the model loses Wikipedia. mini-AGI reports keeping 99.84% of its progress against chance on the other seven subjects while reading 524,000 characters of chess. Running on hardware plenty of people already own turns that from a claim needing a cluster into something a curious engineer can rerun on a laptop.
Who is it for?
ML researchers and hobbyists with a single GPU
Try it
python3 train.py read data/train --save --weights-dir weights --held-out data/val