AI · Jupyter Notebook

Mini GPT

A decoder-only language model trained in PyTorch, with the modern blocks written out in the open.

The training stack uses grouped-query attention, RoPE, RMSNorm, SwiGLU, and a tied embedding / LM head. It tokenizes local parquet shards into binaries and trains from those files. The README frames it as a tutor model built to be read, not just run.