code to train a gpt-2 model to train it on tiny stories dataset according to the TinyStories paper

This Repository provides a Jupyter Notebook for building a small language model from scratch using 'TinyStories' dataset. Covers data preprocessing, BPE tokenization, binary storage, GPU memory management, and training a Transformer in PyTorch. Generate sample stories to test your model. Ideal for learning NLP and PyTorch.

No description available

Reproducing GPT on the TinyStories dataset

Tinystories version of gpt with custom inference engine

Small indentation-based CYOA engine

A TinyStories LM with SAEs and transcoders

a transformer implemented primarily using einops and trained on the tinystories dataset

small language models based on decoder-only transformers pretrained on TinyStories from scratch.

Repo aiming at benchmarking several variant of transformer architectures for pre training, on datasets TinyStories and Pints 1.5

Code for the TinyStories experiments from "Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks".

A replication of Anthropic's Sleeper Agents and Stage-wise Model Diffing on a TinyStories language model.

- Finetuning Llama-2-7b using QLoRA using TinyStories dataset - This finetuning was conducted on the free version of Google collab with less than 15 Gb VRAM thanks to QLora and Parameter-Efficient Fine-Tuning.

Implementation of TinyStories Paper, my first exposure to small language models

A reimplementation of the TinyStories paper for generating a stories dataset in non-English languages

0.2M and 2.5M parameter TinyStories model trained on GRU with and without attention.

Finetuning Llama-2-7b using QLoRA using TinyStories dataset

Train and run a small Llama 2 model from scratch on the TinyStories dataset.

Train Tiny Stories dataset from the paper - "TinyStories: How Small Can Language Models Be and Still Speak Coherent English?"

Train GPT style model on tinystories dataset

A 2M parameter neural language model trained on the TinyStories corpus.

Building transformers from scratch, one concept at a time. 13 Jupyter notebooks evolving from bag-of-words to mixture of experts, trained on TinyStories.

A CLI story generator powered by TinyStories-1M

Small Language Model trained from scratch on TinyStories dataset

Reproduction of TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

TinyStories on Modal

An attempt to use the huggingface transformers library recreate the TinyStories models.

An implementation of small language models for generating children stories. Inspired by "TinyStories" (Eldan and Li, 2023).

(WIP) llama2 scratch ttnn implementation to be trained with TinyStories. Goal: having a little gnome telling about stories on Tenstorrent e75 accelerator :-)

🧠 Gemma 3 architecture implemented from scratch & pre-trained on TinyStories | 164.6M params | MQA, Sliding Window, RoPE, GeGLU | Perplexity 5.96 | Every component hand-coded in PyTorch