Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization, with PyTorch/CUDA

Port of Andrej Karpathy's minbpe to Rust

a Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization in pure C.

Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization. Pure JavaScript port of karpathy/minbpe

No description available

Byte-level byte pair encoding (BPE) in Haskell

port of Andrjey Karpathy's minbpe to Mojo

No description available

Implementation of Mini-bpe written purely in Rust

a c++ version implementation of minbpe

Experiements with llm tokenization in cpp

No description available

This crate is a rust porting of Andrej Karpathy implementation of Byte Pair Encoding (BPE) algorithm available here https://github.com/karpathy/minbpe

No description available

A Minimal Byte Pair Algorithm application in Golang to simulate GPT-2 and GPT-4 Tokenizer

This is the minimall version of gpt tokenizer that uses Byte-pair encoding algorithm on it's core

An implementation of https://github.com/karpathy/minbpe/tree/master in Zig

Mirror of karpathy/minbpe

Java based implementation of minbpe by karpathy@ https://github.com/karpathy/minbpe

翻译Andrej Karpathy的minbpe。原仓库地址: https://github.com/karpathy/minbpe

Fork of Karpathy's minbpe

Tokenizer from scratch based on minbpe.

Educational reimplementation of Byte Pair Encoding (BPE) with regex tokenization, merge order control, and chunk-based encoding – inspired by Karpathy's minbpe.

No description available

Byte Pair Encoding Algorithm

No description available

No description available

A Go implementation of the minbpe tokenizer

My version of @karpathy's Minbpe implementation ;)