tiktoken is a fast BPE tokeniser for use with OpenAI's models.

Specify a github or local repo, github pull request, arXiv or Sci-Hub paper, Youtube transcript or documentation URL on the web and scrape into a text file and clipboard for easier LLM ingestion

Online playground for OpenAPI tokenizers

go version of tiktoken

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.

High-Performance Implementation of OpenAI's TikToken.

Pure Go implementation of OpenAI's tiktoken tokenizer

The missing tiktoken training code

Ready-made tokenizer library for working with GPT and tiktoken

SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.

Go, Wasm bindings for HF Tokenizers and Tiktoken

An elegant LLM chat UI forked from chatgpt-demo of @anse-app. Index site at https://free-chat.asia

Unofficial ruby binding for tiktoken by way of rust

This is a port of the tiktoken

Analysis of OpenAI's ChatGPT

📐 Fast token estimation at 96% accuracy of a full tokenizer in a 2kB bundle

Token calculation for OpenAI models, using `o200k_base` `cl100k_base` `p50k_base` encoding.

OpenAI's tiktoken but with node bindings

Nodes: Wildcard Processor, Get File Path, Save Text File, Download Image from URL, Tiktoken Tokenizer, String Cleaning, String Text Splitter, String Text Extractor, Format Date Time, Load Text-Image Pairs, Metadata Extractor, Audio Visualizer, Load Image Advanced, Colorful Starting Image, Groq LLM, VLM, ALM API

High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.

OpenShield is a new generation security layer for AI models

Openai's tiktoken implementation written in Swift

What are learned in tiktoken?

The ChatGPT Long Term Memory package is a powerful tool designed to empower your projects with the ability to handle a large number of simultaneous users and external sources.

A fast RWKV Tokenizer written in Rust

tiktoken tokenizer for postgres

C++ implementations for various tokenizers (sentencepiece, tiktoken etc).

A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.

Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.

Kotlin multiplatform BPE tokenizer library for OpenAI models