Token calculation for OpenAI models, using `o200k_base` `cl100k_base` `p50k_base` encoding.
Stars
124
Forks
18
Watchers
124
Open Issues
3
Overall repository health assessment
No package.json found
This might not be a Node.js project
test: upgrade test app to net10 and split benchmarks into a dedicated project
971fd2fView on GitHubfix: harden BPE loading with cache fallback and in-memory fallback
3f49fc9View on GitHubModified tikToken.Encode in Program.cs to include allowedSpecialMode for special token handling.
9c04344View on GitHubMerge branch 'master' of https://github.com/aiqinxuancai/TiktokenSharp
535350eView on GitHub