Back to search
Top Contributors
85
commits
22
commits
16
commits
7
commits
6
commits
6
commits
4
commits
3
commits
3
commits
3
commits
C++ implementations for various tokenizers (sentencepiece, tiktoken etc).
Stars
49
Forks
26
Watchers
49
Open Issues
21
Overall repository health assessment
No package.json found
This might not be a Node.js project
85
commits
22
commits
16
commits
7
commits
6
commits
6
commits
4
commits
3
commits
3
commits
3
commits
Exclude abseil utility programs from OSS Buck build (#174)
b642403View on GitHubOptimize StringIntegerMap construction: counting sort + skip debug-only duplicate checks
1c43247View on GitHubFix PCRE2 UTF-8 validation errors for Qwen tokenizers (#171)
d43c287View on GitHub