tiktokenizer

dqbd•PUBLIC

View on GitHub

Online playground for OpenAPI tokenizers

chatgptnextjsopenait3-stacktiktokentokenizer

MIT License

Created on Mar 2, 2023

Updated on Apr 7, 2026

Stars

1.6k

Forks

168

Watchers

1.6k

Open Issues

21

Repository Health Score

💛

73/100

Good

Overall repository health assessment

Score Breakdown

Activity

Active development - updated this week

30/30

100%

Issues Analytics

23

Total Issues

All time

9

Open

39% of total

14

Closed

Recent Commits

Merge pull request #35 from MartinKocour/whisper

David Duong•11 months ago

d71e128View on GitHub

Add huggingface whisper tokenizer

Martin Kocour•1 year ago

d5c426cView on GitHub

Merge pull request #34 from dqbd/dqbd/qwen-deepseek

David Duong•1 year ago

12195f7View on GitHub

feat: add qwen2.5, deepseek R1 models

Tat Dat Duong•1 year ago

7e662cdView on GitHub

chore: remove outdated pricing info

Tat Dat Duong•1 year ago

b515e85View on GitHub

Prevent token id collision

Tat Dat Duong•1 year ago

071fb6eView on GitHub

Fix types

Tat Dat Duong•1 year ago

84fd130View on GitHub

Add o200k_base

Tat Dat Duong•1 year ago

d2350ebView on GitHub

feat: add auto resizing

Tat Dat Duong•1 year ago

c4a63fbView on GitHub

Replace dotenv with dotenv-cli

Tat Dat Duong•1 year ago

08ea735View on GitHub

Maybe now dotenv will be detected?

Tat Dat Duong•1 year ago

640f144View on GitHub

Download tokenizers on build

Tat Dat Duong•1 year ago

c083de6View on GitHub

Merge branch 'master' into pr/darknoon/20-1

Tat Dat Duong•1 year ago

3dc5b2cView on GitHub

Uncomment models, which doesn't seem to work at the moment

Tat Dat Duong•1 year ago

53fff89View on GitHub

Add a loading spinner

Tat Dat Duong•1 year ago

5af2fbeView on GitHub

View all commits

Community

1,555 stars, 168 forks

3/30

10%

Documentation

Has description, wiki, license

20/20

100%

Maintenance

1.4% issue ratio

20/20

100%

Health score is calculated based on activity, community engagement, documentation quality, and maintenance practices

Languages

TypeScript

91.5%

JavaScript

8.4%

CSS

0.1%

Dependencies42 total

24

Production

18

Development

Production Dependencies (24)

@radix-ui/react-checkbox^1.0.3

@radix-ui/react-popover^1.0.5

@radix-ui/react-select^1.2.0

@tanstack/react-query

Top Contributors

1

dqbd

User

48

commits

2

darknoon

User

8

commits

3

shadcn

User

2

commits

4

MartinKocour

User

1

commits

Languages

TypeScript

91.5%

JavaScript

8.4%

CSS

0.1%

Issues Analytics

23

Total Issues

All time

9

Open

39% of total

14

Closed

Dependencies42 total

24

Production

18

Development

Production Dependencies (24)

@radix-ui/react-checkbox^1.0.3

@radix-ui/react-popover^1.0.5

@radix-ui/react-select^1.2.0

@tanstack/react-query

^4.20.2

@trpc/client^10.9.0

@trpc/next^10.9.0

@trpc/react-query^10.9.0

@trpc/server^10.9.0

@vercel/analytics^0.1.11

@xenova/transformers2.17.1

bignumber.js^9.1.1

class-variance-authority^0.4.0

clsx^1.2.1

cmdk^0.2.0

graphemer^1.4.0

lucide-react^0.119.0

next^13.2.1

react18.2.0

react-dom18.2.0

superjson1.9.1

tailwind-merge^1.10.0

tailwindcss-animate^1.0.5

tiktoken^1.0.15

zod^3.20.6

Development Dependencies (18)

@types/eslint^8.21.1

@types/node^18.14.0

@types/prettier^2.7.2

@types/react^18.0.28

@types/react-dom^18.0.11

@typescript-eslint/eslint-plugin^5.53.0

@typescript-eslint/parser^5.53.0

autoprefixer^10.4.7

dotenv-cli^7.4.1

eslint^8.34.0

eslint-config-next^13.2.1

postcss^8.4.14

prettier^2.8.1

prettier-plugin-tailwindcss^0.2.1

tailwindcss^3.2.0

tsx^4.7.3

typescript^4.9.5

vitest^0.30.0

tiktokenizerv0.1.0

61% of total

3mo

Avg Close Time

Avg response

Issues Activity: Last 6 months

Hottest Issues

1

#13 Mind if the OpenAI Cookbook repo points to your Tiktokenizer app?

3

4

open

2

#33 FR add DeepSeek V3/R1 and Qwen models

2

3

closed

3

#6 Tweak Request: Don't clear prompt textbox when switching models.

4

1

closed

4

#12 Why does the tiktokenizer demo output completely different token numbers than the actual npm package?

3

closed

5

#17 token count is inconsistent with OpenAI tokenizer

2

closed

Top Contributors

1

dqbd

User

48

commits

2

darknoon

User

8

commits

3

shadcn

User

2

commits

4

MartinKocour

User

1

commits

Recent Commits

Merge pull request #35 from MartinKocour/whisper

David Duong•11 months ago

d71e128View on GitHub

Add huggingface whisper tokenizer

Martin Kocour•1 year ago

d5c426cView on GitHub

Merge pull request #34 from dqbd/dqbd/qwen-deepseek

David Duong•1 year ago

12195f7View on GitHub

feat: add qwen2.5, deepseek R1 models

Tat Dat Duong•1 year ago

7e662cdView on GitHub

chore: remove outdated pricing info

Tat Dat Duong•1 year ago

b515e85View on GitHub

Prevent token id collision

Tat Dat Duong•1 year ago

071fb6eView on GitHub

Fix types

Tat Dat Duong•1 year ago

84fd130View on GitHub

Add o200k_base

Tat Dat Duong•1 year ago

d2350ebView on GitHub

feat: add auto resizing

Tat Dat Duong•1 year ago

c4a63fbView on GitHub

Replace dotenv with dotenv-cli

Tat Dat Duong•1 year ago

08ea735View on GitHub

Maybe now dotenv will be detected?

Tat Dat Duong•1 year ago

640f144View on GitHub

Download tokenizers on build

Tat Dat Duong•1 year ago

c083de6View on GitHub

Merge branch 'master' into pr/darknoon/20-1

Tat Dat Duong•1 year ago

3dc5b2cView on GitHub

Uncomment models, which doesn't seem to work at the moment

Tat Dat Duong•1 year ago

53fff89View on GitHub

Add a loading spinner

Tat Dat Duong•1 year ago

5af2fbeView on GitHub

View all commits

GitHub Explorer

tiktokenizer

Score Breakdown

Production Dependencies (24)

Production Dependencies (24)

Development Dependencies (18)

Issues Activity: Last 6 months

Hottest Issues