minhphuc429 7da7e887bf feat: use official tokenizers for 99.99% accuracy ...

Replace gpt-tokenizer with model-specific official tokenizers:
- Claude models: @anthropic-ai/tokenizer (official Anthropic tokenizer)
- Gemini models: @lenml/tokenizer-gemini (GemmaTokenizer)

Changes:
- Add @anthropic-ai/tokenizer and @lenml/tokenizer-gemini dependencies
- Remove gpt-tokenizer dependency
- Update count-tokens.js with model-aware tokenization
- Use getModelFamily() to select appropriate tokenizer
- Lazy-load Gemini tokenizer (138MB) on first use
- Default to local estimation for all content types (no API calls)

Tested with all supported models:
- claude-sonnet-4-5, claude-opus-4-5-thinking, claude-sonnet-4-5-thinking
- gemini-3-flash, gemini-3-pro-low, gemini-3-pro-high

2026-01-14 16:04:13 +07:00

2.6 KiB

Raw Blame History

View Raw