Accelerating the search for antimicrobial peptides with Google Antigravity
deepmind.google·2h ago
Data Science
Simon Willison's Blog·6d ago·5 min read
ttok 1.0
TL;DR
The release of ttok 1.0 updates the default tokenizer to align with GPT-5/GPT-6, addressing a critical oversight in version 0.4. This change is based on experimental evidence suggesting token compatibility across these models.
✦ Why It Matters
Update your tokenization processes to ensure compatibility with the latest GPT models today.
Key Takeaways
- Upgrade to ttok 1.0 for improved tokenizer compatibility.
- Verify your tokenization settings to avoid using outdated models.
- Leverage findings from William Liu for model compatibility.
- Stay informed about updates in GPT model tokenization.
Full Summary
ttok is a tool designed for tokenization, which is the process of converting text into a format suitable for machine learning models. In the latest release, ttok 1.0, the default tokenizer was switched from GPT-4 to GPT-5/GPT-6, correcting a previous misalignment.
This decision was prompted by findings from William Liu, who demonstrated that all seven GPT models in the 5 and 6 series share the same token count of 44,794 tokens across 31 fixtures. Importantly, GPT-6 does not introduce any changes in input count for this specific corpus.
The implications of this update are significant for developers using ttok, as it ensures compatibility with the latest models and enhances the overall user experience.