Accelerating the search for antimicrobial peptides with Google Antigravity
deepmind.google·2h ago
Data Science
Simon Willison's Blog·7d ago·5 min read
ttok 0.4
TL;DR
ttok 0.4 is a command-line interface (CLI) tool for counting tokens using OpenAI's tiktoken library. The latest update includes a new command to list available models and fixes for previous warnings.
✦ Why It Matters
Use ttok to efficiently count tokens in your text files today.
Key Takeaways
- Update your ttok tool to the latest version for new features.
- Use the --list-models command to explore available models.
- Integrate ttok with uvx for seamless token counting.
- Fix any Click warnings in your CLI tools for better performance.
Full Summary
ttok is a CLI tool designed to count tokens, which are units of text processed by language models. It utilizes OpenAI's open-source tiktoken library, which provides efficient tokenization.
After a period without updates, the tool has been refreshed to address a Click warning and improve continuous integration (CI) processes. A notable new feature is the --list-models command, allowing users to easily view available models for token counting.
Additionally, ttok can be integrated with uvx, enabling users to count tokens from any text file seamlessly. This update enhances usability and ensures compatibility with current development practices.