We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
Hacker News·29d ago·5 min read
Price per 1M tokens is meaningless
TL;DR
Comparing AI models solely by price per 1M tokens is misleading due to varying tokenization methods and efficiency. Different models can produce different token counts for the same input, affecting cost comparisons.
✦ Why It Matters
Evaluate AI model efficiency based on task completion costs rather than just token pricing to optimize expenses.
Key Takeaways
- Assess AI models based on task completion costs, not just token price.
- Understand the impact of tokenization on cost comparisons.
- Monitor changes in tokenization methods from AI labs regularly.
- Use benchmarks like Artificial Analysis to evaluate model performance.
Full Summary
Many companies are realizing that AI costs can escalate quickly, especially when comparing models by price per 1M tokens. Each AI lab uses its own tokenizer, which determines how text is split into tokens, leading to significant discrepancies in token counts for the same input.
For instance, OpenAI's models can vary widely in tokenization, and recent changes by Anthropic have increased token counts by 30%. Moreover, the value derived from each token can differ greatly; much of the token usage may be for 'thinking' processes that enhance output quality but are not immediately visible.
A comparison of various AI models shows that while some may appear cheaper per token, they may not be as efficient in completing tasks, leading to higher overall costs. For example, GPT-5.5, despite being more expensive per token than Claude Opus 4.8, completes tasks at nearly half the cost.