Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
Data Science
Simon Willison's Blog·May 20·5 min read
How fast is 10 tokens per second really?
TL;DR
When LLM (large language model) vendors advertise token throughput speeds like "30 tokens per second," it's unclear what that feels like in practice. Mike Veerman built an interactive HTML simulator that visualizes token output at speeds ranging from 5 to 800 tokens per second.
✦ Why It Matters
Visualizing token speeds helps engineers make informed model selection decisions based on actual perceived responsiveness, not abstract metrics.
Key Takeaways
- Experiment with the app to visualize token output speeds.
- Compare different models based on their token generation rates.
- Use insights from the simulation to inform model selection.
- Understand the practical implications of advertised performance metrics.
Full Summary
Understanding the output speed of large language models (LLMs) is crucial for developers and researchers working with AI. Mike Veerman created an HTML application that simulates LLM token output speeds ranging from 5 to 800 tokens per second.
This interactive tool allows users to visualize how different speeds translate into real-time text generation, making it easier to grasp the implications of advertised performance metrics. For instance, a model claiming '30 tokens/second' can be better understood through this simulation, providing a tangible sense of its capabilities.
By experimenting with various speeds, users can appreciate the differences in responsiveness and throughput. This insight can inform decisions on model selection and deployment in applications requiring real-time interaction.