We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
Lobsters·Jun 30·5 min read
Serving Local AI on my Jetson through Durable Streams
TL;DR
There was a need for a self-hosted text-to-speech solution that allows users to convert text into audio without relying on third-party services. A local AI model, Kokoro-82M, was deployed on an NVIDIA Jetson Orin Nano to create a text-to-speech app.
✦ Why It Matters
Engineers can leverage local AI models for efficient, self-hosted applications without third-party dependencies.
Key Takeaways
- Explore local AI for self-hosted applications.
- Implement incremental audio output for better user experience.
- Utilize NVIDIA Jetson for efficient model deployment.
- Design a clean API for versatile application support.
Full Summary
As local AI becomes more practical, the need for self-hosted solutions grows, particularly for applications like text-to-speech (TTS). A text-to-speech app was built using the Kokoro-82M model on an NVIDIA Jetson Orin Nano, which features 1024 CUDA cores and 32 tensor cores, capable of 67 trillion operations per second (TOPS).
The app allows users to select text, choose a voice, and generate audio links for later access. While the inference speed is slower than typical web requests, the model can produce speech faster than real-time.
The design leverages incremental audio output, enabling users to hear generated sentences as they are produced, rather than waiting for the entire paragraph to finish. This approach enhances user experience and maintains the efficiency of the workload.
The findings suggest that local inference can be effectively managed with a clean API for various applications.