Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·Jun 12·5 min read
Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Models
TL;DR
Omni-modal large language models (OLLMs) struggle to generate synchronized speech and 3D facial animations, which are crucial for realistic human-computer interaction. The Ex-Omni model was developed to address this by decoupling semantic reasoning from temporal generation using a blendshape-aware speech unit generator and a blendshape decoder.
✦ Why It Matters
Engineers can leverage Ex-Omni to create more engaging and realistic interactive applications that require synchronized speech and facial animations.
Key Takeaways
- Explore Ex-Omni for improved human-computer interaction.
- Utilize the TQGF mechanism for better semantic integration.
- Consider the InstructS2SF-1200K dataset for training models.
- Implement blendshape techniques for realistic facial animations.
Full Summary
Omni-modal large language models (OLLMs) aim to integrate various forms of data, such as text and images, but have not effectively produced synchronized speech and 3D facial animations. Ex-Omni is an open-source model that enhances OLLMs by incorporating a blendshape-aware speech unit generator, which creates speech units that guide the timing of facial animations, and a blendshape decoder that translates these units into facial movements.
A novel token-as-query gated fusion (TQGF) mechanism allows for controlled integration of semantic information. The model was pre-trained on a new dataset, InstructS2SF-1200K, containing 1.2 million samples.
Results indicate that Ex-Omni not only maintains strong speech understanding and generation capabilities but also improves synchronization between audio and visual outputs, achieving lower latency in facial animation generation. These advancements suggest significant potential for enhancing human-computer interaction in applications like virtual assistants and gaming.