Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
One Token per Multimodal Evidence: Latent Memory for Resource-Constrained QA
TL;DR
Existing question answering (QA) systems consume excessive tokens and storage due to raw text and image evidence. Latent Memory is a new approach that uses a single high-dimensional latent token for each evidence item, reducing resource consumption.
✦ Why It Matters
Engineers can leverage Latent Memory to build more efficient QA systems that reduce resource consumption significantly.
Key Takeaways
- Explore Latent Memory for efficient QA system design.
- Implement latent token compression to reduce resource usage.
- Evaluate performance on multimodal QA benchmarks.
- Consider training with reconstruction and contrastive objectives.
Full Summary
Current QA systems often rely on large language models (LLMs) and vision-language models (VLMs) that process raw text and images, leading to high token usage and storage demands. Latent Memory introduces a novel memory paradigm that compresses each piece of evidence into a single high-dimensional latent token using a small compressor LLM/VLM.
This allows the system to operate in a unified latent representation space, where queries retrieve relevant latent tokens instead of raw evidence. The compressor is trained with multiple objectives, including reconstruction and contrastive learning, to ensure each latent token is informative for various tasks.
Evaluations on seven text-only and multimodal QA benchmarks demonstrated that Latent Memory performs competitively against advanced retrieval-augmented generation (RAG) systems while consuming significantly fewer tokens. Notably, it excelled in image-grounded QA tasks, showcasing its efficiency and effectiveness.
These findings suggest a promising direction for resource-constrained applications in AI.