NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·1h ago
AI Research
arXiv CS.AI·May 25·6 min read
PGT: Procedurally Generated Tasks for improving visual grounding in MLLMs
TL;DR
Multimodal Large Language Models (MLLMs—AI systems combining vision and language) struggle with fine-grained visual tasks like spatial reasoning. Procedurally Generated Tasks (PGT) overlays geometric shapes on images to create dense training supervision that isolates visual understanding from semantic knowledge.
✦ Why It Matters
Engineers can use PGT to diagnose and fix spatial reasoning failures in vision-language models without architectural redesign.
Key Takeaways
- Overlay geometric primitives on training images to create explicit spatial supervision signals.
- PGT improves fine-grained understanding without requiring model architecture changes.
- Test spatial reasoning separately from semantic knowledge using procedurally generated benchmarks.
- Augment existing MLLM datasets with synthetic geometric tasks for +13-20% gains.
Full Summary
Multimodal Large Language Models (MLLMs)—neural networks processing both images and text—excel at general perception but fail at precise visual grounding tasks requiring spatial reasoning and quantitative understanding. Researchers introduced Procedurally Generated Tasks (PGT), a data generation framework that synthetically overlays unambiguous geometric primitives (shapes, lines, coordinates) onto real images.
This creates dense supervision signals that decouple visual grounding capability from semantic priors (learned associations). The team trained LLaVA-v1.5-Instruct, a popular MLLM, on augmented datasets combining original and PGT data.
Results showed +20% improvement on What'sUp benchmark and +13.3% on CV-Bench-2D (spatial reasoning tests), with state-of-the-art models gaining +5.5% to +8.3%. Findings suggest spatial reasoning deficits stem from insufficient training signals rather than architectural limitations.