NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 25·6 min read
Memorization Dynamics of Fill-in-the-Middle Pretraining
TL;DR
Fill-in-the-middle (FIM) training—a method that teaches language models to complete text in the middle of a passage—has unclear effects on verbatim memorization of training data. Researchers trained matched Llama 3.2 models using FIM versus standard left-to-right objectives on repeated text excerpts, measuring what exact phrases the models memorized.
✦ Why It Matters
Engineers training models with FIM should expect different memorization patterns than LTR and design evaluation protocols accordingly.
Key Takeaways
- FIM training produces shorter, partial-match memorization; LTR produces longer exact-match memorization.
- Memorization under FIM scales linearly with training data repetitions in tested ranges.
- FIM models rely on prefix context for recall, not suffix context alone.
- Evaluate memorization across multiple span lengths and probe formats to avoid missing behavior.
Full Summary
Fill-in-the-middle (FIM) is a pretraining objective—a training goal that shapes how models learn—used to enable causal language models (models that predict text sequentially) to infill missing text spans. Prior work left unclear how FIM affects verbatim memorization, the exact reproduction of training data.
Von Arx and Dieudonné conducted controlled experiments pretraining matched Llama 3.2 models (a specific model architecture) with FIM and standard left-to-right (LTR) objectives on a corpus containing intentionally repeated Gutenberg excerpts. Using prefix-based probes (queries starting with context before the target span), FIM recovered short or partial matches more often, while LTR assigned higher confidence to long exact continuations.
Verbatim extraction under FIM grew approximately linearly with repetition count. Critically, suffix context alone proved insufficient for FIM memorization; models still relied heavily on prefix information.
The findings demonstrate that single-span-length or single-format evaluations miss important memorization nuances.