Reimagining service delivery in the agentic era with Google Public Sector
cloud.google.com·19h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs
TL;DR
Layer pruning in large language models (LLMs) causes a mismatch in hidden states, degrading performance. Ghosted Layers is a training-free recovery module that aligns these activations using a closed-form optimal linear operator.
✦ Why It Matters
Engineers can use Ghosted Layers to enhance pruned LLMs without incurring the costs of retraining.
Key Takeaways
- Apply Ghosted Layers to recover performance in pruned models.
- Use a small calibration set for optimal activation alignment.
- Expect improved accuracy and perplexity metrics.
- Maintain efficiency gains from layer pruning.
How It Works
Ghosted Layers solves the activation alignment problem by deriving a linear operator that adjusts the outputs of surviving layers to match the expected distributions. This operator is calculated using a small calibration dataset, allowing for effective recovery of performance without the need for retraining the entire model.