Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs
TL;DR
Layer pruning in large language models (LLMs) causes a mismatch in hidden states, degrading performance. Ghosted Layers is a training-free recovery module that aligns these activations using a closed-form optimal linear operator.
✦ Why It Matters
Engineers can use Ghosted Layers to enhance pruned LLMs without incurring the costs of retraining.
Key Takeaways
- Implement Ghosted Layers to recover pruned models.
- Use a small calibration set for optimal alignment.
- Avoid retraining to save time and resources.
- Maintain performance while reducing model size.
Full Summary
Layer pruning is a technique that removes entire blocks from Transformer models to reduce their size, but it creates a mismatch between the hidden states processed by the remaining layers. Ghosted Layers is introduced as a solution to this problem, functioning as a recovery module that does not require retraining the model.
It calculates an optimal linear operator from a small calibration dataset to realign the activations between layers. This approach allows for the effective recovery of performance metrics that would otherwise suffer due to the pruning.
Results indicate that using Ghosted Layers can significantly mitigate the performance drop, although specific numerical improvements were not detailed. The implications of this work suggest that engineers can maintain model efficiency while preserving performance, making it easier to deploy smaller models in production environments.