Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·May 25·6 min read
HARNESS-LM: A Three-Phase Training Recipe for Harnessing SLMs in Sponsored Search Retrieval
TL;DR
Sponsored search systems struggle to balance retrieval quality (finding relevant ads) with production speed constraints. HARNESS-LM is a three-phase training framework that compresses large retrieval models into smaller, faster versions through knowledge distillation (transferring learned patterns from a large model to a small one).
✦ Why It Matters
Engineers can deploy faster, cheaper retrieval models without sacrificing quality by applying HARNESS-LM's three-phase distillation recipe to their systems.
Key Takeaways
- Use three-phase training: teacher fine-tuning, L2 representation alignment, then contrastive refinement for optimal student performance.
- Target sub-600M parameters to achieve 27x latency reduction while maintaining 98% of teacher accuracy in production.
- Validate embedding dimensionality, model scale, and architecture choices empirically before deployment to ensure real-world gains.
- Expect measurable business metrics: 1% revenue uplift and 20x throughput improvement on modern GPUs like A100.
Full Summary
Sponsored search platforms like Bing Ads must retrieve relevant advertisements quickly for millions of concurrent users, creating tension between retrieval quality and latency. Large embedding models (neural networks that convert text into numerical representations for similarity matching) achieve strong benchmark performance but are too slow for production.
HARNESS-LM addresses this by training a high-performance teacher model using a billion-parameter Small Language Model (SLM), then distilling its knowledge into a compact 600M-parameter student encoder through three phases: L2 alignment (matching query representations mathematically), embedding optimization, and contrastive refinement (learning to distinguish relevant from irrelevant results). On real Bing Ads data, the compressed model retained 98% of the teacher's precision while reducing query-encoder latency by 27x and increasing throughput 20x on NVIDIA A100 GPUs.
Online A/B testing showed measurable business impact: +1% revenue, +0.6% impressions, and +0.4% clicks compared to the existing 190M-parameter production system.