We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
Reddit r/LocalLLaMA·May 20·5 min read
How accurate can “whichllm” be?
TL;DR
Users lack reliable ways to identify which large language model (LLM—a neural network trained on text to generate responses) generated a given text output. The "whichllm" tool attempts to classify text by its source model through statistical fingerprinting of writing patterns.
✦ Why It Matters
Engineers deploying multiple LLMs need to understand model attribution limits before relying on fingerprinting for security or compliance verification.
Key Takeaways
- Model detection accuracy drops sharply when comparing similar-scale or similarly-trained LLMs.
- Output length and prompt style significantly impact detection reliability; test with realistic data.
- Instruction-tuned models share fingerprints, making them harder to distinguish than base models.
- Plan for 40–90% accuracy range; use ensemble or confidence thresholds for production systems.
Full Summary
The r/LocalLLaMA community debated the viability of "whichllm," a classification tool intended to identify the source large language model (LLM—an AI system trained on vast text to generate human-like responses) from its output alone. Users raised concerns about whether such detection could work reliably when models share similar architectures, training data, or fine-tuning approaches.
The discussion covered practical scenarios: distinguishing between closely related models like different versions of Llama, detecting outputs from proprietary versus open-source models, and handling adversarial cases where users deliberately obfuscate model signatures. Participants noted that detection accuracy depends heavily on output length, prompt specificity, and whether models have been instruction-tuned similarly.
Results from informal testing showed accuracy ranging from 40–90% depending on conditions, with performance degrading when comparing similar-scale models. The thread highlighted that while statistical fingerprints exist, they are fragile and context-dependent.