Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
Data Science
Towards Data Science·Aug 22·5 min read
Why We Fine-Tuned SigLip (And Why That’s Not Always the Right Call)
TL;DR
SigLip was fine-tuned using LoRA to address an under-labeling issue, demonstrating improved performance. However, the decision to fine-tune depends on specific project needs.
✦ Why It Matters
Evaluate your project's data and goals to decide if fine-tuning with LoRA is beneficial.
Key Takeaways
- Evaluate whether to use an API or train a custom classifier based on your needs.
- Fine-tune models like SigLIP to improve classification performance and reduce under-labeling.
- Select the appropriate foundation model to maximize performance gains.
- Consider cost implications when classifying large volumes of images.
Full Summary
Alma Media created a multi-label image classification system to automatically tag real estate photos with 23 classes, including LIVING ROOM and KITCHEN. The team explored whether to use a third-party API or train their own model, ultimately opting for fine-tuning the Google SigLIP model.
They trained on a proprietary dataset of 40,000 annotated images, comparing a frozen model with a fine-tuned version using low-rank adapters (LoRA). Results showed that the fine-tuned model achieved an F1 score of 82.6%, significantly improving recall and reducing under-labeling to 3.4%.
The study also revealed that selecting the right foundation model, like DINOv2, could yield substantial performance gains without fine-tuning. These findings underscore the importance of model choice and fine-tuning in achieving high-quality image classification.