Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
Data Science
Towards Data Science·Aug 14·5 min read
My Model Was Cheating on Its Own Test
TL;DR
A car price prediction model inadvertently accessed test data through a flawed preprocessing pipeline, resulting in an inflated R squared score of twelve points. This incident highlights the importance of robust data handling in machine learning.
✦ Why It Matters
Implement strict data separation protocols in your machine learning workflows to avoid model validation errors.
Key Takeaways
- Always split data before fitting any preprocessing steps.
- Use training data only to inform scaling and outlier handling.
- Validate your model on a separate validation set.
- Check for data leakage to ensure honest model performance.
How It Works
The model uses a multilayer perceptron (MLP) to predict car prices, which involves preprocessing steps like scaling numeric features and encoding categorical variables. The preprocessing must be done correctly to avoid using test data inappropriately, which can inflate performance metrics.