data science related
Venugopal Manneni

Machine learning projects often fail not because of algorithms, but because of avoidable mistakes in data prep, model design, and deployment. Lessons from real-world experience:

  1. Skipping EDA – Jumping straight to modeling hides missing values, outliers, or skew.
    ✅ Always visualize data (histograms, box plots, heatmaps) or use tools like pandas-profiling / sweetviz.
  2. Not Handling Missing Data Properly – Dropping rows/columns or filling with zeros distorts patterns.
    ✅ Use mean/median/mode imputation, KNN imputer, or ML-based imputation.
  3. Ignoring Data Leakage – Accidentally letting target info creep into training → inflated accuracy.
    ✅ Check correlations, enforce proper train/test splits, avoid future info in time-series.
  4. Overfitting – Too many layers/parameters, memorizing training data.
    ✅ Apply regularization (L1/L2), dropout, early stopping.
  5. Underfitting – Oversimplified models miss patterns.
    ✅ Increase model capacity, add better features, reduce over-regularization.
  6. Not Scaling Features – Raw features break SVM, KNN, logistic regression.
    ✅ Use StandardScaler, MinMaxScaler, or RobustScaler.
  7. Poor Train-Test Splitting – Random split on time-series or imbalanced data leads to bias.
    ✅ Use TimeSeriesSplit or stratify=y for class balance.
  8. Wrong Evaluation Metric – Accuracy on imbalanced data = misleading.
    ✅ Prefer Precision, Recall, F1, ROC-AUC, or PR curves.
  9. No Hyperparameter Tuning – Default params rarely optimal.
    ✅ Start with GridSearchCV, RandomizedSearchCV, or Optuna for Bayesian optimization.
  10. Ignoring Class Imbalance – High accuracy but useless in fraud detection/churn.
    ✅ Try SMOTE, class weights, or minority class oversampling.
  11. Assuming More Data Always Helps – Quantity ≠ quality.
    ✅ Focus on diverse, clean, relevant data; use augmentation instead of noisy expansion.
  12. Neglecting Feature Engineering – Relying only on raw features.
    ✅ Create domain-driven transformations, rolling features, encodings, PCA.

source: https://medium.com/pythoneers/13-machine-learning-mistakes-that-sabotage-your-models-and-how-to-fix-them-c00f86914411

Print Friendly, PDF & Email
Common Machine Learning Mistakes (and How to Avoid Them)

Venugopal Manneni

A doctor in statistics from Osmania University. I have been working in the fields of Analytics and research for the last 15 years. My expertise is to architecting the solutions for the data driven problems using statistical methods, Machine Learning and deep learning algorithms for both structured and unstructured data. In these fields I’ve also published papers. I love to play cricket and badminton.

Post navigation