OCDevel AI Podcast

The Machine Learning Workflow and the Train/Test Split: Data, Features, Training, and Honest Evaluation

OCDevel AI Podcast — The ML Workflow and Train/Test Split

The anchor episode of Phase 1: the end-to-end machine learning workflow as a loop, and the train/test split as the discipline that keeps your evaluation honest. Builds directly on the NumPy/pandas episode; unlocks every model that follows, because they all share the same fit/predict/score harness.

Education segment (anchor)

  • Workflow as a loop, not a line: problem framing, data collection, EDA, cleaning, feature engineering, model choice, training, evaluation, tuning, deploy, monitor, iterate. Roughly 80% of project time is data + features, 20% modeling. Often drawn as CRISP-DM with back-arrows.
  • Generalization is the goal: performance on unseen data from the same distribution. Training error is an optimistically biased estimate of true error, which is why you need held-out data.
  • Overfitting vs underfitting, capacity, and the bias-variance tradeoff (intuition only).
  • The split: 80/20, 70/30, 90/10 for big data; stratify for imbalance, never shuffle time series. Three-way split (train/validation/test) and k-fold cross-validation (Kohavi's 1995 case for 10-fold).
  • Data leakage taxonomy: preprocessing before split, target leakage (the Caruana et al. pneumonia case), temporal and group leakage, distribution shift.
  • scikit-learn facts verified against version 1.9.0: train_test_split, cross_val_score, the cross-validation guide, common pitfalls, DummyClassifier, and the Kaggle Data Leakage lesson.
  • Worked examples: Iris with a dummy baseline (~0.33) vs logistic regression (~0.97); California housing with a mean baseline (R² ~0.0) vs linear regression (R² ~0.58); correct vs leaky scaling.

News segment (week of June 18–22, 2026)

  • Noam Shazeer leaves Google for OpenAI: Gemini co-lead and "Attention Is All You Need" co-author departs (also see 9to5Google). Google reportedly paid ~$2.7B in 2024 to bring him back.
  • AI funding roundup: Odyssey ($310M Series B, world models), Hydra Host ($100M), Bland AI ($50M), Radical Numerics ($50M).
  • Open-weights leads (reportedly): GLM-5.2, MiniMax M3, VibeThinker-3B, Nemotron 3 Ultra.