Airline Satisfaction: Model Validation

Read the case study · View validation and submission results. An exploratory model-validation case using Kaggle Playground S6E10 generated competition data.

Retained evidence

evidence/catboost-review.json preserves Experiment 75's saved metrics. evidence/catboost-experiment.py preserves the confirmation code. Fold 2 AUC: baseline 0.9612149528, blend 0.9612867352. Fold 3: baseline 0.9609630528, blend 0.9609970996. Fixed 30% CatBoost weight; three-fold blend mean AUC 0.9611053056. AUC is not accuracy.

Fold 1 selected the model/weight. Other folds have prior screening history and early stopping uses their labels. There is no untouched final-test estimate. Saved review metrics were checked for arithmetic consistency; training was not rerun for this release. The later submitted blend (Experiment 79) scored 0.96066 public ROC AUC, below the previous own-model submission of 0.96071. The saved account best is 0.96188 from a separate community-assisted ensemble; it is not the CatBoost result. No current rank or final private-test result is claimed.

Inspect the saved submission review

Reproduction boundary

The script requires the original Kaggle runtime’s X_65, fold definitions, target encodings and baseline out-of-fold predictions. It is an inspection extract, not a standalone pipeline. Raw competition data, models, predictions and unrelated community artifacts are excluded. Fresh evaluation requires rebuilding the feature environment under competition terms.

Sources and attribution

• Competition
• Original public dataset
• Route-feature inspiration
• LightGBM feature inspiration
• A separate community-assisted ensemble used Golem OOF community predictions. Its score is not a CatBoost result or independent implementation claim.