Airline Satisfaction: Model Validation

The question

Can CatBoost add useful information to an existing airline-satisfaction model? I compared a fixed blend across validation folds, then checked whether the result carried through to a competition submission.

The input is Kaggle’s Playground Series S6E10 competition data. Its generated records are a modelling benchmark, not an airline’s operational customer survey. The task predicts satisfaction; it does not identify what would cause a passenger to become satisfied.

The experiment

A fixed blend combines 70% of the existing tree/neural predictions with 30% CatBoost predictions. CatBoost uses native categorical inputs, additional rating categories and route-distance features. Competition target encodings are cross-fitted inside each outer training fold.

The retained confirmation code evaluates folds 2 and 3 after development on fold 1. Early stopping uses outer validation labels, and those folds have prior screening history. These are exploratory checks, not untouched final-test evidence.

What I learned

A model can add complementary predictions in validation without improving the submitted result. I kept the stronger existing submission rather than promoting the weaker candidate. Further work needs a fresh evaluation design and a meaningful gain against the exact retained best before spending more compute. A production decision would also need calibration, error analysis and a clear use case.

Evaluation scope and limits

The saved account-best public AUC is 0.96188 from a separate community-assisted ensemble using published prediction inputs and my own model components. It is not the CatBoost result or an independently developed end-to-end model. Public leaderboard scores and cross-validation scores use different evaluation data; no current rank or final private-test result is claimed.

The code and saved review are retained evidence. The code depends on earlier feature preparation and model predictions from the original experiment environment; it is not a standalone training pipeline. The later submitted blend is documented in the saved submission review. Training was not rerun during this website review.

Sources and further inspection