Ranking quality
ROC-AUC
A recommender only ever logs what it chose to show, so every offline number is measured on traffic the model helped select. For two weeks Kuaishou injected uniformly random videos into live feeds, which gives an unbiased comparison over the identical calendar window — so any gap is the exposure mechanism, not drift.
The cost turns out to be asymmetric. Switch the exposure below and watch which metrics move.
Why it matters. Ranking transfers almost perfectly — lift falls only 2.0% when exposure becomes random. But mean predicted probability goes from 0.96× actual to 2.07×: the model is twice as confident as reality on traffic it did not select. A team validating offline would watch the ranking hold, conclude the model transfers, and ship probabilities that are silently wrong.
Each bar is measured against that split's own base rate, which differ by 3.80×. The curves track closely at every depth — the ordering the model produces is real, not an artefact of what the recommender chose to show.
| Calibrator | Random traffic | Gate | Served traffic | Gate |
|---|
Calibration is a property of the exposure policy, not of the model. Fit a calibrator on nine days of randomized logs and apply it forward: it ships on randomized traffic and is rejected on served traffic, by almost exactly the same factor in the other direction. The Brier gate catches this on its own, with no special-casing.