Choice of Implicit Signal Matters: Accounting for User Aspirations in Podcast Recommendations

Recommender systems are modulating what billions of people are exposed to on a daily basis. Typically, these systems are optimized for user engagement signals such as clicks, streams, likes, or a weighted combination of such sets. Despite the pervasiveness of this practice, little research has been done to explore the downstream impacts of optimization choice on users, creators and the ecosystem they are offered in. We used a platform that caters recommendations to millions of people and show in practice what you optimize for can have a large impact on the content users are exposed to, as well as what they end up consuming.

In this work, we use podcast recommendations with two engagement signals: Subscription vs. Plays to show that the choice of user engagement matters. We deployed recommendation models optimized for each signal in production and observed that consumption outcomes substantially defer depending on the target used. Upon further investigation, we observed that users' patterns of podcast engagement depend on the type of podcast, and each podcast can cater to specific user goals & needs. Optimizing for streams can bias the recommendations towards certain podcast types, undermine users' aspirational interests and put some show categories at disadvantage. Finally, using calibration we demonstrate that informed balanced recommendations can help address this issue and thereby satisfy diverse user interests.