Exciting Mood Changes: A Time-aware Hierarchical Transformer for Change Detection Modelling

Longitudinal language modelling has been receiving increasing attention, especially in downstream tasks such as mental health monitoring of individuals where modelling linguistic content in a temporal fashion is crucial. A key limitation in existing work is effective modelling of temporal sequences within Transformer-based language models. Here we address this challenge by introducing a novel approach for predicting 'Moments of Change' (MoC) in the mood of online users, by simultaneously considering users' linguistic and temporal context. A Hawkes process-inspired transformation layer is applied over a hierarchical transformer architecture to model the influence of time on users' posts -capturing both their immediate and historical dynamics. We perform experiments on the two existing datasets for the MoC task and showcase clear performance gains when leveraging the proposed layer. Our ablation study reveals the importance of considering temporal dynamics in detecting subtle and rare mood changes. Our results indicate that considering linguistic and temporal information in a hierarchical manner provides valuable insights into the temporal dynamics of modelling user generated content over time, with applications in mental health monitoring.