Chinese MentalBERT: Domain-Adaptive Pre-training on Social Media for Chinese Mental Health Text Analysis

In the current environment, psychological issues are prevalent and widespread, with social media serving as a key outlet for individuals to share their feelings. This results in the generation of vast quantities of data daily, where negative emotions have the potential to precipitate crisis situations. There is a recognized need for models capable of efficient analysis. While pre-trained language models have demonstrated their effectiveness broadly, there's a noticeable gap in pre-trained models tailored for specialized domains like psychology. To address this, we have collected a huge dataset from Chinese social media platforms and enriched it with publicly available datasets to create a comprehensive database encompassing 3.36 million text entries. To enhance the model's applicability to psychological text analysis, we integrated psychological lexicons into the pre-training masking mechanism. Building on an existing Chinese language model, we performed adaptive training to develop a model specialized for the psychological domain. We assessed our model's effectiveness across four public benchmarks, where it not only surpassed the performance of standard pre-trained models but also showed a inclination for making psychologically relevant predictions. Due to concerns regarding data privacy, the dataset will not be made publicly available. However, we have made the pre-trained models and codes publicly accessible to the community via: https://anonymous.4open. science/r/Chinese-MentalBERT-0893 . 1 Introduction Mental illnesses, particularly depression, impose a considerable strain on global societies. The World Health Organization reports that approximately 3.8% of the global population suffers from depression (Organization et al., 2023). Notably, the incidence of depression in China accounts for as high as 6.9% of the prevalence (Huang et al., 2019). Individuals experiencing emotional distress often 043 resort to passive coping mechanisms and seldom 044 seek professional help (Rüsch et al., 2005). Tra-045 ditional channels for emotional crisis intervention, 046 such as hotlines and psychological clinics, are not 047 designed to proactively identify individuals facing 048 emotional challenges (Organization et al., 2014). 049 Moreover, the resources for such interventions are 050 frequently inadequate. The stigma associated with 051 mental illness has led many to use social networks 052 as a primary outlet for expressing their emotional 053 struggles (Primack et al., 2017). Platforms like X 054 (Twitter), and Sina Weibo in China serve as venues 055 for individuals to share their feelings and opinions 056 in real time, with posts often providing immediate 057 insights into one's daily experiences and emotional 058 states (De Choudhury et al., 2013). Within specific 059 topics or hashtags on social media, there is a pro-060 nounced focus on the expression of negative emo-061 tions, with some users displaying evident suicidal 062 tendencies (Robinson et al., 2016). This situation 063 underscores the critical necessity to develop tools 064 aimed at enabling the early detection of such dis-065 tress signals and implementing timely intervention 066 strategies (Coppersmith et al., 2018). 067 Language pre-training models, such as 068 BERT (Devlin et al., 2018), have demonstrated 069 remarkable success across a variety of language 070 tasks and have seen extensive application (Koro-071 teev, 2021). Recently, the development of large 072 language model (LLM) technology has garnered 073