Can AI Relate: Testing Large Language Model Response for Mental Health Support

Disclaimer: This paper does not endorse the use of large language models (LLMs) for psychotherapy. Our goal is to benchmark equity and quality of care provided by LLMs already deployed in mental health settings.

We show that LLMs like GPT-4 use implicit and explicit cues to infer patient demographics like race. We then show that there are statistically significant discrepancies between patient subgroups: Responses to Black posters consistently have lower empathy than for any other demographic group (2%-13% lower than the control group). Promisingly, we do find that the manner in which responses are generated significantly impacts the quality of the response. We conclude by proposing safety guidelines for the potential deployment of LLMs for mental health response.