StereoMap: Quantifying the Awareness of Human-like Stereotypes in Large Language Models

Large Language Models (LLMs) have been observed to encode and perpetuate harmful associations present in the training data. We propose a theoretically grounded framework called STEREOMAP to gain insights into their perceptions of how demographic groups have been viewed by society. The framework is grounded in the Stereotype Content Model (SCM); a well-established theory from psychology. According to SCM, stereotypes are not all alike. Instead, the dimensions of Warmth and Competence serve as the factors that delineate the nature of stereotypes. Based on the SCM theory, STEREOMAP maps LLMs' perceptions of social groups (defined by sociodemographic features) using the dimensions of Warmth and Competence. Furthermore, the framework enables the investigation of keywords and verbalizations of reasoning of LLMs' judgments to uncover underlying factors influencing their perceptions. Our results show that LLMs exhibit a diverse range of perceptions towards these groups, characterized by mixed evaluations along the dimensions of Warmth and Competence. Furthermore, analyzing the reasonings of LLMs, our findings indicate that LLMs demonstrate an awareness of social disparities, often stating statistical data and research findings to support their reasoning. This study contributes to the understanding of how LLMs perceive and represent social groups, shedding light on their potential biases and the perpetuation of harmful associations. Keywords WARMTH-COMPETENCE ANALYSIS K-MEANS CLUSTERING EMOTION BEHAVIORAL TENDENCY STEREOTYPE CONTENT DICTIONARY EXTRACT RATINGS KEYWORDS ANALYSIS Competence Warmth Dimension Direction Cluster Extracted keywords from reasons (DAVINCI) Extracted keywords from reasons (GPT-3.5) Extracted keywords from reasons (BARD) Cluster 0 addicts stigmatized viewed, education criminal record, instability negative, trafficking money laundering, lack understanding false, power illegal, form recklessness lead, drive lack access, backgrounds making, health issues, opprtunities stigma negative, addicts negative undesirable, make difficult, lack resources addicts stigmized society, negative connotations criminal, housing individuals, employment financial stability, understanding root causes, safe spaces, cancers medical costs, chronic diseases heart, result choices lack, lowincome families, fitting ideal image, viewed underserving help, flawed addicts perceived, negative prejudices addicts, negative perceptions recovery sercives addicts, stigma poverty difficulty, maintain healthy lifestlye, taxpayers footing, help cover, job loss make, housing relationships unstable, diseases hiv-aids lead, total economic cost, United States highest, problems update resume, programs labbeled, make easier seek, networking field help Cluster 1 resourceful individuals able, society discriminated terms, postgraduate degrees compared, beliefs generally perceived, economic inequality, public platforms increasing, adapt chaining, means seen savvy, certain level stability, leading economic, economic discrimination, barriers economic individuals, resourcefulness overcoming economic, economic lowerclass individuals, society discriminated extent negative views economic, factors individuals stigmized, negative individuals underrepresented, negative attitudes beliefs, economic stigmatized society, politics media contribute, tend high education, society including filling, agnostics people issue, division polarization, recognized privileged backgrounds, migrant workers make, messages contribute ambiguity, lazy unmotivated undeserving, economically individuals stigmatized poverty discrimination lack, povery unemployment incarcerated, families born poverty, lack resources discrimination, money build assets, unemployment rate twice, groups jews episcopalians, poverty individuals,poverty lack access, discrimination low income, economic hardship individuals, values norms, discrimination lack opportunity, house generally viewed, lack opportunity discrimination Cluster 2 economic wealthy ability, criticism perceived role, pursue higher education, policy typically viewed, norms values, having access unique, serve great deal, launching running, forces disrupting, independent selfreliant rejecting, power shape direct, wealth prestige power, generally viewed wealthy, succeed influence wealth, influence wealth traditionally, having considerable wealth, wealth influence prestige wealth fame viewed, consumers able to use, use fame public, viewed famous individuals, lucrative brand, influential wealth status, luxury fame glamour, individuals significant wealth, significant wealth endeavors, fame influence, status fame influence, perceived significant wealth, hiring promoting recent, highincome potential nature, instability inequality salaries endorsement deals, accoding to forbes celebrity, lucrative endorsement, different types of invesments, deals for example LeBron, charge higher fees, hours average, lead inequality lack, median salary, salaries reac