Can SAEs reveal and mitigate racial biases of LLMs in healthcare?

LLMs are increasingly being used in healthcare. This promises to free physicians from drudgery, enabling better care to be delivered at scale. But the use of LLMs in this space also brings risks; for example, such models may worsen existing biases. How can we spot when LLMs are (spuriously) relying on patient race to inform predictions? In this work we assess the degree to which Sparse Autoencoders (SAEs) can reveal (and control) associations the model has made between race and stigmatizing concepts. We first identify SAE latents in gemma-2 models that appear to correlate with Black individuals. We find that these latents activate on reasonable input sequences (e.g., "African American") but also problematic words like "incarceration". We then show that we can use this latent to "steer" models to generate outputs about Black patients, and further that this can induce problematic associations in model outputs as a result. For example, activating latents associated with Black individuals increases the risk assigned to the probability that a patient will become "belligerent". We also find that even in this controlled setting where we causally intervene to manipulate only patient race, elicited CoT reasoning chains do not communicate race as a factor in the resulting assessments. We evaluate the degree to which such "steering" via latents might be useful for mitigating bias. We find that this offers improvements in simple settings, but is less successful for more realistic and complex clinical tasks. Overall, our results are mixed, and suggest that SAEs may offer a useful tool in clinical applications of LLMs to identify problematic reliance on demographics, as compared to CoT explanations (which should not be trusted in such settings). But mitigating bias via SAE steering may be only marginally effective for more realistic tasks.