How Misinformation Density Affects Health Information Search

Search engine results can include misinformation that is inaccurate, misleading, or even harmful. But people may not recognize or realize false information results when searching online. We suspect that the percentage of misinformation search results (misinformation density) may influence people's search activities, learning outcomes, and search experience. We conducted a zoom-mediated "lab" user study to examine this matter. The experiment used a between-subjects design. We asked 60 participants to finish two health information search tasks using search engines with High, Medium, or Low misinformation density levels. To create these experimental settings, we trained task-dependent text classifiers to manipulate the number of correct and misinformation results displayed on SERPs. We collected participants' search activities, responses to pre-task and post-task surveys, and answers to taskrelated factual questions before and after searching. Our results indicate that search result misinformation density strongly affects users' search behavior. High misinformation density made people search more frequently, use longer queries, and click on more results. However, such increased search activities did not lead to better search outcomes. Participants using the High misinformation density search engine answered factual questions less accurately and learned very limitedly from a search session than the two other systems. Moreover, participants in systems with a balanced amount of correct and misinformation results (Medium) could learn factual knowledge as effectively as others in a system with little misinformation (Low). Surprisingly, participants using different misinformation density systems did not rate their perceived goodness of search systems with significant differences, indicating that search engine misinformation may adversely but imperceptibly affect people and society. Our findings have disclosed the effects of misinformation density on health information search and offered insights to improve online health information search. CCS CONCEPTS • Information systems → Users and interactive retrieval; Web search engines.