Supernotes: Driving Consensus in Crowd-Sourced Fact-Checking

X's Community Notes, a crowd-sourced fact-checking system, allows users to annotate potentially misleading posts. Notes rated as helpful by a diverse set of users are prominently displayed below the original post. While demonstrably effective at reducing misinformation's impact when notes are displayed, there is an opportunity for notes to appear on many more posts: for 91% of posts where at least one note is proposed, no notes ultimately achieve sufficient support from diverse users to be shown on the platform. This motivates the development of Supernotes: AI-generated notes that synthesize information from several existing community notes and are written to foster consensus among a diverse set of users. Our framework uses an LLM to generate many diverse Supernote candidates from existing proposed notes. These candidates are then evaluated by a novel scoring model, trained on millions of historical Community Notes ratings, selecting candidates that are most likely to be rated helpful by a diverse set of users. To test our framework, we ran a human subjects experiment in which we asked participants to compare the Supernotes generated by our framework to the best existing community notes for 100 sample posts. We found that participants rated the Supernotes as significantly more helpful, and when asked to choose between the two, preferred the Supernotes 75.2% of the time. Participants also rated the Supernotes more favorably than the best existing notes on quality, clarity, coverage, context, and argumentativeness. Finally, in a follow-up experiment, we asked participants to compare the Supernotes against LLMgenerated summaries and found that the participants rated the Supernotes significantly more helpful, demonstrating that both the LLM-based candidate generation and the consensus-driven scoring play crucial roles in creating notes that effectively build consensus among diverse users. X Account @x_account BREAKING: In a stunning announcement, President Biden has announced that if he's reelected, he will cancel medical debt for millions of Americans. Let's go. POTUS does not have authority to cancel debt, SCOTUS ruled on previous attempt to cancel student loans. Biden would not be able to keep this campaign promise. https://www.npr.org/2023/06/30/1182216970/supreme-court-student- loan-forgiveness-decision-biden He's not going to do it. 1) He is currently the sitting president, being re-elected changes nothing 2) He does not have the power to do this Commonsense.com Biden's administration plans to ban medical debt from credit reports https://abcnews.go.com/Politics/sweeping-change-biden-administration-ban- medical-debt-credit/story?id=110997906