LLMs Killed Q&A Stars? Analyzing the Impact of LLM-Generated Answers on an Online Q&A Platform

Online question-and-answer (Q&A) platforms facilitate knowledge exchange through posted questions and answers. Recent advances in large language models (LLMs) have shown their strong capability in generating high-quality answers, leading to a recent surge in LLM-generated answers (LGAs) on Q&A platforms. In this paper, we conduct an in-depth analysis of how LGAs affect Naver Knowledge iN, the most popular Q&A platform in South Korea. To this end, we implement nine state-of-the-art LLM-generated text (LGT) detection methods and evaluate their performance on answers collected from Naver Knowledge iN. We then build an ensemble detector by stacking the three best-performing LGT detection methods, achieving an AUC of 0.9987 with a false positive rate below 1%. Using this LGA detector, we identify 75,558 LGAs among 1.46M answers. We find that LGAs tend to be longer, use more punctuation marks, and exhibit higher lexical diversity. However, LGAs do not show clear differences in user reactions, such as upvotes, downvotes, or selection rates by questioners. We also find that LGAs are primarily intended for knowledge sharing rather than personal experiences sharing. Finally, we observe a shift in the Q&A platform: questions increasingly move from simple fact-seeking to those involving complex contexts and seeking personal opinions or past experiences.