MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs

Open-ended question answering (QA) is a key task for evaluating the capabilities of large language models (LLMs). Compared to closedended QA, it demands longer answer statements, more nuanced reasoning processes, and diverse expressions, making refined and interpretable automatic evaluation both crucial and challenging. Traditional metrics like ROUGE and BERTScore struggle to capture semantic similarities due to different patterns between model responses and reference answers. Current LLM-based evaluation approaches, such as pairwise or listwise comparisons of candidate answers, lack intuitive interpretability. While pointwise scoring of each response provides some descriptions, it fails to adapt across different question contents. Most notably, existing methods overlook the distinction between factoid and non-factoid questions. To address these challenges, we propose MinosEval, a novel evaluation method that first distinguishes open-ended questions and then ranks candidate answers using different evaluation strategies. For factoid questions, it applies an adaptive key-point scoring strategy, while for nonfactoid questions, it uses an instance-aware listwise ranking strategy. Experiments on multiple open-ended QA datasets, including self-built ones with more candidate responses to complement community resources, show that MinosEval better aligns with human annotations and offers more interpretable results.

Question: Did the Magellanic Flotilla use a sextant to measure latitude and longitude while traveling around the world? Answer: … Magellan's voyage around the world took place from 1519 to 1522, and the principle of the sextant was proposed by Isaac Newton, who was born on January 4, 1643 … and the use of a sextant was not possible.

Question: List the five tallest mountains on Earth and the average of their peaks. Answer: the five tallest mountains on earth and their altitudes are: 1.Mount Everest: 8848 meters … average of their altitudes is (8848+8611+8586+8516+8485)/5 = 8609.2 meters.

Question: what are the themes and symbolism in the short story 'The Raven' by Edgar Allan Poe, one of the major writers of American literature? Answer: 'The Raven' is a poem by Edgar Allan Poe about the sorrow and loneliness of losing a loved one. The themes focus on tragedy … revealing the powerlessness and pain of human beings in the face of eternal loss.

Question: … please generate a resume for me that is customized for the position of [Sales Manager] … My name is [Ma Ming Feng Xiao Xiao] and I am a software developer, you model the rest of the information. Answer: Ma Ming Feng Xiao Xiao … Senior professional with 10 years of experience working across multiple disciplines, including software development and project management. Adept at analyzing market needs, driving team execution, and … Non-Factoid Point 1: In honor of Mumtaz Mahal, wife of Shah Jahan Instance: Excellent: … I'm always here to listen and cheer you on if you find it hard to cope … Good: … I'll always be here to support you! ... Fair: I know you've been having a hard time lately, but don't worry too much … Poor: Oh, it sounds like you've been under a lot of stress lately … Bad: … It should get better in a few days … Factoid Question Non-Factoid Question