Mis-prompt: Benchmarking Large Language Models for Proactive Error Handling

Large language models (LLMs) have demonstrated significant advancements in error handling. Current error-handling works are performed in a passive manner, with explicit errorhandling instructions. However, in real-world scenarios, explicit error-handling instructions are usually unavailable. In this paper, our work identifies this challenge as how to conduct proactive error handling without explicit error handling instructions. To promote further research, this work introduces a new benchmark, termed Mis-prompt, consisting of four evaluation tasks, an error category taxonomy, and a new evaluation dataset. Furthermore, this work analyzes current LLMs' performance on the benchmark, and the experimental results reveal that current LLMs show poor performance on proactive error handling, and SFT on error handling instances improves LLMs' proactive error handling capabilities. The dataset will be publicly available. Ortegal is a comarca in the province of A Coruña, in Galicia, Spain. The three municipalities in the Ortegal comarca with a total population of 12,238 in 2018 are: 1. Cerdido 2.Mañón 3. Val de San Vicente. These three make up the municipalities in Ortegal. Ortegal has three municipalities with a total population of 12,238 in 2018, one of which was called Cerdido. What are the names of the other two municipalities? GPT-4o GPT-4o Passive Error Handling Proactive Error Handling You will be given a question. First, check if there are any mistakes in the question. If the question is correct, provide the answer. If there is an error, first tell the user that there is an error in the question, identify what is wrong, and correct the question if necessary. Then, guide the user to rephrase the question. Here is the question: Ortegal has three municipalities with a total population of 12,238 in 2018, one of which was called Cerdido. What are the names of the other two municipalities? There is an error in your original question. You mentioned that Ortegal has three municipalities, but it should be that Ortegal has four municipalities. The correct question should be: Ortegal has four municipalities, with a total population of 12,238 in 2018, one of which is Cerdido. What are the names of the other three municipalities? The answer is: Besides Cerdido, the other three municipalities are: Ortigueira, Cariño, and Mañón.