Relevant or Random: Can LLMs Truly Perform Analogical Reasoning?

Analogical reasoning is a unique ability of humans to address unfamiliar challenges by transferring strategies from relevant past experiences. One key finding in psychology is that compared with irrelevant past experiences, recalling relevant ones can help humans better handle new tasks. Coincidentally, the NLP community has also recently found that selfgenerating relevant examples in the context can help large language models (LLMs) better solve a given problem than hand-crafted prompts. However, it is yet not clear whether relevance is the key factor eliciting such capability, i.e., can LLMs benefit more from selfgenerated relevant examples than irrelevant ones? In this work, we systematically explore whether LLMs can truly perform analogical reasoning on a diverse set of reasoning tasks. With extensive experiments and analysis, we show that self-generated random examples can surprisingly achieve comparable or even better performance on certain tasks, e.g., 4% performance boost on GSM8K with random biological examples. We find that the accuracy of self-generated examples is the key factor and subsequently design two novel methods with improved performance and significantly reduced inference costs. Overall, we aim to advance a deeper understanding of LLM analogical reasoning and hope this work stimulates further research in the design of self-generated contexts. * Equal contribution, order decided by coin flip. Input Output Q: The second and ninth terms of an arithmetic sequence are 2 and 30, respectively. What is the fiftieth term? # Instruction: ## Recall relevant examples: ## Solve the initial problem: # Relevant examples: Q: In an arithmetic sequence, the first term is 3 and the A: To find the tenth term, we can use the formula for the common difference is 4. What is the tenth term? nth term of the sequence ... So the tenth term is 39. ... # Solve the initial problem: We can use the formula for the nth term of an arithmetic sequence a_n = a_1 + (n-1)d. We are given the values of a_2 and a_9 ... So the fiftieth term is 194. Prompt: self-generate relevant examples Your task is to tackle mathematical problems. When presented with a math problem, recall relevant problems as examples. Afterward, proceed to solve the initial problem. # Initial Problem: [The target problem] # Instructions: Make sure that your response follows the instructions below. ## Analogous Problems: Offer five diverse examples of math problems that are relevant or analogous to the initial problem. For each problem, elaborate on the solution and conclude with the ultimate answer (enclosed in ). For each problem: -After "Q: ", describe the problem -After "A: ", explain the solution and enclose the ultimate answer in . ## Solve the Initial Problem: Q: Copy and paste the initial problem here. A: Explain the solution and enclose the ultimate answer in here. Prompt: self-generate random examples Your task is to tackle mathematical problems. When presented with a math problem, recall random problems as examples. Afterward, proceed to solve the initial problem. # Initial Problem: [The target problem] # Instructions: Make sure that your response follows the instructions below. ## Random Problems: Randomly offer five diverse examples of math problems. For each problem, elaborate on the solution and conclude with the ultimate answer (enclosed in ). For each problem: -After "Q: ", describe the problem -After "A: ", explain the solution and enclose the ultimate answer in . ## Solve the Initial Problem: Q: Copy and paste the initial problem here. A: Explain the solution and enclose the ultimate answer in here.