OCEAN: Offline Chain-of-thought Evaluation and Alignment in Large Language Models

Offline evaluation of LLMs is crucial in understanding their capacities, though current methods remain underexplored in existing research. In this work, we focus on the offline evaluation of the chain-of-thought capabilities and show how to optimize LLMs based on the proposed evaluation method. To enable offline feedback with rich knowledge and reasoning paths, we use knowledge graphs (e.g., Wikidata5M) to provide feedback on the generated chain of thoughts. Due to the heterogeneity between LLM reasoning and knowledge graph structures, direct interaction and feedback from knowledge graphs on LLM behavior are challenging, as they require accurate entity linking and grounding of LLM-generated chains of thought in the knowledge graph. To address the above challenge, we propose an offline chain-of-thought evaluation framework, OCEAN, which models chainof-thought reasoning in LLMs as a Markov Decision Process (MDP), and evaluate the policy's alignment with knowledge graph preference modeling. To overcome the reasoning heterogeneity and grounding problems, we leverage on-policy knowledge graph exploration and reinforcement learning to model a knowledge graph policy that generates token-level likelihood distributions for LLM-generated chain-of-thought reasoning paths, simulating knowledge graph reasoning preference. Then we incorporate the knowledge-graph feedback on the validity and alignment of the generated reasoning paths into inverse propensity scores and propose KG-IPS estimator. Theoretically, we prove the unbiasedness of the proposed KG-IPS estimator and provide a lower bound on its variance. With the off-policy evaluated value function, we can directly enable off-policy optimization to further enhance chain-of-thought alignment. Our empirical study shows that OCEAN can be efficiently optimized for generating chain-of-thought reasoning paths with higher estimated values without affecting LLMs' general abilities in downstream tasks or their internal knowledge.