Synergizing Large Language Models and Knowledge-Based Reasoning for Interpretable Feature Engineering

Feature engineering stands as a pivotal step in enhancing the performance of machine learning (ML) models, particularly with tabular data. However, traditional feature engineering methods are often time-consuming and requires case-by-case domain knowledge. In addition, as ML systems become more common, interpretability becomes increasingly important, especially among domain experts. To this end, we propose ReaGen, an automated feature engineering (AutoFE) approach that combines knowledge graphs (KGs) with large language models (LLMs) to generate interpretable features. ReaGen begins by symbolic REAsoning over the KG to extract relevant information based on datasets description. Then, it uses an LLM to iteratively GENerate meaningful features. Finally, to overcome challenges such as hallucinations and handling long contexts typical in LLMs, our model performs logical reasoning on the KG to ensure that the generated features maintain interpretability. ReaGen provides Python code for automatic feature generation and detailed explanations of feature utility. It leverages both LLM's internal knowledge and retrieved information from KGs. Experiments on public datasets demonstrate that ReaGen significantly improves prediction accuracy while ensuring high interpretability through human-like explanations for each feature. This work highlights the potential of integrating LLMs and KGs in feature engineering, paving the way for interpretable ML models.