AlphaQT-Bench: Diagnosing the Gap between Financial Code Generation and Quantitative Reasoning in LLMs

Large Language Models (LLMs) are increasingly applied to alpha mining in quantitative finance, marking a shift from generating simple symbolic formulas to producing executable, code-based strategies. While code generation offers greater expressiveness, it introduces critical risks absent in symbolic approaches, including temporal causality violations (look-ahead bias) and stateful logic bugs. Existing benchmarks largely rely on outcome-driven metrics (e.g., backtesting profitability), which often conflate market stochasticity or unintended information leakage with genuine reasoning competence. We introduce ALPHAQT-BENCH, a diagnostic benchmark for instruction-grounded financial code generation under strict semantic and temporal constraints. Unlike general-purpose coding benchmarks, ALPHAQT-BENCH adopts a multi-layer evaluation protocol that assesses: (1) executability, (2) causality safety via a dynamic truncation test, (3) functional accuracy, and (4) structural compliance through vectorization analysis. Experiments across 12 representative LLMs reveal a substantial gap between surface-level success (e.g., executability) and verified quantitative correctness, as many models fail under causal, structural, or functional constraints. By shifting evaluation from profitability to process reliability, ALPHAQT-BENCH provides a principled safety audit for emerging LLM-based quantitative systems.