Baguan-TS: dual in-context learning model for time series forecasting with covariates

Transformers enable in-context learning (ICL) for rapid, gradient-free adaptation in time series forecasting, yet most ICL-style approaches rely on tabularized, hand-crafted features, while end-toend sequence models lack inference-time adaptation. We bridge this gap with a unified framework, Baguan-TS, which integrates the raw-sequence representation learning with ICL, instantiated by a 3D Transformer that attends jointly over temporal, variable, and context axes. To make this high-capacity model practical, we tackle two key hurdles: (i) calibration and training stability, improved with a feature-agnostic, target-space retrieval-based local calibration; and (ii) output oversmoothing, mitigated via context-overfitting strategy. On public benchmark with covariates, Baguan-TS consistently outperforms established baselines, achieving the highest win rate and significant reductions in both point and probabilistic forecasting metrics. Further evaluations across diverse real-world energy datasets demonstrate its robustness, yielding substantial improvements.

• We develop a Y-space RBfcst local calibration module-feature-agnostic and episode-specific-that improves calibration, data efficiency, and scalability when training larger 3D Transformers. In our experiments, this module consistently improves overall forecasting accuracy and robustness under injected noise compared to training without retrieval.

• We introduce a context-overfitting strategy that explicitly balances sample denoising and sample selection, stabilizing in-context learning in high-capacity models. The strategy consistently lowers training loss and restores periodic spike reconstruction, mitigating oversmoothing without harming trend accuracy.