CalCo: A Hierarchical Bayesian Framework for Scalable Human-LLM Hybrid Labeling

With the emergence of Large Language Models (LLMs), data annotation workflows are increasingly shifting toward hybrid systems where LLM-generated labels complement or replace human effort. However, effectively incorporating these non-deterministic signals into existing human labeling processes remains a challenge, particularly in complex industrial settings involving multiple correlated tasks and diverse model architectures. In this work, we introduce CalCo, a comprehensive system deployed at scale, designed to integrate LLM and human labels in a principled manner. At the core of CalCo is a novel hierarchical Bayesian model that explicitly captures the complex dependencies inherent in the multi-task, multi-LLM annotation process. Unlike traditional approaches that treat annotation inputs in isolation, CalCo jointly models the inter-dependencies across related tasks and the correlations among diverse LLMs. This holistic approach allows the system to share statistical strength across questions and capture shared error profiles among models. We demonstrate the efficacy of our approach through extensive empirical results on both simulated and real-world datasets, showing that CalCo significantly outperforms baseline methods in a wide range of applications.