Large Language Bayes

Many domain experts do not have the time or expertise to write formal Bayesian models. This paper takes an informal problem description as input, and combines a large language model and a probabilistic programming language to define a joint distribution over formal models, latent variables, and data. A posterior over latent variables follows by conditioning on observed data and integrating over formal models. This presents a challenging inference problem. We suggest an inference recipe that amounts to generating many formal models from the large language model, performing approximate inference on each, and then doing a weighted average. This is justified and analyzed as a combination of self-normalized importance sampling, MCMC, and importance-weighted variational inference. Experimentally, this produces sensible predictions from only data and an informal problem description, without the need to specify a formal model. PROBLEM I've recorded the precipitation in Amherst, Massachusetts for each day in December 2024. Maybe there's some kind of pattern? Predict how much it will rain on the next day, January 1, 2025. Make sure to reflect that precipitation is never negative. DATA int num_days array[num_days] real precipitation // results in December GOAL int next // outcome for next day