The Cost of Scaling Down Large Language Models: Reducing Model Size Affects Memory before In-context Learning

Historically, the progress of artificial intelligence (AI) has been mercurial, alternating between surprising breakthroughs and periods of comparative drought. While that will continue to be true to some extent, the incentives and trends are clearer than they have been, and there is even a subfield dedicated to mathematically predicting how much progress to expect from additional investments. These predictions suggest that further performance gains will come from increasing the scale of investment in the current approaches, but that there are sharply diminishing returns. For example, simply increasing the computing budget from $10 million to$100 million increased the pass rate for AI-generated computer programs from about 65% to about 75%. A billion-dollar version of the model would apparently only reach about 80%, and a trillion-dollar version only 90%. However, the record pass rates already exceed those numbers because users are more inventive in how they apply existing models. This highlights how researcher ingenuity can outperform large-scale investment. Continued investment still has its place. Even shrinking jumps in performance from scaling may continue to justify the rising costs. It may also be that only relatively small improvements in performance "unlock" valuable or risky capabilities that justify policymaker intervention to either enable or avoid them. There are initial signs, though, that these diminishing marginal returns are already dampening the drive for ever-larger models. The growth in compute to train the largest models appears to be slowing. Among publicly available models, users and developers prefer to download intermediate-scale models even when larger and more powerful ones are freely available from the same providers. These trends raise questions about the types of models that will be most impactful and the relative importance of the compute, data, and algorithms that governments might hope to control. They also raise questions about the need for policymakers to intervene to promote or impede that progress and their opportunities to do so.