Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights

Large model Small model Gaussian search window tasks GSM8k expert (math) ROCStories expert (writing) MBPP expert (Programming) USPTO expert (chemistry) (a) (b) (c) Post-training with RandOpt O(1) training and FLOP-efficient with Better Acc Scaling Law Figure 1: (a) Schematic of the main effects we observe (see Fig 2 for a version with real data). Left: Small models live in a needle in a haystack regime, where good solutions to downstream tasks occupy a tiny fraction of the surrounding weights. In this regime, it is important to have a smart search algorithm, such as gradient descent or other forms of iterative optimization. Right: Large models are surrounded by a veritable thicket of task-specific solutions. In this regime, random sampling is sufficient to quickly land on promising adaptations, which can then be ensembled to yield strong behavior, an approach we call RandOpt. (b) Solution density -i.e. density of task-improving weights in a Gaussian neighborhood of the pretrained weights -scales with model size. (c) RandOpt is O(1) in training steps, FLOP-efficient, and competitive in converged accuracy with GRPO and ES. Results are shown on the Countdown task with Olmo-3-7B-Instruct; RandOpt uses 5000 random weight guesses and ensembles the top K; K-pass baselines use Test-time Majority Vote (TT-MV). More results are shown in Fig. 6 and Table 4 .