Escaping Whack-a-Mole: Optimizing Documentation as Repo-Specific Playbooks for Coding Agents

As large language models increasingly function as autonomous coding agents, code documentation should be designed not for human readability, but for agent executability — serving as repo-specific playbooks that specify precise behaviors agents can follow. We formulate agent-oriented documentation generation as a black-box optimization problem over the documentation space, where quality is defined solely by downstream code correctness. A central challenge for conventional LLM refinement methods is output coupling—program entities are interdependent, and refining the documentation of one entity can invalidate its callers, resulting in a persistent whack-a-mole phenomenon during inference-time scaling. We propose DocSearch, a dependency-guided bi-level search framework that systematically exploits test-time feedback. The outer level conducts a priority search over the program-entity dependency DAG, enforcing a callee-before-caller refinement order to prevent downstream interference. The inner level performs a beam search over documentation refinements, using diversified error message sampling from self-generated unit tests to better exploit diagnostic signals and escape local optima. On DevEval+, DocSearch achieves 90.7% solve rate with GPT-4o, outperforming the strongest baseline by 32.6%. Cross-language experiments further demonstrate that optimized documentation transfers effectively to different target programming languages. Code is available at https://github.com/ccsnow127/docsearch.