Prioritized Reinforcement Learning for Analog Circuit Optimization With Design Knowledge

Analog circuit design and optimization manifests as a critical phase in IC design, which still heavily relies on extensive and time-consuming manual designing by experienced experts. In recent years, the development of reinforcement learning (RL) algorithms draws attention with related techniques being introduced into the analog design field for circuit optimization. However, for robust and efficient analog circuit design, a smart and rapid search for high-quality design points is more desired than finding a globally optimal agent as in traditional RL applications, which was a point not fully considered in some previous works. In this work, we propose three techniques within the RL framework aiming at fast high-quality design point search in a data efficient manner. In particular, we (i) incorporate design knowledge from experienced designers into the critic network design to achieve a better reward evaluation with less data; (ii) guide the RL training with non-uniform sampling techniques prioritizing exploitation over high quality designs and exploration for poorly-trained space; (iii) leverage the trained critic network and limited additional circuit simulation for smart and efficient sampling to get high-quality design points. The experimental results demonstrate the effectiveness and efficiency of our proposed techniques.