Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection

Proactive Failure Detection (a) Reactive Failure Detection Task: Move the pan with the lobster to the stove, and be careful not to let the lobster drop out. (c) Real-world Test Pan Lobster Stove jump out Figure 1. For the task "Move the pan with lobster to the stove without losing the lobster", (a) reactive failure detection identifies failures after they occur, and (b) proactive failure detection prevents foreseeable failures. In (a), at t R 4 , the robot detects the failure after the lobster unpredictably jumps out due to the heat. In (b), pan tilting is detected at t P 3 and corrected it at t P ′ 3 , requiring real-time precision. We formulate both tasks as spatio-temporal constraint satisfaction problems, leveraging our proposed constraint elements for precise, real-time checking. For example, in (a), a large relative distance between pan and lobster indicates failure; in (b), a large angle between the pan and the horizontal plane needs correction. (c) shows that our method combined with an open-loop policy forms a closed-loop system, enabling proactive (e.g., detecting moving glass during grasping) and reactive (e.g., removing toy after grasping) failure detection in cluttered scenes.