Generalized Policy Iteration using Tensor Approximation for Hybrid Control

Optimal Control of dynamic systems involving hybrid actions is a challenging task in robotics. To address this, we present a novel algorithm called Generalized Policy Iteration using Tensor Train (TTPI) that belongs to the class of Approximate Dynamic Programming (ADP). We use a low-rank tensor approximation technique called Tensor Train (TT) to approximate the state-value and advantage function which enables us to efficiently handle hybrid action space. We demonstrate the superiority of our approach over previous baselines for some benchmark problems with hybrid action spaces. Additionally, the robustness and generalization of the policy for hybrid systems are showcased through a real-world robotics experiment involving a non-prehensile manipulation task.

Robotic systems often exhibit complex nonlinear dynamics that may involve hybrid actions. The need for real-time control, high precision, and adequate robustness to cope with disturbances or changes in the environment can result in demanding computational requirements that are challenging to meet with classical control methods. Optimal Control (OC) based on the principles of Dynamic Programming (DP) is a popular tool in robotics but they are still limited to systems with continuous actions and differentiable dynamics.

Approximate DP (ADP) and Reinforcement Learning (RL) overcome the curse of dimensionality faced by classical DP algorithms by using function approximation techniques (Sutton & Barto, 2005;Bertsekas, 2012). OC is closely related to ADP and uses the system's model to obtain an optimal policy, while RL focuses on learning a policy through trial-and-error interactions with the environment. Both methods aim to find a compact representation of the value functions to obtain a control policy. ADP faces difficulty in approximating the value function throughout the entire state space, conversely, RL restricts its approximation to a smaller region where data is collected, resulting in limited generalizability but greater scalability. However, the existing approaches for both ADP and RL face challenges in handling hybrid action space. Furthermore, existing ADP approaches also find it challenging to cope with large action spaces and hybrid states.

In this paper, we present a novel ADP algorithm, called Generalized Policy Iteration using Tensor Train (TTPI) which overcomes the challenges faced by existing ADP methods for hybrid system control. TTPI is an approximate version of the Generalized Policy Iteration (GPI) algorithm-a DP algorithm that encompasses both Value Iteration (VI) and Policy Iteration (PI) algorithms. We use Tensor Train (TT) (Oseledets, 2011), a low-rank tensor approximation technique (Grasedyck et al., 2013), to model the state-value and the advantage function.

TT is a versatile function approximator that allows us to simultaneously handle continuous and discrete state and action variables. It approximates a given function as a sum of products of univariate functions, allowing for fast algebraic operations and interpretation. The use of TT-Cross (Oseledets & Tyrtyshnikov, 2010;Savostyanov & Oseledets, 2011), a powerful gradient-free method to approximate functions in TT format in a nonparametric manner, allows us to achieve TT approximation of statevalue and advantage function with a desired accuracy in a fast manner, thus exploiting the knowledge of the system model and the reward function. Moreover, the TT representation of the advantage function enables us to use optimization techniques such as TTGO (Shetty et al., 2023) to retrieve policies for hybrid action spaces.