Neural Wave Equation for Irregularly Sampled Sequence Data

Sequence labeling problems arise in several real-world applications such as healthcare and robotics. In many such applications, sequence data are irregularly sampled and are of varying complexities. Recently, efforts have been made to develop neural ODE-based architectures to model the evolution of hidden states continuously in time, to address irregularly sampled sequence data. However, they assume a fixed architectural depth and limit their flexibility to adapt to data sets with varying complexities. We propose the neural wave equation, a novel deep learning method inspired by the wave equation, to address this through continuous modeling of depth. Neural Wave Equation models the evolution of hidden states continuously across time as well as depth by using a non-homogeneous wave equation parameterized by a neural network. Through d'Alembert's analytical solution of the wave equation, we also show that the neural wave equation provides denser connections across the hidden states, allowing for better modeling capability. We conduct experiments on several sequence labeling problems involving irregularly sampled sequence data and demonstrate the superior performance of the proposed neural wave equation model.

1. We propose neural wave equations -a non-homogeneous wave equation with its source function parameterized by a neural network, for sequence labeling problems.

2. We speculate on the potential benefits of using non-homogenous partial differential equations for sequence modeling over continuous time RNN models with discrete depth by looking at their analytical solutions.

3. We empirically demonstrate the effectiveness of the neural wave equations on several sequence labeling tasks with irregularly sampled data.