Learning Disentangled Semantic Spaces of Explanations via Invertible Neural Networks

Most previous work on controlled text generation have concentrated on the style transfer task: modifying sentences with regard to markers of sentiment, formality, affirmation/negation. Disentanglement of generative factors over Variational Autoencoder (VAE) spaces has been a key mechanism for delivering this type of style transfer control. In this work, we focus on a more general form of controlled text generation, targeting the modification and control of more general semantic features. To achieve this, we introduce a flow-based invertible neural network (INN) mechanism plugged into the Optimus-based AutoEncoder architecture to deliver better properties of separability. Experimental results demonstrate that the model can conform the distributed latent space into a better semantically disentangled space, resulting in a more general form of language interpretability and control when compared to the recent state-of-the-art language VAE models (i.e., Optimus). Recently, Zhang et al. (2022) demonstrated that a more general form of semantic control can be achieved in the latent space of Optimus (Li et al., 2020b), the first standard transformer-based VAE, V-eats ARG1 livingthing ARG1-oxygen