PESTO: Switching Point Based Dynamic and Relative Positional Encoding for Code-Mixed Languages (Student Abstract)

NLP applications for code-mixed (CM) or mix-lingual text have gained a significant momentum recently, the main reason being the prevalence of language mixing in social media communications in multi-lingual societies like India, Mexico, Europe, parts of USA etc. Word embeddings are basic building blocks of any NLP system today, yet, word embedding for CM languages is an unexplored territory. The major bottleneck for CM word embeddings is switching points, where the language switches. These locations lack in contextually and statistical systems fail to model this phenomena due to high variance in the seen examples. In this paper we present our initial observations on applying switching point based positional encoding techniques for CM language, specifically Hinglish (Hindi -English). Results are only marginally better than SOTA, but it is evident that positional encoding could be an effective way to train position sensitive language models for CM text. Switching Points: The Bottleneck Switching Points (SPs) are the positions in CM text, where the language switches. Consider the text -aap HI se HI request EN hain HI (request you to). Here, when the language switches from Hindi to English (se HI request EN ) a HI-EN (HIndi-ENglish) SP occurs. Correspondingly, a EN-HI SP occurs at request EN hain HI . In this work we to look at sentiment analysis of CM languages, specifically Hinglish through the lens of language modeling. We propose PESTO -a switching point based dynamic and relative positional encoding. PESTO learns to emphasis on switching points in CM text. Our model marginally outperforms the SOTA.