Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases

Who is the producer of Spamalot? (AND Theater_Producer (JOIN (R producer) Spamalot)) -How many plays has Bob Boyett produced? (COUNT (AND Theater_Production (JOIN producer Bob_Boyett)) -Find plays that were staged in large theaters that could hold at least 20,000 people. (AND Theater_Production (JOIN (R staged_here) (JOIN (GE capacity 20000))) -How many theater productions has Oprah produced? (COUNT (AND Theater_Production (JOIN producer Oprah_Winfrey)) -Bob Boyett's production was housed in what theater capable of holding at least 10,000 people? (AND Theater (AND (GE capacity 10000) (JOIN staged_here (JOIN producer Bob_Boyett)))) -How many TV programs has Bob Boyett created? (COUNT (AND TV_Program (JOIN (R program_created) Bob_Boyett)) Knowledge Base KBQA Model Training Data I.I.D. Generalization Compositional Generalization Zero-Shot Generalization Figure 1: On large-scale KBs, collecting sufficient training data for KBQA to ensure i.i.d. distribution at test time is very difficult, if possible at all. We argue that practical KBQA models should have three levels of built-in generalization rather than solely relying on training data: (1) i.i.d. generalization to questions following the training distribution, (2) compositional generalization to novel compositions of schema items seen in training (marked blue), and (3) zero-shot generalization to unseen schema items or even domains (marked red). Our definition of generalization is based on the underlying logical forms (shown as S-expressions). Orthogonally, as illustrated by the examples, KBQA models should also have strong generalization to linguistic variation. Figure best viewed in color.