Pareto-Based Heterogeneous Knowledge Distillation for MLPs on Graphs

Heterogeneous Graph Neural Networks (HGNNs) have demonstrated remarkable capabilities in capturing effective information in heterogeneous graphs, achieving outstanding performance in various learning tasks. However, their heavy dependency on neighbor information may result in high latency, which restricts their practicality in real-world applications. Recent studies have attempted to overcome such latency in Graph Neural Networks (GNNs) by distilling knowledge into student models that do not rely on graph structure. But these approaches primarily focus on replicating teachers' predictive outcomes while neglecting the structural knowledge they encoded. This limitation makes such approaches less effective when graphs become complex, particularly in heterogeneous graphs. Motivated by this challenge, we propose HGKD, a novel hierarchical knowledge distillation framework that transfers both structural knowledge and predictive outcomes from HGNN teachers to a multi-layer perceptron (MLP) student. Additionally, we provide two variants of HGKD that help the student learn from multiple teacher models via Pareto learning, and incorporate low-cost neighbor information. We evaluate HGKD and its variants on a range of heterogeneous graph datasets. The results demonstrate that the student model achieves performance comparable to, or even exceeding, that of HGNN teachers, despite not relying on graph structures during inference.