DistillHGNN: A Knowledge Distillation Approach for High-Speed Hypergraph Neural Networks

This paper introduces a novel framework designed to significantly enhance the inference speed and memory efficiency of Hypergraph Neural Networks (HGNNs) while maintaining their high accuracy. Our approach, named DistillHGNN, employs an advanced teacher-student knowledge distillation strategy, where the teacher model comprises an HGNN and a Multi-Layer Perceptron (MLP). In this setup, the HGNN generates embeddings, which the MLP subsequently processes to predict soft labels. The student model consists of a lightweight Graph Convolutional Network (GCN), TinyGCN, paired with an MLP and optimised for online prediction. We leverage contrastive learning to train both TinyGCN and HGNN simultaneously, facilitating the transfer of high-order and structural knowledge from the HGNN to the TinyGCN. Additionally, the teacher employs a mechanism to transfer knowledge to the student model through soft labels. This dual transfer mechanism enables the student to effectively capture complex dependencies while benefiting from a lightweight GCN's faster inference and lower computational cost. The student is trained using both labelled data and soft labels provided by the teacher, with contrastive learning further ensuring that the student retains high-order relationships. This makes the proposed method efficient and suitable for real-time applications, achieving performance comparable to traditional HGNNs but with significantly reduced resource requirements. Experimental results on several real-world datasets demonstrate that our method significantly reduces inference time while maintaining accuracy comparable to HGNN, and it achieves higher accuracy than state-of-the-art techniques, like LightHGNN, with a similar inference time.