Kolmogorov-Arnold Transformer

Figure 1: (Left) Architecture of standard transformer (e.g. ViT), ViT+KAN which substitutes the MLP with a KAN, and our KAT model. In KAT, the MLP layers in transformers are replaced with GR-KAN layers. (Right) Performance on the ImageNet dataset. KAT * indicates that the model was initialized using a pre-trained ViT. Generally, KAT outperforms both the ViT and DeiT models. ViT+KAN performs poorly on ImageNet-level training.