SoK: Cryptographic Neural-Network Computation

We studied 53 privacy-preserving neural-network papers in 2016-2022 based on cryptography (without trusted processors or differential privacy), 16 of which only use homomorphic encryption, 19 use secure computation for inference, and 18 use non-colluding servers (among which 12 support training), solving a wide variety of research problems. We dissect their cryptographic techniques and "love-hate relationships" with machine learning alongside a genealogy highlighting noteworthy developments. We also re-evaluate the state of the art under WAN. We hope this can serve as a go-to guide connecting different experts in related fields.