LOHEN: Layer-wise Optimizations for Neural Network Inferences over Encrypted Data with High Performance or Accuracy

Fully Homomorphic Encryption (FHE) presents unique challenges in programming due to the contrast between traditional and FHE language paradigms. A key challenge is selecting optimal ciphertext configurations (CCs) to ensure security, performance, and accuracy in FHE applications. Optimal CC selection can be complex and labor-intensive. Thus, the conventional practice is settling down on a suboptimal solution of globally choosing one CC that induces acceptable performance. When FHE is applied to neural networks (NNs), the distinct layered architecture of NN models opens the door for a layer-wise optimization approach, as an alternative to the conventional practice, which selects locally optimal CCs for individual layers tailored to their structural properties. Sadly, this approach incurs significant overhead from CC switching between layers during NN inference. This paper introduces LOHEN, a technique crafted to attain high performance of NN inference by optimizing layer-wise CC selection to minimize the increased overhead incurred by CC switching. Not only that, LOHEN is engineered to attain high accuracy of NN inference, yet delivering performance comparable to stateof-the-art studies, by adopting a layer-wise application of multiple FHE schemes to the target NN. Through two experimental settings, we exhibit LOHEN's capability that allows developers to customize layer-wise optimizations to adjust the desired levels of performance and accuracy of NN inference over encrypted data, subject to their demands. In the first experiment where we adopt CKKS as our FHE scheme, LO-HEN improves inference performance of various NN models by 1.08-2.88×. In the second where we employ two schemes (CKKS+TFHE), LOHEN ensures zero accuracy loss in inference, yet improving performance by 1.34-1.59× compared to an optimized multi-scheme baseline.