Powerformer: Efficient and High-Accuracy Privacy-Preserving Language Model with Homomorphic Encryption
Dongjin Park, Eunsang Lee, Joon-Woo Lee
Abstract
We propose Powerformer , an efficient homomorphic encryption (HE)-based privacy-preserving language model (PPLM) designed to reduce computational overhead while maintaining model performance. Powerformer incorporates three key techniques to optimize encrypted computations: 1) A novel distillation technique that replaces softmax and layer normalization with computationally efficient power and linear functions, ensuring no performance degradation while enabling seamless encrypted computation. 2) A pseudo-sign composite approximation method that accurately approximates GELU and tanh functions with minimal computational overhead. 3) A homomorphic matrix multiplication algorithm specifically optimized for Transformer models, enhancing efficiency in encrypted environments. By integrating these techniques, Powerformer based on the BERT-base model achieves a 45% reduction in computation time compared to the state-of-the-art HE-based PPLM without any loss in accuracy.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- On the (In-)Security of the Shuffling Defense in the Transformer Secure Inference cites BOLT: Privacy-Preserving, Accurate and Efficient Inference for Transformers
- MOAI: Module-Optimizing Architecture for Non-Interactive Secure Transformer Inference cites THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
- MOAI: Module-Optimizing Architecture for Non-Interactive Secure Transformer Inference cites BOLT: Privacy-Preserving, Accurate and Efficient Inference for Transformers
- Hyperion: Private Token Sampling with Homomorphic Encryption cites MOAI: Module-Optimizing Architecture for Non-Interactive Secure Transformer Inference
- BOLT: Privacy-Preserving, Accurate and Efficient Inference for Transformers cites THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
- BOLT: Privacy-Preserving, Accurate and Efficient Inference for Transformers cites Secure Outsourced Matrix Computation and Application to Neural Networks
- ROSETTA: Efficient and Accurate Privacy-Preserving LLM Decoding via Hybrid CKKS/TFHE Evaluation cites THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
- ROSETTA: Efficient and Accurate Privacy-Preserving LLM Decoding via Hybrid CKKS/TFHE Evaluation cites MOAI: Module-Optimizing Architecture for Non-Interactive Secure Transformer Inference
- ROSETTA: Efficient and Accurate Privacy-Preserving LLM Decoding via Hybrid CKKS/TFHE Evaluation cites BOLT: Privacy-Preserving, Accurate and Efficient Inference for Transformers
- Sok: Private Transformer-based Model Inference cites THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
- Sok: Private Transformer-based Model Inference cites MOAI: Module-Optimizing Architecture for Non-Interactive Secure Transformer Inference
- Sok: Private Transformer-based Model Inference cites BOLT: Privacy-Preserving, Accurate and Efficient Inference for Transformers
Cited by top-tier papers5
- MOAI: Module-Optimizing Architecture for Non-Interactive Secure Transformer InferenceLinru Zhang, Xiangning Wang, Sim Jun Jie, Zhicong Huang et al.ICLR 2026 · 24 citations
- Hyperion: Private Token Sampling with Homomorphic EncryptionLawrence Lim, Jiaming Liu, Vikas Kalagi, Divyakant Agrawal et al.ACL 2026 · 1 citation
- On the (In-)Security of the Shuffling Defense in the Transformer Secure InferenceZhengyi Li, Yakai Wang, Jingwen Leng, Kang Yang et al.ACL 2026
- ROSETTA: Efficient and Accurate Privacy-Preserving LLM Decoding via Hybrid CKKS/TFHE EvaluationJiangrui Yu, Baosheng Zhang, Liang Kong, Lin Ding et al.CCS 2026
- Sok: Private Transformer-based Model InferenceYuntian Chen, Tianpei Lu, Zhanyong Tang, Bingsheng Zhang et al.USENIX Security 2026
Builds on10
- TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
- Secure Outsourced Matrix Computation and Application to Neural NetworksXiaoqian Jiang, Miran Kim, Kristin E. Lauter, Yongsoo SongCCS 2018 · 359 citations
- BOLT: Privacy-Preserving, Accurate and Efficient Inference for TransformersQi Pang, Jinhao Zhu, Helen Möllering, Wenting Zheng et al.S&P 2024 · 149 citations
- High-Precision Bootstrapping of RNS-CKKS Homomorphic Encryption Using Optimal Minimax Polynomial Approximation and Inverse Sine FunctionJoon-Woo Lee, Eunsang Lee, Yongwoo Lee, Young-Sik Kim et al.EUROCRYPT 2021 · 110 citations
- THE-X: Privacy-Preserving Transformer Inference with Homomorphic EncryptionTianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong et al.ACL Findings 2022 · 86 citations
Related papers
- PCFormer: Accelerating Privacy-preserving Transformer Inference by Partition and CombinationBo Zeng, Zhi Pang, Yuyang Zhang, Kai Zhao et al.AAAI 2026
- THOR: Secure Transformer Inference with Homomorphic EncryptionJungho Moon, Dongwoo Yoo, Xiaoqian Jiang, Miran KimCCS 2025 · 1 citation
- Breaking the Layer Barrier: Remodeling Private Transformer Inference with Hybrid CKKS and MPCTianshi Xu, Wen-jie Lu, Jiangrui Yu, Yi Chen et al.USENIX Security 2025
- Encryption-Friendly LLM ArchitectureDonghwan Rho, Taeseong Kim, Minje Park, Jung Woo Kim et al.ICLR 2025
- Iron: Private Inference on TransformersMeng Hao, Hongwei Li, Hanxiao Chen, Pengzhi Xing et al.NeurIPS 2022 · 209 citations
