Lune

ACL2025Top-tier venue

Powerformer: Efficient and High-Accuracy Privacy-Preserving Language Model with Homomorphic Encryption

Dongjin Park, Eunsang Lee, Joon-Woo Lee

Abstract

We propose Powerformer , an efficient homomorphic encryption (HE)-based privacy-preserving language model (PPLM) designed to reduce computational overhead while maintaining model performance. Powerformer incorporates three key techniques to optimize encrypted computations: 1) A novel distillation technique that replaces softmax and layer normalization with computationally efficient power and linear functions, ensuring no performance degradation while enabling seamless encrypted computation. 2) A pseudo-sign composite approximation method that accurately approximates GELU and tanh functions with minimal computational overhead. 3) A homomorphic matrix multiplication algorithm specifically optimized for Transformer models, enhancing efficiency in encrypted environments. By integrating these techniques, Powerformer based on the BERT-base model achieves a 45% reduction in computation time compared to the state-of-the-art HE-based PPLM without any loss in accuracy.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • On the (In-)Security of the Shuffling Defense in the Transformer Secure Inference cites BOLT: Privacy-Preserving, Accurate and Efficient Inference for Transformers
  • MOAI: Module-Optimizing Architecture for Non-Interactive Secure Transformer Inference cites THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
  • MOAI: Module-Optimizing Architecture for Non-Interactive Secure Transformer Inference cites BOLT: Privacy-Preserving, Accurate and Efficient Inference for Transformers
  • Hyperion: Private Token Sampling with Homomorphic Encryption cites MOAI: Module-Optimizing Architecture for Non-Interactive Secure Transformer Inference
  • BOLT: Privacy-Preserving, Accurate and Efficient Inference for Transformers cites THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
  • BOLT: Privacy-Preserving, Accurate and Efficient Inference for Transformers cites Secure Outsourced Matrix Computation and Application to Neural Networks
  • ROSETTA: Efficient and Accurate Privacy-Preserving LLM Decoding via Hybrid CKKS/TFHE Evaluation cites THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
  • ROSETTA: Efficient and Accurate Privacy-Preserving LLM Decoding via Hybrid CKKS/TFHE Evaluation cites MOAI: Module-Optimizing Architecture for Non-Interactive Secure Transformer Inference
  • ROSETTA: Efficient and Accurate Privacy-Preserving LLM Decoding via Hybrid CKKS/TFHE Evaluation cites BOLT: Privacy-Preserving, Accurate and Efficient Inference for Transformers
  • Sok: Private Transformer-based Model Inference cites THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
  • Sok: Private Transformer-based Model Inference cites MOAI: Module-Optimizing Architecture for Non-Interactive Secure Transformer Inference
  • Sok: Private Transformer-based Model Inference cites BOLT: Privacy-Preserving, Accurate and Efficient Inference for Transformers
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. MOAI: Module-Optimizing Architecture for Non-Interactive Secure Transformer InferenceLinru Zhang, Xiangning Wang, Sim Jun Jie, Zhicong Huang et al.ICLR 2026 · 24 citations
  2. Hyperion: Private Token Sampling with Homomorphic EncryptionLawrence Lim, Jiaming Liu, Vikas Kalagi, Divyakant Agrawal et al.ACL 2026 · 1 citation
  3. On the (In-)Security of the Shuffling Defense in the Transformer Secure InferenceZhengyi Li, Yakai Wang, Jingwen Leng, Kang Yang et al.ACL 2026
  4. ROSETTA: Efficient and Accurate Privacy-Preserving LLM Decoding via Hybrid CKKS/TFHE EvaluationJiangrui Yu, Baosheng Zhang, Liang Kong, Lin Ding et al.CCS 2026
  5. Sok: Private Transformer-based Model InferenceYuntian Chen, Tianpei Lu, Zhanyong Tang, Bingsheng Zhang et al.USENIX Security 2026

Builds on10

  1. TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
  2. Secure Outsourced Matrix Computation and Application to Neural NetworksXiaoqian Jiang, Miran Kim, Kristin E. Lauter, Yongsoo SongCCS 2018 · 359 citations
  3. BOLT: Privacy-Preserving, Accurate and Efficient Inference for TransformersQi Pang, Jinhao Zhu, Helen Möllering, Wenting Zheng et al.S&P 2024 · 149 citations
  4. High-Precision Bootstrapping of RNS-CKKS Homomorphic Encryption Using Optimal Minimax Polynomial Approximation and Inverse Sine FunctionJoon-Woo Lee, Eunsang Lee, Yongwoo Lee, Young-Sik Kim et al.EUROCRYPT 2021 · 110 citations
  5. THE-X: Privacy-Preserving Transformer Inference with Homomorphic EncryptionTianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong et al.ACL Findings 2022 · 86 citations

Related papers

  1. PCFormer: Accelerating Privacy-preserving Transformer Inference by Partition and CombinationBo Zeng, Zhi Pang, Yuyang Zhang, Kai Zhao et al.AAAI 2026
  2. THOR: Secure Transformer Inference with Homomorphic EncryptionJungho Moon, Dongwoo Yoo, Xiaoqian Jiang, Miran KimCCS 2025 · 1 citation
  3. Breaking the Layer Barrier: Remodeling Private Transformer Inference with Hybrid CKKS and MPCTianshi Xu, Wen-jie Lu, Jiangrui Yu, Yi Chen et al.USENIX Security 2025
  4. Encryption-Friendly LLM ArchitectureDonghwan Rho, Taeseong Kim, Minje Park, Jung Woo Kim et al.ICLR 2025
  5. Iron: Private Inference on TransformersMeng Hao, Hongwei Li, Hanxiao Chen, Pengzhi Xing et al.NeurIPS 2022 · 209 citations
Dusk over the sea between two cliffs drawn in fine vertical lines