TIPSO-GAN: Malicious Network Traffic Detection Using a Novel Optimized Generative Adversarial Network

Detecting advanced cyber threats, particularly zero-day vulnerabilities, poses significant challenges in network security. This paper presents TIPSO-GAN, an optimized Generative Adversarial Network (GAN) for detecting malicious traffic. TIPSO-GAN addresses common GAN-based intrusion detection system (IDS) issues, such as training instability and mode collapse, by framing GAN training as a swarm optimization problem, harnessing collective intelligence for complex optimization. To enhance Particle Swarm Optimization (PSO), TIPSO- GAN employs three strategies: (1) adaptive inertia weights for a balance of exploration and exploitation, (2) a diversity preservation strategy to prevent premature convergence, and (3) a feedback loop to reinitialize stagnant particles. TIPSO-GAN integrates transfer learning with a Temporal-Decaying Multi- Head Self-Attention mechanism to prioritize recent features, aiding in unseen malicious traffic detection. A combination of reconstruction loss and focal loss in the objective function further ensures realistic normal samples while focusing on challenging malicious samples. Across CIC-IDS2018, CICAPT-IIoT2024, and CIC-DDoS2019, TIPSO-GAN achieves 99.1±0.1, 98.9±0.1, and 98.7±0.1 F1, outperforming the strongest baseline by 0.2–1.0 F1 and exceeding transformer IDS models. On CICAPT-IIoT2024, it reaches 0.999±0.002 macro PR-AUC, ahead of the next best method (0.960±0.005). Under strict zero-day evaluations, TIPSO-GAN attains 92.3 F1 in LOFO tests and 79–83 F1 in cross-dataset experiments while maintaining recall above 0.80. Despite PSO-enhanced training, TIPSO-GAN maintains 0.42 ms latency, ∼2400 flows/s throughput, and a 2.1 GB footprint, with stable performance up to $10^{8}$ flows. Our code is accessible at https://github.com/osampas27/tipsoganmod.