Exploring the Efficacy of Multi-Agent Reinforcement Learning for Autonomous Cyber Defence: A CAGE Challenge 4 Perspective

Recent advances in multi-agent reinforcement learning (MARL) have created opportunities to solve complex real-world tasks. Cybersecurity is a notable application area, where defending networks against sophisticated adversaries remains a challenging task typically performed by teams of security operators. In this work, we explore novel MARL strategies for building autonomous cyber network defenses that address challenges such as large policy spaces, partial observability, and stealthy, deceptive adversarial strategies. To facilitate efficient and generalized learning, we propose a hierarchical Proximal Policy Optimization (PPO) architecture that decomposes the cyber defense task into specific sub-tasks like network investigation and host recovery. Our approach involves training sub-policies for each sub-task using PPO enhanced with domain expertise. These sub-policies are then leveraged by a master defense policy that coordinates their selection to solve complex network defense tasks. Furthermore, the sub-policies can be fine-tuned and transferred with minimal cost to defend against shifts in adversarial behavior or changes in network settings. We conduct extensive experiments using CybORG Cage 4, the state-of-the-art MARL environment for cyber defense. Comparisons with multiple baselines across different adversaries show that our hierarchical learning approach achieves top performance in terms of convergence speed, episodic return, and several interpretable metrics relevant to cybersecurity, including the fraction of clean machines on the network, precision, and false positives. Contribution(s) 1. A scalable hierarchical multi-agent reinforcement learning method for cyber defense that decomposes the complex cyber defense task into multiple sub-tasks. Context: Prior work uses hierarchical MARL in other domains such as multi-robot learning, while current RL-based methods in the cyber defense domain are single agent. 2. A design guided by cybersecurity domain expertise to enhance the RL agents' observation space and facilitate learning of better policies. Context: Prior work on RL cyber defense uses the observation space provided by a cyber environment such as CybORG, without expanding it. 3. Defensive strategies that transfer either directly or via fine-tuning against a range of deceptive, stealthy adversaries in the CybORG CAGE 4 cyber environment. Context: We show that the proposed H-MARL methods generalize to three types of stealthy adversarial agents, besides the default red agent in CybORG CAGE 4, and we also demonstrate transferability to new red agents after fine-tuning. 4. Definition and analysis of multiple interpretable metrics for providing insights to security operators on the developed defenses. Context: Prior work in RL for cyber defense mainly analyzes the cumulative return, but does not discuss interpretable metrics, which are very relevant to security operators.