Clearing the Clutter: Real-Time Program-Specific Log Consolidation for APT Detection

Enterprises are increasingly threatened by Advanced Persistent Threats (APTs), carried out by skilled adversaries and remaining undetected for months. A promising approach to detect such intrusions is to parse logs into provenance graphs that capture data dependencies. However, a major challenge with this approach is that logs can rapidly grow to enormous sizes, imposing severe memory overhead. While existing research has introduced forensic-informed methods to reduce log size, these methods achieve modest reductions and may rely on offline processing, limiting their scalability for real-time analysis.In this work, we present Nano, a real-time log reduction approach that consolidates subject dependencies into program-specific provenance. Nano introduces two novel data structures, the profile hierarchy and the access network. Rather than preserving parent-child relationships between subjects, the profile hierarchy abstracts subject origins by capturing execution relationships between programs. For subjects created through an identical execution order of programs, the access network consolidates their activities into dependencies between executing programs and system entities, while retaining dependencies between attack activities. Our evaluation, using logs from government-agency sponsored red team exercises, demonstrates that Nano can effectively detect attacks comparable to existing rule-based intrusion detection systems, while reducing logs by up to 219 times at runtime.