RTrace: Towards Better Visibility of Shared Library Execution

Software supply chain security has become a critical concern in recent years. Modern software systems increasingly depend on third-party dependencies to accelerate development. Shared libraries are the prevalent form of software sharing and hence, of third-party dependencies in modern software systems. As more attacks target the software supply chain, understanding the behavior of these dependencies is essential for identifying vulnerabilities and malicious code. Hence, accurately tracing function calls within shared libraries is critical for effective software security analysis. However, existing library function tracers often fail to meet this need. As we show in this work, state-of-the-art library function tracers are limited in effectiveness and scalability, missing a significant number of function calls and failing with more complex workloads, resulting in incomplete or misleading views of runtime behavior. In this paper, we present RTrace, a tracing tool designed to address the limitations of existing solutions. We analyze the root causes of why widely used tracers miss function calls and identify common pitfalls such as relying on incorrect symbol information and inability to monitor early or indirect function invocations. RTrace overcomes these challenges by incorporating comprehensive runtime monitoring, function boundary detection, and support for implicit and unconventional function calls. We compare RTrace to four state-of-the-art tracers, namely, emphltrace, emphdrltrace, emphldaudit, and emphIntelPT. Our evaluation across 21 applications and 92 shared libraries shows that RTrace significantly outperforms existing tools in detecting function call. RTrace achieves an F1-score of at least 0.92 on all benchmarks, whereas the best existing tracer reaches only 0.74, providing more accurate visibility into shared library runtime behavior. Finally, we show how RTrace can be used to assist in detecting malicious package and in vulnerability analysis by providing a more complete view of shared library function usage.