A Theoretical Approach to Determine the Optimal Size of a Thread Pool for Real-Time Systems

Parallel workloads most commonly execute onto pools of thread, allowing to dispatch and run individual nodes (e.g., implemented as C++ functions) at the user-space level. This is relevant in industrial cyber-physical systems, cloud, and edge computing, especially in systems leveraging deep neural networks (e.g., TensorFlow), where the computations are inherently parallel. When using thread pools, it is common to implement fork-join parallelism using blocking synchronization mechanisms provided by the operating system (such as condition variables), with the side effect of temporarily reducing the number of worker threads. Consequently, the served tasks may suffer from additional delays, thus potentially harming timing guarantees if such effects are not properly considered. Prior works studied such phenomena, providing methods to guarantee the timing behavior. However, the challenges introduced by thread pools with blocking synchronization cause current analyses to incur a notable pessimism. This paper tackles the problem from a different angle, proposing solutions to determine the optimal size of a thread pool in such a way as to avoid the undesired effects that arise from blocking synchronization.