Optimal and Approximate Parallelism-Based Computation Offloading Algorithms for Real-Time Multimodal Learning at the Edge

Multimodal learning has been introduced as a popular learning paradigm that can integrate inputs from multimodal video data. To accelerate video analytics at the edge, video frames are usually scalarized and compressed into various resolutions to offload to the edge server to achieve a balance between accuracy and latency. In this paper, we investigate the problem of the Joint Schedule of Offloading decision and Resolution selection (JSOR) for real-time multimodal learning at the edge. Firstly, the parallelism between the computation and communication between the edge device and server is identified and modeled. Then, the problem of JSOR to maximize the accuracy while minimizing energy consumption under the latency constraints, is formulated and proved to be NP-hard. To the best of our knowledge, this is the first work that takes the parallelism during the offloading process into account for the JSOR problem. An optimal algorithm based on dynamic programming is proposed with a decision graph, which is constructed to integrate the offloading decision and resolution selection together with the processing latency. To further reduce the time complexity, several pruning strategies and an approximate algorithm are also proposed. Additionally, to maximize the long-term average utility, an adaptive online algorithm based on Lyapunov optimization and reinforcement learning is also proposed. Finally, through extensive simulations and real implementations on the NVIDIA Jetson AGX Orin platform, we demonstrated the effectiveness of the proposed algorithms in terms of accuracy and energy consumption.