TinyTTA: Efficient Test-time Adaptation via Early-exit Ensembles on Edge Devices

The increased adoption of Internet of Things (IoT) devices has led to the generation of large data streams with applications in healthcare, sustainability, and robotics. In some cases, deep neural networks have been deployed directly on these resource-constrained units to limit communication overhead, increase efficiency and privacy, and enable real-time applications. However, a common challenge in this setting is the continuous adaptation of models necessary to accommodate changing environments, i.e., data distribution shifts. Test-time adaptation (TTA) has emerged as one potential solution, but its validity has yet to be explored in resource-constrained hardware settings, such as those involving microcontroller units (MCUs). TTA on constrained devices generally suffers from i) memory overhead due to the full backpropagation of a large pre-trained network, ii) lack of support for normalization layers on MCUs, and iii) either memory exhaustion with large batch sizes required for updating or poor performance with small batch sizes. In this paper, we propose TinyTTA, to enable, for the first time, efficient TTA on constrained devices with limited memory. To address the limited memory constraints, we introduce a novel self-ensemble and batch-agnostic early-exit strategy for TTA, which enables continuous adaptation with small batch sizes for reduced memory usage, handles distribution shifts, and improves latency efficiency. Moreover, we develop the TinyTTA Engine, a first-of-its-kind MCU library that enables on-device TTA. We validate TinyTTA on a Raspberry Pi Zero 2W and an STM32H747 MCU. Experimental results demonstrate that TinyTTA improves TTA accuracy by up to 57.6%, reduces memory usage by up to six times, and achieves faster and more energy-efficient TTA. Notably, TinyTTA is the only framework able to run TTA on MCU STM32H747 with a 512 KB memory constraint while maintaining high performance.

Deploying deep neural networks on IoT devices, such as microcontroller units (MCUs), holds significant potential in many applications requiring real-time data analysis and low-latency responses, such as real-time human health monitoring [1] and robotics [2]. However, a practical challenge in deploying these models on MCUs in real-world scenarios is their adaptation capability. The data encountered in real settings often exhibit distribution shifts due to unforeseen noise and environment. For instance, sensor data can be affected by natural fluctuations and distortions, including weather changes (e.g., fog and snow) and sensor-related issues (e.g., defocus blur) [3,4]. Existing mitigation strategies, such as fine-tuning [5], typically necessitate updating the entire model, which may be 38th Conference on Neural Information Processing Systems (NeurIPS 2024).

impractical for edge devices due to constraints on memory, energy, and computational resources. Consequently, developing effective adaptation techniques for edge devices to address such distribution shifts while maintaining efficiency is a critical and pressing need.

Test-time adaptation (TTA) emerges as an effective strategy to mitigate generalization issues [6]. By adapting a pre-trained model to unlabeled test samples, TTA aims to align the model with the current data distribution through entropy minimization. This technique reduces the model's prediction entropy without requiring extensive retraining [6]. However, conventional TTA approaches face several limitations that prevent them from being applicable to edge devices. Firstly, TTA typically requires updates to either the entire network or several layers. Even when updating only a few layers, it still incurs substantial memory overhead due to the need for backpropagation. This process requires storing activations for a large pre-trained model, therefore imposing substantial memory requirements. Secondly, few works have considered mixed distribution shifts and have uniformly updated networks for samples exhibiting varying distribution shifts. This approach not only results in varying accuracy for different levels of distribution shifts but also increases computational overhead, as samples with minor shifts may not, in theory, necessitate updating the entire network. Thirdly, the most effective TTA strategies often involve adapting the normalization layers [3,7,8]. However, in practice, these layers are usually fused with convolutional layers when developed for MCUs to reduce computational and memory demands [9][10][11]. This poses practical challenges as currently there is no available library support for normalization layers in such hardware environments. Lastly, the effectiveness of normalization-based TTA typically relies on large batch sizes, such as 64 instances per update [3], which significantly increases computational overhead. Instead, constrained hardware such as MCUs typically only allow a single batch of data due to limited memory resources. While smaller batch sizes could be more practical fo