Predicting Performance Improvement of Human Activity Recognition Model by Additional Data Collection

The development of a machine-learning-based human activity recognition (HAR) system using body-worn sensors is mainly composed of three phases: data collection, model training, and evaluation. During data collection, the HAR developer collects labeled data from participants wearing inertial sensors. In the model training phase, the developer trains the HAR model on the collected training data. In the evaluation phase, the developer evaluates the trained HAR model on the collected test data. When the HAR model cannot achieve the target recognition accuracy, the developer iterates the above procedures by taking certain measures, including collecting additional training data, until the re-trained model achieves the target accuracy. However, collecting labeled data for HAR requires additional time and incurs high monetary costs. In addition, it is difficult to determine the amount and type of data to collect for achieving the target accuracy while reducing costs. To address this issue, this paper proposes a new method that predicts the performance improvement of the current HAR model, i.e., it determines the level of performance improvement achievable by re-training the HAR model with additional data, before collecting the additional data. Thus, the method enables the HAR developer to establish a strategy for additional data collection by providing advice such as "If labeled data for the Walking and Running activities from two additional participants is collected, the HAR accuracy of the current HAR model for Walking will improve by 20%." To achieve this, a neural network called AIP-Net is proposed to estimate the improvement in performance by analyzing the feature space of the current HAR model using the proposed entropy-based attention mechanism. The performance of AIP-Net was evaluated on eight HAR datasets using leave-one-dataset-out cross-validation.