A Miss Is as Good as A Mile: Metamorphic Testing for Deep Learning Operators

Deep learning (DL) is a critical tool for real-world applications, and comprehensive testing of DL models is vital to ensure their quality before deployment. However, recent studies have shown that even subtle deviations in DL operators can result in catastrophic consequences, underscoring the importance of rigorous testing of these components. Unlike testing other DL system components, operator analysis poses unique challenges due to complex inputs and uncertain outputs. The existing DL operator testing approach has limitations in terms of testing efficiency and error localization. In this paper, we propose Meta , a novel operator testing framework based on metamorphic testing that automatically tests and assists bug location based on metamorphic relations (MRs). Meta distinguishes itself in three key ways: (1) it considers both parameters and input tensors to detect operator errors, enabling it to identify both implementation and precision errors; (2) it uses MRs to guide the generation of more effective inputs (i.e., tensors and parameters) in less time; (3) it assists the precision error localization by tracing the error to the input level of the operator based on MR violations. We designed 18 MRs for testing 10 widely used DL operators. To assess the effectiveness of Meta, we conducted experiments on 13 released versions of 5 popular DL libraries. Our results revealed that Meta successfully detected 41 errors, including 14 new ones that were reported to the respective platforms and 8 of them are confirmed/fixed. Additionally, Meta demonstrated high efficiency, outperforming the baseline by detecting <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"> mml:mrow mml:mo∼</mml:mo> mml:mn2</mml:mn> </mml:mrow> </mml:math> times more errors of the baseline. Meta is open-sourced and available at https://github.com/TDY-raedae/Medi-Test .