A Rigorous Behavior Assessment of CNNs Using a Data-Domain Sampling Regime

Training data sampling methods Quantitative Data sampling domain (A toy example with 20 samples) Neural networks Figure 1: A toy example of a training data sampling regime. We quantify behaviors of convolutional neural networks (CNNs) in response to training inputs. Four data-domain sampling methods are used to supervise CNNs before sending them to take the same test.

Abstract-We present a data-domain sampling regime for quantifying CNNs' graphic perception behaviors. This regime lets us evaluate CNNs' ratio estimation ability in bar charts from three perspectives: sensitivity to training-test distribution discrepancies, stability to limited samples, and relative expertise to human observers. After analyzing 16 million trials from 800 CNN models and 6,825 trials from 113 human participants, we arrived at a simple and actionable conclusion: CNNs can outperform humans and their biases simply depend on the training-test distance. We show evidence of this simple, elegant behavior of the machines when they interpret visualization images. osf.io/gfqc3 provides registration, the code for our sampling regime, and experimental results.