Beyond Technical Metrics: Understanding the Gap Between AI Performance and Deaf User Experience in Chinese Natural Sign Language Generation

Current sign language generation relies on technical metrics that often overlook capturing actual Deaf user comprehension. This is particularly challenging for Chinese Natural Sign Language (CNSL) with its scene-dependent expressions and spatial grammar. We assembled a scene-aware CNSL generation prototype using established components to serve as a controlled evaluation stimulus. In partnership with four Deaf co-researchers, we developed a seven-dimensional evaluation framework combining objective comprehension tests with validated subjective measures. An evaluation with 24 Deaf participants revealed that while technical metrics indicated success, objective comprehension tests showed lower comprehension and higher cognitive load, particularly for complex spatial grammar. These results demonstrate that current evaluation methods do not align with user needs. Our framework shifts from computer benchmarking toward human-centered assessment, prioritizing comprehension, cognitive load, and cultural authenticity. The findings underscore the importance of creating accessibility technology with, rather than for, Deaf communities.