LASER: An LLM-based ASR Scoring and Evaluation Rubric

Standard ASR evaluation metrics like Word Error Rate (WER) tend to unfairly penalize morphological and syntactic nuances that do not significantly alter sentence semantics. We introduce an LLM-based scoring rubric LASER that leverages state-of-the-art LLMs' in-context learning abilities to learn from prompts with detailed examples. Hindi LASER scores using Gemini 2.5 Pro achieved a very high correlation score of 94% with human annotations. Hindi examples in the prompt were also effective in analyzing errors in other Indian languages such as Marathi, Kannada and Malayalam. We also demonstrate how a smaller LLM like Llama 3 can be finetuned on word-pair examples derived from reference and ASR predictions to predict penalty types with close to 89% accuracy. Error type Example variations Penalty Numerical Phrases "1300" vs "Terah sau" or "Ek hajar teen sau" No penalty Abbreviations "ATM" vs "Ay Ti Em" vs "Ay tee yum" No penalty Compound Words "bhajan sangraha" vs "bhajansangraha" No penalty Transliterations (Native spellings) "ayskreem" vs "aaiskrim" or "skul" vs "skool" No penalty Actual transliterations "ice cream" vs "ayskrim" or "aaiskrim" No penalty Acceptable alternate spellings "sundar with a bindu" vs "sundar with a half na" No penalty Proper nouns "Priya" vs "Pria" vs "Preeya" vs "Preya" No penalty Slang and Colloquial terms "Yaha" vs "Ye" or "vaha" vs "vo" or "par" vs "pe" No penalty Small (single character) spelling errors "ladki" vs "ladkee" or "bahut" vs "bahoot" Minor penalty Small grammatical errors (gender/tense/number) "hain" vs "hai" or "uska" vs "uski" vs "usko" Minor penalty Spelling errors that alter meaning "kumar" vs "kamar" or "saman" vs "samanya" Major penalty Incorrect word substitutions "sundar" vs "bhadda" or "mota" vs "chhota" Major penalty Significant omissions or additions "-" vs "sundar" or "mota" vs "-" Major penalty Reordering of words that changes meaning "bahut accha khana" vs "bahut khana accha" Major penalty