Evaluating Non-AI Experts' Interaction with AI: A Case Study In Library Context

Public libraries in the U.S. are increasingly facing labor shortages, tight budgets, and overworked staff, creating a pressing need for conversational agents to assist patrons. The democratization of generative AI has empowered public service professionals to develop AI agents by leveraging large language models. To understand the needs of non-AI library professionals in creating their own conversational agents, we conducted semi-structured interviews with library professionals (n=11) across the U.S. Insights from these interviews informed the design of AgentBuilder, a prototype tool that enables non-AI experts to create conversational agents without coding skills. We then conducted think-aloud sessions and follow-up interviews to evaluate the prototype experience and identify the key evaluation criteria emphasized by library professionals (n=12) when developing conversational agents. Our findings highlight how these professionals perceive the prototype experience and reveal five essential evaluation criteria: interpreting user intent, faithful paraphrasing, proper alignment with authoritative sources, tailoring the tone of voice, and handling unknown answers effectively. These insights provide valuable guidance for designing AI-supported "end-user creation tools" in public service domains beyond libraries.