Lune

CSCW2021Top-tier venue

Can Crowds Customize Instructional Materials with Minimal Expert Guidance?: Exploring Teacher-guided Crowdsourcing for Improving Hints in an AI-based Tutor

Kexin Bella Yang, Tomohiro Nagashima, Junhui Yao, Joseph Jay Williams, Kenneth Holstein, Vincent Aleven

Abstract

AI-based educational technologies may be most welcome in classrooms when they align with teachers' goals, preferences, and instructional practices. Teachers, however, have scarce time to make such customizations themselves. How might the crowd be leveraged to help time-strapped teachers? Crowdsourcing pipelines have traditionally focused on content generation. It is an open question how a pipeline might be designed so the crowd can succeed in a revision/customization task. In this paper, we explore an initial version of a teacher-guided crowdsourcing pipeline designed to improve the adaptive math hints of an AI-based tutoring system so they fit teachers' preferences, while requiring minimal expert guidance. In two experiments involving 144 math teachers and 481 crowdworkers, we found that such an expert-guided revision pipeline could save experts' time and produce better crowd-revised hints (in terms of teacher satisfaction) than two comparison conditions. The revised hints however, did not improve on the existing hints in the AI tutor, which were carefully-written but still have room for improvement and customization. Further analysis revealed that the main challenge for crowdworkers may lie in understanding teachers' brief written comments and implementing them in the form of effective edits, without introducing new problems. We also found that teachers preferred their own revisions over other sources of hints, and exhibited varying preferences for hints. Overall, the results confirm that there is a clear need for customizing hints to individual teachers' preferences. They also highlight the need for more elaborate scaffolds so the crowd can have specific knowledge of the requirements that teachers have for hints. The study represents a first exploration in the literature of how to support crowds with minimal expert guidance in revising and customizing instructional materials.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Interpretable Directed Diversity: Leveraging Model Explanations for Iterative Crowd IdeationYunlong Wang, Priyadarshini Venkatesh, Brian Y. LimCHI 2022 · 20 citations
  2. ML-Based Teaching Systems: A Conceptual FrameworkPhilipp Spitzer, Niklas Kühl, Daniel Heinz, Gerhard SatzgerCSCW 2023 · 6 citations
  3. Learning AI Auditing: A Case Study of Teenagers Auditing a Generative AI ModelLuis Morales-Navarro, Michelle A. Gan, Evelyn Yu, Lauren Vogelstein et al.CSCW 2025 · 6 citations

Related papers

  1. Involving Crowdworkers with Lived Experience in Content-Development for Push-Based Digital Mental Health Tools: Lessons Learned from Crowdsourcing Mental Health MessagesRachel Kornfield, David C. Mohr, Rachel Ranney, Emily G. Lattie et al.CSCW 2022 · 20 citations
  2. Designing AI Peers for Collaborative Mathematical Problem Solving with Middle School Students: A Participatory Design StudyWenhan Lyu, Yimeng Wang, Murong Yue, Yifan Sun et al.CHI 2026 · 4 citations
  3. CrowdMOT: Crowdsourcing Strategies for Tracking Multiple Objects in VideosSamreen Anjum, Chi Lin, Danna GurariCSCW 2020 · 5 citations
  4. Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective TasksHope Schroeder, Deb Roy, Jad KabbaraACL Findings 2025 · 16 citations
  5. AnnotateGPT: Designing Human-AI Collaboration in Pen-Based Document AnnotationBenedict Leung, Mariana Shimabukuro, Christopher CollinsCHI 2026 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines