LLM Agents for Coordinating Multi-User Information Gathering

This paper introduces PEOPLEJOIN, a benchmark for evaluating LM-mediated collaborative problem solving. Given a user request, PEO-PLEJOIN agents must identify teammates who might be able to assist, converse with these teammates to gather information, and finally compile a useful answer or summary for the original user. PEOPLEJOIN comprises two evaluation domains: PEOPLEJOIN-QA, focused on questions about tabular data, and PEOPLEJOIN-DOCCREATION, focused on document creation tasks. The two domains are adapted from existing NLP benchmarks for database question answering and multi-document summarization; here, however, the information needed to complete these tasks is distributed across synthetic "organizations" of 2-20 users, simulating natural multi-user collaboration scenarios. We implemented several popular LLM agent architectures, evaluating their accuracy and efficiency at completing tasks, and highlight new research questions that can be studied using PEOPLE-JOIN. 1 Alice Alice's Agent Bhushan Chen Knowledge Base Can you find a list of students with pet dogs? doc_search("students with pet dogs") Pet records: id: PET017, name: Fido, type: Dog people_search("knows about students or dogs") ["Chen might know about student records", "Bhushan might know about student pet IDs", "Emilia might know about faculty salaries"] Bhushan and Chen might be able to help. I will reach out to them. Hi Bhushan, can you share the names of students with pet dogs? I don't know about pet types, but here's a list of student IDs and pet IDs: 00158: PET027, 10532: PET015, 09731: PET008 Hi Chen, I'm helping Alice get information about dog owners. Can you tell me the names of the students with the IDs 00158, ….