Towards Event Extraction with Massive Types: LLM-based Collaborative Annotation and Partitioning Extraction

Developing a general-purpose system that can extract events with massive types is a longstanding target in Event Extraction (EE). In doing so, the basic challenge comes from the absence of an efficient and effective annotation framework to construct the corresponding datasets. In this paper, we propose an LLM-based collaborative annotation framework. Through collaboration among multiple LLMs and a subsequent voting process, it refines annotations of triggers from distant supervision and then carries out argument annotation. Finally, we create EEMT, the largest EE dataset to date, featuring over 200,000 samples, 3,465 event types, and 6,297 role types. Evaluation on the human-annotated test set demonstrates that the proposed framework achieves the F1 scores of 90.1% and 85.3% for event detection and argument extraction, strongly validating its effectiveness. Besides, to alleviate the excessively long prompts caused by massive types, we propose an LLM-based Partitioning method for EE called LLM-PEE. It first recalls candidate event types and then splits them into multiple partitions for LLMs to extract. After fine-tuning on the EEMT training set, the distilled LLM-PEE with 7B parameters outperforms state-of-the-art methods by 5.4% and 6.1% in event detection and argument extraction. Besides, it also surpasses mainstream LLMs by 12.9% on the unseen datasets, which strongly demonstrates the event diversity of the EEMT dataset and the generalization capabilities of the LLM-PEE method.