Supporting Organizations in Improving Employee Bulk E-mail - A Tool Design and Evaluation Study

Organizations often send bulk emails to employees to make them aware of policy changes, organization plans, and events. Many of these emails, however, are long digests with many separate messages that waste employees' time and reduce their awareness. This study introduces CommTool--a prototype tool to help organizational communicators better understand their emails' performance and cost. We first interviewed 5 communicators and identified the need to measure the performance of each message within bulk email. Then we iteratively designed and deployed an organizational bulk email evaluation platform (CommTool), which enables communicators to get diverse message-level metrics such as reading time, relevance rate, comments, etc. We evaluated these designs through a 2-month field deployment with 5 communicators and 149 organization employees. We found that 1) the message-level metrics, such as reading time and relevance rate, helped communicators understand their audience and design bulk emails; 2) the cost and reputation metrics did not influence the organization leaders' decisions. We summarize with suggestions on designing organizational bulk email evaluation platforms that provide message-level performance and cost information.