MetaInsight: Automatic Discovery of Structured Knowledge for Exploratory Data Analysis

Automatic Exploratory Data Analysis (EDA) focuses on automatically discovering pieces of knowledge in the form of interesting data patterns. However, the knowledge conveyed by these suggested data patterns is disjointed or lacks of organization. Therefore, it is difficult for users to gain structured knowledge. As the number of suggested patterns grows, these stand-alone patterns are less likely to motivate users to conduct follow-up analysis, which hinders the suggested patterns from being effectively utilized to facilitate EDA. In this paper, we propose MetaInsight, a structured representation of knowledge extracted from multi-dimensional data, which aims to facilitate EDA effectively. Specifically, we propose a novel formulation of basic data patterns to capture essential characteristics of the raw data distribution to achieve knowledge extraction. Then, based on the mined homogeneous data patterns (HDPs) and inter-pattern similarity, MetaInsights are identified by categorizing basic data patterns (within an HDP) into commonness(es) and exceptions, thus achieving structured knowledge representation. The commonness(es) and exceptions concretize knowledge obtained by the induction and validation processes, which are two typical analysis mechanisms conducted in EDA. We propose a novel scoring function to quantify the usefulness of MetaInsights, an effective and efficient mining procedure and a ranking algorithm to automatically discover high-quality MetaInsights from multi-dimensional data. We demonstrate the effectiveness and efficiency of MetaInsight (w.r.t. facilitating EDA) through experiments on real-world datasets and user studies on both expert and non-expert users.