Interactive Data Analysis with Lively Typed Tables

Programming systems tailored for working with tabular data ( tabular programming systems ), such as spreadsheets and computational notebooks, are essential tools in data science. However, widely adopted systems are limited by the absence of static typing, which restricts the editor support they can provide, particularly when code is organized into reusable functions. Statically typed alternatives are limited by the fact that many useful operations on tables produce results whose column schema is data-dependent, e.g. pivots, unstack operations, or one-hot encodings. This paper introduces Hazel Lab , a new tabular programming system built as an extension of Hazel, a live gradually typed functional programming environment. It aims to combine the expressivity of dynamically typed systems with the editor support of static typing by incorporating several novel mechanisms into Hazel. Tables are manipulated as sequences of labeled tuples, and we add several useful gradually typed operations on labeled tuples to increase expressiveness, including operations that convert field names to and from strings. These operations support best-effort static typing and fall back to the unknown type when a schema cannot be statically determined. We evaluate the expressiveness of these core abstractions using the Brown Benchmark For Table Types (B2T2), finding that Hazel Lab is able to reasonably express every example. In order to improve type-based feedback, including error localization, when the fallback to the unknown type is needed, we introduce live typing , which builds on the fact that Hazel is a maximally live programming environment, even when there are static errors in the code, to feed dynamically observed instantiations of statically unknown types back into the static type checker. This feedback is complementary to the dynamic feedback that Hazel already distinctively provides by way of its live probes. We introduce rich probes—an extension of live probes with domain-specific table views that allow users to edit their functional data pipelines through direct manipulation interactions. We evaluate the usability of our overall design by conducting a lab study with 7 participants, asking them to perform a variety of data cleaning and analysis tasks. The study evaluates the usability and usefulness of the proposed features for users already familiar with statically typed functional programming, rather than to assess transfer to data science workflows by scientists without that training. We find that participants could effectively use the table operations for data cleaning and transformation tasks, and that live typing helped them both understand and debug existing analyses. Most participants responded positively to adopting the evaluated features in their own programming environments, with none responding negatively.