Blackbox Observability of Features and Feature Interactions

Configurable software systems offer user-selectable features to tailor them to the target hardware and user requirements. It is almost a rule that, as the number of features increases over time, unintended and inadvertent feature interactions arise. Despite numerous definitions of feature interactions and methods for detecting them, there is no procedure for determining whether the effect of a feature interaction could be, in principle, observed from an external perspective. In this paper, we devise a decision procedure to verify whether the effect of a given feature or potential feature interaction could be isolated by blackbox observations of a set of system configurations. For this purpose, we introduce the notion of blackbox observability, which is based on recent work on counterfactual reasoning on configuration decisions. Direct observability requires a single reference configuration to isolate the effect in question, while the broader notion of general observability relaxes this precondition and suffices with a set of reference configurations. We report on a series of experiments on community benchmarks as well as real-world configuration spaces and models. We found that (1) deciding observability is indeed tractable in real-world settings, (2) constraints in real-world configuration spaces frequently limit observability, and (3) blackbox performance models often include effects that are de facto not observable.