Spangle: A Distributed In-Memory Processing System for Large-Scale Arrays

With increasing volumes of scientific data, a scalable and parallel computing framework is required for scientific analysis in computer simulations and experiments. Scientific data are commonly generated in multi-dimensional arrays, and the array data model is appropriate to store them for analysis, including for data mining and arithmetic computation. In this paper, we introduce an array processing system called Spangle. It is implemented on top of Apache Spark, a popular map-reduce framework for complex computation workloads. To support array data computation, we extended Resilient Distributed Dataset (RDD) based on the array data model named ArrayRDD. ArrayRDD is an inherently parallel data structure that provides fault-tolerance. In addition, by adopting the array data model, Spangle provides an interface for expressing machine learning algorithms, which heavily rely on linear algebra. We tailored two popular algorithms, PageRank and Stochastic Gradient Descent, for large-scale datasets in Spangle.