One Pass to Parse Them All: Fused Parallel CSV Processing

CSV remains one of the most widely used formats for exchanging tabular data, making efficient CSV processing an important problem. Yet most CSV parsers are sequential, failing to exploit the parallelism of modern hardware. While parallel CSV parsing approaches have been proposed in the literature, none of these seem to be used in practice. Conversely, a simple idea for synchronization-free speculative parsing that is used for parallel parsing, e.g., in DuckDB, has never been described in the literature, nor has it been exploited efficiently. In this paper, we close this gap. We contribute a) a description of how real-world CSV files can be parsed in parallel on commodity multicore CPUs, b) a new programming model for general-purpose CSV parsers that unifies parallel parsing and parallel data processing into one pass over the data, and c) a new vectorization strategy with efficient index and zero-copy record construction to accelerate parsing. Our evaluation shows that csveee, our parser, outperforms widely-used CSV parsers in single- and multi-threaded performance, and scales near-linearly to achieve throughput of up to 180 GB/s — 22× faster than DuckDB — all while remaining practical for integration into real-world data processing systems.