Skip to main content
FOSDEM PWA

Apache Arrow, Hostage Negotiator: Revisiting the case for Client Protocol Redesign

UB2.252A (Lameere) | Day 1 | 17:00 - 17:20 | Speakers: Matthew Topol

Apache Arrow, Hostage Negotiator: Revisiting the case for Client Protocol Redesign
A picture of a devroom at FOSDEM 2024
Open in browser

Notes

Abstract

In 2017, Mark Raasveldt and Hannes Mühleisen (who went on to create DuckDB presented a VLDB paper entitled “Don’t Hold My Data Hostage – A Case For Client Protocol Redesign.” Their paper proposed the use of columnar serialization to achieve order-of-magnitude improvements in query result transfer performance. Eight years later, this talk revisits Raasveldt and Mühleisen’s argument and describes the central role that the Apache Arrow project has played in realizing this vision—through the dissemination of Arrow IPC, Arrow Flight, Arrow Flight SQL, Arrow over HTTP, and ADBC across numerous open source and commercial query systems. The talk concludes with a call to action to introduce Arrow-based transport to the systems that continue to “hold data hostage.”

Speakers

Matthew Topol

Matt Topol is a PMC member of both the Apache Arrow and Apache Iceberg projects being prolific in both communities. He is currently the primary maintainer of the Arrow Go libraries (including Parquet, Flight and FlightSQL), the creator of the iceberg-go implementation, and wrote the first book on Apache Arrow, "In-Memory Analytics with Apache Arrow". Matt is also a member of the ASF and recently started a new company, Columnar, with two colleagues from the Apache Arrow community focusing on data connectivity using Arrow Database Connectivity (ADBC).

In his spare time, Matt likes to bash his head against a keyboard, develop/run delightfully demented games of fantasy for his victims--er--friends, and share his knowledge with anyone interested who'll listen to his rants.

Notice: The placeholder video image is licensed under CC BY-SA 4.0. The original image can be found hereChanges made to the image are: Cropped the image to a new ratio, part of the image was cut off.