Online Event Processing: Achieving consistency where distributed transactions have failed

Martin Kleppmann, Alastair R. Beresford, and Boerge Svingen

Communications of the ACM, Volume 62, Issue 5, pages 43-49, May 2019.

• Also appears in ACM Queue, Volume 17, Issue 1, February 2019 (HTML version, PDF version).

Abstract

Distributed transactions have failed as a mechanism for ensuring consistency across heterogeneous storage technologies in today’s large-scale applications. Instead, we are witnessing the emergence of a programming model that relies on append-only event logs rather than transactions, and which we call OnLine Event Processing (OLEP) in contrast to OLTP. In this article we show that, although an event log is a very simple abstraction, applications that rely on such logs can efficiently provide strong consistency guarantees, such as atomicity and enforcing invariants, which are normally associated with ACID transactions. We provide case studies from real industrial data systems that have adopted the OLEP approach, demonstrating the practical advantages of building upon event logs.