Chapter 12: Stream Processing

Reference 1 in Chapter 12: The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing

Tyler Akidau, Robert Bradshaw, Craig Chambers and 8 others

Proceedings of the VLDB Endowment 2015

vldb.org PDF doi

Reference 2 in Chapter 12: Structure and Interpretation of Computer Programs

Harold Abelson, Gerald Jay Sussman and Julie Sussman

1996

web.mit.edu PDF

Reference 3 in Chapter 12: The Many Faces of Publish/Subscribe

Patrick Th. Eugster, Pascal A. Felber, Rachid Guerraoui and Anne-Marie Kermarrec

ACM Computing Surveys 2003

cs.ru.nl PDF doi

Reference 4 in Chapter 12: Pushing Back

Matthew Sackman

2016

wellquite.org

Reference 5 in Chapter 12: How (Not) to Write a Pipeline

Thomas Figg (tef)

cohost.org 2023

web.archive.org

Reference 6 in Chapter 12: Brubeck, a statsd-Compatible Metrics Aggregator

Vicent Martí

2015

github.blog

Reference 7 in Chapter 12: MoldUDP64 Protocol Specification V 1.00

Seth Lowenberger

2009

nasdaqtrader.com PDF

Reference 8 in Chapter 12: Measure Anything, Measure Everything

Ian Malpass

2011

codeascraft.com

Reference 9 in Chapter 12: 25 Graphite, Grafana and statsd Gotchas

Dieter Plaetinck

2016

grafana.com

Reference 10 in Chapter 12: Web Hooks to Revolutionize the Web

Jeff Lindsay

progrium.com 2007

progrium.github.io

Reference 11 in Chapter 12: Queues Are Databases

Jim N. Gray

1995

arxiv.org PDF

Reference 12 in Chapter 12: JSR-343 Java Message Service (JMS) 2.0 Specification

Mark Hapner, Rich Burridge, Rahul Sharma and 3 others

jms-spec.java.net 2013

jcp.org

Reference 13 in Chapter 12: AMQP: Advanced Message Queuing Protocol Specification

Sanjay Aiyagari, Matthew Arrott, Mark Atwell and 16 others

2008

rabbitmq.com PDF

Reference 14 in Chapter 12: Architectural Overview of Pub/Sub

2025

cloud.google.com

Reference 15 in Chapter 12: Lessons from Scaling PostgreSQL Queues to 100k Events Per Second

Aris Tzoumas

2025

rudderstack.com

Reference 16 in Chapter 12: Kafka Connect Deep Dive—Error Handling and Dead Letter Queues

Robin Moffatt

2019

confluent.io

Reference 17 in Chapter 12: Message Reprocessing: How to Implement the Dead Letter Queue

Dunith Danushka

redpanda.com

Reference 18 in Chapter 12: KIP-1034: Dead Letter Queue in Kafka Streams

Damien Gasparina, Loic Greffier and Sebastien Viale

2024

cwiki.apache.org

Reference 19 in Chapter 12: Kafka: A Distributed Messaging System for Log Processing

Jay Kreps, Neha Narkhede and Jun Rao

6th International Workshop on Networking Meets Databases 2011

microsoft.com PDF

Reference 20 in Chapter 12: Benchmarking Apache Kafka: 2 Million Writes Per Second (On Three Cheap Machines)

Jay Kreps

2014

engineering.linkedin.com

Reference 21 in Chapter 12: How We’re Improving and Advancing Kafka at LinkedIn

Kartik Paramasivam

2015

engineering.linkedin.com

Reference 22 in Chapter 12: Kafka Versus RabbitMQ: A Comparative Study of Two Industry Reference Publish/Subscribe Implementations

Philippe Dobbelaere and Kyumars Sheykh Esmaili

11th ACM International Conference on Distributed and Event-Based Systems 2017

arxiv.org doi

Reference 23 in Chapter 12: Why Message Queues Endure: A History

Kate Holterhoff

2024

redmonk.com

Reference 24 in Chapter 12: KIP-932: Queues for Kafka

Andrew Schofield

2023

cwiki.apache.org

Reference 25 in Chapter 12: The Advantages of Queues on Logs

Jack Vanlightly

2023

jack-vanlightly.com

Reference 26 in Chapter 12: The Log: What Every Software Engineer Should Know About Real-Time Data's Unifying Abstraction

Jay Kreps

2013

engineering.linkedin.com gone archived copy

Reference 27 in Chapter 12: Change Data Capture Is Having a Moment. Why?

Andy Hattemer

2021

materialize.com

Reference 28 in Chapter 12: Streaming MySQL Tables in Real-Time to Kafka

Prem Santosh Udaya Shankar

2016

engineeringblog.yelp.com

Reference 29 in Chapter 12: DBLog: A Watermark Based Change-Data-Capture Framework

Andreas Andreakis and Ioannis Papapanagiotou

arXiv 2020

arxiv.org PDF

Reference 30 in Chapter 12: Percolator

Jiri Pechanec

2021

debezium.io

Reference 31 in Chapter 12: Debezium Connector for Cassandra

Debezium maintainers

debezium.io

Reference 32 in Chapter 12: Announcing Kafka Connect: Building Large-Scale Low-Latency Data Pipelines

Neha Narkhede

2016

confluent.io

Reference 33 in Chapter 12: Kafka Change Data Capture Breaks Database Encapsulation

Chris Riccomini

2018

cnr.sh

Reference 34 in Chapter 12: 'Change Data Capture Breaks Encapsulation'. Does It, Though?

Gunnar Morling

2023

decodable.co

Reference 35 in Chapter 12: Revisiting the Outbox Pattern

Gunnar Morling

2024

decodable.co

Reference 36 in Chapter 12: Maintenance of Materialized Views: Problems, Techniques, and Applications

Ashish Gupta and Inderpal Singh Mumick

IEEE Data Engineering Bulletin 1995

web.archive.org PDF

Reference 37 in Chapter 12: DBSP: Incremental Computation on Streams and Its Applications to Databases

Mihai Budiu, Tej Chajed, Frank McSherry and 2 others

SIGMOD Record 2024

sigmodrecord.org PDF doi

Reference 38 in Chapter 12: Transaction Processing: Concepts and Techniques

Jim Gray and Andreas Reuter

1992

learning.oreilly.com

Reference 39 in Chapter 12: Accounting for Computer Scientists

Martin Kleppmann

2011

martin.kleppmann.com

Reference 40 in Chapter 12: Immutability Changes Everything

Pat Helland

7th Biennial Conference on Innovative Data Systems Research 2015

cidrdb.org PDF

Reference 41 in Chapter 12: Making Sense of Stream Processing

Martin Kleppmann

2016

martin.kleppmann.com PDF

Reference 42 in Chapter 12: Stream Processing Hard Problems—Part 1: Killing Lambda

Kartik Paramasivam

2016

engineering.linkedin.com

Reference 43 in Chapter 12: CQRS: What? Why? How?

Stéphane Derosiaux

2019

sderosiaux.medium.com

Reference 44 in Chapter 12: Immutability, MVCC, and Garbage Collection

Baron Schwartz

xaprb.com 2013

web.archive.org

Reference 45 in Chapter 12: Re: Turning the Database Inside-out with Apache Samza

Daniel Eloff, Slava Akhmechet, Jay Kreps and et al

2015

news.ycombinator.com

Reference 46 in Chapter 12: Datomic Documentation: Excision

Cognitect, Inc

docs.datomic.com

Reference 47 in Chapter 12: Fossil Documentation: Deleting Content from Fossil

2025

fossil-scm.org

Reference 48 in Chapter 12: The irony of distributed systems is that data loss is really easy but deleting data is surprisingly hard

Jay Kreps

2015

x.com

Reference 49 in Chapter 12: Crypto Shredding: How It Can Solve Modern Data Retention Challenges

Brent Robinson

2019

medium.com

Reference 50 in Chapter 12: Forward Secure Asynchronous Messaging from Puncturable Encryption

Matthew D. Green and Ian Miers

IEEE Symposium on Security and Privacy 2015

isi.jhu.edu PDF doi

Reference 51 in Chapter 12: What’s the Difference Between ESP and CEP?

David C. Luckham

complexevents.com

Reference 52 in Chapter 12: The CQL Continuous Query Language: Semantic Foundations and Query Execution

Arvind Arasu, Shivnath Babu and Jennifer Widom

The VLDB Journal 2006

microsoft.com PDF doi

Reference 53 in Chapter 12: Data in Flight: How Streaming SQL Technology Can Help Solve the Web 2.0 Data Crunch

Julian Hyde

ACM Queue 2009

queue.acm.org doi

Reference 54 in Chapter 12: HyperLogLog

Philippe Flajolet, Éric Fusy, Olivier Gandouet and Frédéric Meunier

algo.inria.fr PDF

Reference 55 in Chapter 12: Questioning the Lambda Architecture

Jay Kreps

2014

oreilly.com

Reference 56 in Chapter 12: An Overview of Apache Streaming Technologies

Ian Reppel

2016

ianreppel.org

Reference 57 in Chapter 12: Why Local State Is a Fundamental Primitive in Stream Processing

Jay Kreps

2014

oreilly.com

Reference 58 in Chapter 12: Deep Dive into the RisingWave Stream Processing Engine—Part 2: Computational Model

RisingWave Labs

2023

risingwave.com

Reference 59 in Chapter 12: Differential Dataflow

Frank McSherry, Derek G. Murray, Rebecca Isaacs and Michael Isard

6th Biennial Conference on Innovative Data Systems Research 2013

cidrdb.org PDF

Reference 60 in Chapter 12: Incremental Computation in the Database

Andy Hattemer

2020

materialize.com

Reference 61 in Chapter 12: Percolator

Shay Banon

2011

elastic.co

Reference 62 in Chapter 12: Real-Time Full-Text Search with Luwak and Samza

Alan Woodward and Martin Kleppmann

2015

martin.kleppmann.com

Reference 63 in Chapter 12: The World Beyond Batch: Streaming 102

Tyler Akidau

2016

oreilly.com

Reference 64 in Chapter 12: Streaming Analytics with Apache Flink

Stephan Ewen

Kafka Summit 2016

slideshare.net

Reference 65 in Chapter 12: MillWheel: Fault-Tolerant Stream Processing at Internet Scale

Tyler Akidau, Alex Balikov, Kaya Bekiroğlu and 7 others

Proceedings of the VLDB Endowment 2013

vldb.org PDF doi

Reference 66 in Chapter 12: Improving Snowplow's Understanding of Time

Alex Dean

2015

snowplow.io

Reference 67 in Chapter 12: Azure Stream Analytics: Windowing Functions

2025

learn.microsoft.com

Reference 68 in Chapter 12: Photon: Fault-Tolerant and Scalable Joining of Continuous Data Streams

Rajagopal Ananthanarayanan, Venkatesh Basker, Sumit Das and 7 others

ACM International Conference on Management of Data 2013

research.google.com PDF doi

Reference 69 in Chapter 12: Doing the Impossible: Exactly-Once Messaging Patterns in Kafka

Ben Kirwin

2014

ben.kirw.in

Reference 70 in Chapter 12: Data on the Outside Versus Data on the Inside

Pat Helland

2nd Biennial Conference on Innovative Data Systems Research 2005

cidrdb.org PDF

Reference 71 in Chapter 12: The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling

Ralph Kimball and Margy Ross

2013

learning.oreilly.com

Reference 72 in Chapter 12: I'm coining the phrase 'effectively-once' for message processing with at-least-once + idempotent operations

Viktor Klang

2016

x.com

Reference 73 in Chapter 12: High-Throughput, Low-Latency, and Exactly-Once Stream Processing with Apache Flink

Kostas Tzoumas, Stephan Ewen and Robert Metzger

ververica.com 2015

web.archive.org gone

Reference 74 in Chapter 12: Lightweight Asynchronous Snapshots for Distributed Dataflows

Paris Carbone, Gyula Fóra, Stephan Ewen and 2 others

arXiv 2015

arxiv.org

Reference 75 in Chapter 12: Fast Data: Smart and at Scale

Ryan Betts and John Hugg

2015

voltactivedata.com PDF

Reference 76 in Chapter 12: Exactly-Once Semantics Are Possible: Here’s How Kafka Does It

Neha Narkhede and Guozhang Wang

2019

confluent.io

Reference 77 in Chapter 12: KIP-98—Exactly Once Delivery and Transactional Messaging

Jason Gustafson, Flavio Junqueira, Apurva Mehta and 2 others

2016

cwiki.apache.org

Reference 78 in Chapter 12: Idempotence Is Not a Medical Condition

Pat Helland

Communications of the ACM 2012

dl.acm.org doi

Reference 79 in Chapter 12: Re: Trying to Achieve Deterministic Behavior on Recovery/Rewind

Jay Kreps

samza-dev 2014

lists.apache.org

Reference 80 in Chapter 12: A Survey of Rollback-Recovery Protocols in Message-Passing Systems

E. N. (Mootaz) Elnozahy, Lorenzo Alvisi, Yi-Min Wang and David B. Johnson

ACM Computing Surveys 2002

cs.utexas.edu PDF doi

Reference 81 in Chapter 12: Kafka Streams—How Does It Fit the Stream Processing Landscape?

Adam Warski

2016

softwaremill.com

Reference 82 in Chapter 12: Batch as a Special Case of Streaming and Alibaba's contribution of Blink

Stephan Ewen, Fabian Hueske and Xiaowei Jiang

2019

flink.apache.org