Chapter 11: Batch Processing

Reference 1 in Chapter 11: How to Beat the CAP Theorem

Nathan Marz

2011

nathanmarz.com

Reference 2 in Chapter 11: Agile Architecture

Molly Bartlett Dishman and Martin Fowler

O'Reilly Software Architecture Conference 2015

youtube.com

Reference 3 in Chapter 11: MapReduce: Simplified Data Processing on Large Clusters

Jeffrey Dean and Sanjay Ghemawat

6th USENIX Symposium on Operating System Design and Implementation 2004

usenix.org PDF

Reference 4 in Chapter 11: Massively Parallel Databases and MapReduce Systems

Shivnath Babu and Herodotos Herodotou

Foundations and Trends in Databases 2013

microsoft.com PDF doi

Reference 5 in Chapter 11: MapReduce: A Major Step Backwards

David J. DeWitt and Michael Stonebraker

databasecolumn.vertica.com 2008

homes.cs.washington.edu

Reference 6 in Chapter 11: The Elephant Was a Trojan Horse: On the Death of Map-Reduce at Google

Henry Robinson

2014

the-paper-trail.org

Reference 7 in Chapter 11: R.I.P. MapReduce. After having served us well since 2003, today we removed the remaining internal codebase for good

Urs Hölzle

x.com 2019

twitter.com

Reference 8 in Chapter 11: Command-Line Tools Can Be 235x Faster than Your Hadoop Cluster

Adam Drake

aadrake.com 2014

adamdrake.com

Reference 9 in Chapter 11: sort: Sort Text Files

2025

gnu.org

Reference 10 in Chapter 11: The Quantcast File System

Michael Ovsiannikov, Silvius Rus, Damian Reeves and 3 others

Proceedings of the VLDB Endowment 2013

db.disi.unitn.eu PDF doi

Reference 11 in Chapter 11: Introduction to HDFS Erasure Coding in Apache Hadoop

Andrew Wang, Zhe Zhang, Kai Zheng and 2 others

blog.cloudera.com 2015

cloudera.com

Reference 12 in Chapter 11: Building and Operating a Pretty Big Storage System Called S3

Andy Warfield

2023

allthingsdistributed.com

Reference 13 in Chapter 11: Apache Hadoop YARN: Yet Another Resource Negotiator

Vinod Kumar Vavilapalli, Arun C. Murthy, Chris Douglas and 13 others

4th Annual Symposium on Cloud Computing 2013

opencourse.inf.ed.ac.uk PDF doi

Reference 14 in Chapter 11: Reducibility Among Combinatorial Problems

Richard M. Karp

Complexity of Computer Computations. The IBM Research Symposia Series 1972

cs.purdue.edu PDF gone doi

Reference 15 in Chapter 11: NP-Complete Scheduling Problems

J. D. Ullman

Journal of Computer and System Sciences 1975

cs.montana.edu PDF doi

Reference 16 in Chapter 11: The Complete Guide to Spot Instances on AWS, Azure and GCP

Gilad David Maayan

2021

datacenterdynamics.com

Reference 17 in Chapter 11: Large-Scale Cluster Management at Google with Borg

Abhishek Verma, Luis Pedrosa, Madhukar Korupolu and 3 others

10th European Conference on Computer Systems 2015

dl.acm.org doi

Reference 18 in Chapter 11: Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing

Matei Zaharia, Mosharaf Chowdhury, Tathagata Das and 6 others

9th USENIX Symposium on Networked Systems Design and Implementation 2012

usenix.org PDF

Reference 19 in Chapter 11: Hadoop Application Architectures

Mark Grover, Ted Malaska, Jonathan Seidman and Gwen Shapira

2015

learning.oreilly.com

Reference 20 in Chapter 11: Learning Spark

Jules S. Damji, Brooke Wenig, Tathagata Das and Denny Lee

2020

learning.oreilly.com

Reference 21 in Chapter 11: Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks

Michael Isard, Mihai Budiu, Yuan Yu and 2 others

2nd European Conference on Computer Systems 2007

microsoft.com doi

Reference 22 in Chapter 11: Nephele: Efficient Parallel Data Processing in the Cloud

Daniel Warneke and Odej Kao

2nd Workshop on Many-Task Computing on Grids and Supercomputers 2009

stratosphere2.dima.tu-berlin.de PDF doi

Reference 23 in Chapter 11: In-Memory Query Execution in Google BigQuery

Hossein Ahmadi

2016

cloud.google.com

Reference 24 in Chapter 11: Hadoop: The Definitive Guide

Tom White

2015

learning.oreilly.com

Reference 25 in Chapter 11: Peeking into Apache Flink's Engine Room

Fabian Hüske

2015

flink.apache.org

Reference 26 in Chapter 11: Hive 0.14 Cost Based Optimizer (CBO) Technical Overview

Mostafa Mokhtar

hortonworks.com 2015

web.archive.org

Reference 27 in Chapter 11: Spark SQL: Relational Data Processing in Spark

Michael Armbrust, Reynold S. Xin, Cheng Lian and 8 others

ACM International Conference on Management of Data 2015

people.csail.mit.edu PDF doi

Reference 28 in Chapter 11: Spark vs. Pandas, Part 2—Spark

Kaya Kupferschmidt

2020

towardsdatascience.com

Reference 29 in Chapter 11: Tracking Payments at Scale

Ammar Chalifah

bolt.eu.com 2025

bolt.eu

Reference 30 in Chapter 11: Demand Forecasting at Getir Built with Amazon Forecast

Nafi Ahmet Turgut, Hamza Akyıldız, Hasan Burak Yel and 4 others

2023

aws.amazon.com

Reference 31 in Chapter 11: Enhancing Homepage Feed Relevance by Harnessing the Power of Large Corpus Sparse ID Embeddings

Jason (Siyu) Zhu

2023

linkedin.com

Reference 32 in Chapter 11: Apache Spark @Scale: A 60 TB+ Production Use Case

Avery Ching, Sital Kedia and Shuojie Wang

2016

engineering.fb.com

Reference 33 in Chapter 11: How ACH Works: A Developer Perspective—Part 1

Edward Kim

2014

engineering.gusto.com

Reference 34 in Chapter 11: How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh

Zhamak Dehghani

2019

martinfowler.com

Reference 35 in Chapter 11: What the Heck Is a Data Mesh?!

Chris Riccomini

2021

cnr.sh

Reference 36 in Chapter 11: Data Contracts

Chad Sanderson, Mark Freeman and B. E. Schmidt

2025

oreilly.com

Reference 37 in Chapter 11: Data Fabric vs. Data Mesh: What's the Difference?

Daniel Abadi

2021

starburst.io

Reference 38 in Chapter 11: Lakehouse: A New Generation of Open Platforms That Unify Data Warehousing and Advanced Analytics

Michael Armbrust, Ali Ghodsi, Reynold Xin and Matei Zaharia

11th Annual Conference on Innovative Data Systems Research 2021

cidrdb.org PDF

Reference 39 in Chapter 11: A Bridging Model for Parallel Computation

Leslie G. Valiant

Communications of the ACM 1990

dl.acm.org doi

Reference 40 in Chapter 11: Spinning Fast Iterative Data Flows

Stephan Ewen, Kostas Tzoumas, Moritz Kaufmann and Volker Markl

Proceedings of the VLDB Endowment 2012

vldb.org PDF doi

Reference 41 in Chapter 11: Pregel: A System for Large-Scale Graph Processing

Grzegorz Malewicz, Matthew H. Austern, Aart J. C. Bik and 4 others

ACM International Conference on Management of Data 2010

kowshik.github.io PDF doi

Reference 42 in Chapter 11: OpenAI Chats About Scaling LLMs at Anyscale’s Ray Summit

Richard MacManus

2023

thenewstack.io

Reference 43 in Chapter 11: Why Local State Is a Fundamental Primitive in Stream Processing

Jay Kreps

2014

oreilly.com

Reference 44 in Chapter 11: Open Sourcing Venice—LinkedIn’s Derived Data Platform

Félix GV

2022

linkedin.com