The Trouble with Distributed Systems

Reference 2 in Chapter 9: Getting Real About Distributed System Reliability

Jay Kreps

2012

blog.empathybox.com

Reference 3 in Chapter 9: You Can’t Sacrifice Partition Tolerance

Coda Hale

codahale.com 2010

perma.cc

Reference 4 in Chapter 9: Notes on Distributed Systems for Young Bloods

Jeff Hodges

2013

somethingsimilar.com

Reference 5 in Chapter 9: Congestion Avoidance and Control

Van Jacobson

ACM Symposium on Communications Architectures and Protocols 1988

cs.usask.ca PDF doi

Reference 6 in Chapter 9: The Ultimate SO_LINGER Page, or: Why Is My TCP Not Reliable

Bert Hubert

2009

blog.netherlabs.nl

Reference 7 in Chapter 9: End-To-End Arguments in System Design

Jerome H. Saltzer, David P. Reed and David D. Clark

ACM Transactions on Computer Systems 1984

groups.csail.mit.edu PDF doi

Reference 8 in Chapter 9: The Network Is Reliable

Peter Bailis and Kyle Kingsbury

ACM Queue 2014

Reference 9 in Chapter 9: Taming Uncertainty in Distributed Systems with Help from the Network

Joshua B. Leners, Trinabh Gupta, Marcos K. Aguilera and Michael Walfish

10th European Conference on Computer Systems 2015

cs.nyu.edu PDF doi

Reference 10 in Chapter 9: Understanding Network Failures in Data Centers: Measurement, Analysis, and Implications

Phillipa Gill, Navendu Jain and Nachiappan Nagappan

ACM SIGCOMM Conference 2011

conferences.sigcomm.org PDF doi

Reference 11 in Chapter 9: But recently a farmer had started grazing a herd of cows nearby. And whenever they stepped on the fiber link, they bent it enough to cause a blip

Urs Hölzle

2020

x.com

Reference 12 in Chapter 9: Hundreds Lose Internet Service in Northern B.C. After Beaver Chews Through Cable

CBC News

2021

cbc.ca gone archived copy

Reference 13 in Chapter 9: The Global Internet Is Being Attacked by Sharks, Google Confirms

Will Oremus

2014

slate.com

Reference 14 in Chapter 9: Down to the Wire: The Ship Fixing Our Internet

Jess Auerbach Jahajeeah

2023

continent.substack.com

Reference 15 in Chapter 9: More Details About the October 4 Outage

Santosh Janardhan

2021

engineering.fb.com

Reference 16 in Chapter 9: Georgian Woman Cuts off Web Access to Whole of Armenia

Tom Parfitt

2011

theguardian.com

Reference 17 in Chapter 9: Shadow Fleets’ and Subaquatic Sabotage: Are Europe’s Undersea Internet Cables Under Attack?

Antonio Voce, Tural Ahmedzade and Ashley Kirk

2025

theguardian.com

Reference 18 in Chapter 9: XFT: Practical Fault Tolerance Beyond Crashes

Shengyun Liu, Paolo Viotti, Christian Cachin and 2 others

12th USENIX Symposium on Operating Systems Design and Implementation 2016

Reference 19 in Chapter 9: Downtime Last Saturday

Mark Imbriaco

2012

github.blog

Reference 20 in Chapter 9: A Byzantine Failure in the Real World

Tom Lianza and Chris Snook

2020

blog.cloudflare.com

Reference 21 in Chapter 9: Toward a Generic Fault Tolerance Technique for Partial Network Partitioning

Mohammed Alfatafta, Basil Alkhatib, Ahmed Alquraan and Samer Al-Kiswany

14th USENIX Symposium on Operating Systems Design and Implementation 2020

usenix.org

Reference 22 in Chapter 9: Re: bnx2 Cards Intermittantly Going Offline

Marc A. Donges

netdev 2012

spinics.net

Reference 23 in Chapter 9: Inside a CODE RED: Network Edition

Troy Toman

2020

signalvnoise.com

Reference 24 in Chapter 9: Jepsen: Elasticsearch

Kyle Kingsbury

2014

Reference 25 in Chapter 9: A Few Arguments About Redis Sentinel Properties and Fail Scenarios

Salvatore Sanfilippo

2014

antirez.com

Reference 26 in Chapter 9: CAP: If All You Have Is a Timeout, Everything Looks Like a Partition

Nicolas Liochon

2015

blog.thislongrun.com

Reference 27 in Chapter 9: Queues Don’t Matter When You Can JUMP Them!

Matthew P. Grosvenor, Malte Schwarzkopf, Ionel Gog and 4 others

12th USENIX Symposium on Networked Systems Design and Implementation 2015

ternarysearch.blogspot.com

Reference 28 in Chapter 9: Debugging Network Stalls on Kubernetes

Theo Julienne

2019

github.blog

Reference 29 in Chapter 9: The Impact of Virtualization on Network Performance of Amazon EC2 Data Center

Guohui Wang and T. S. Eugene Ng

29th IEEE International Conference on Computer Communications 2010

cs.rice.edu PDF gone doi

Reference 30 in Chapter 9: etcd: Distributed Locking and Service Discovery

Brandon Philips

Strange Loop 2014

youtube.com

Reference 31 in Chapter 9: A Systematic Look at EC2 I/O

Steve Newman

blog.scalyr.com 2012

sentinelone.com

Reference 32 in Chapter 9: The ϕ Accrual Failure Detector

Naohiro Hayashibara, Xavier Défago, Rami Yared and Takuya Katayama

2004

hdl.handle.net

Reference 33 in Chapter 9: Phi Accrual Failure Detector

Jeffrey Wang

ternarysearch.blogspot.co.uk 2013

Reference 34 in Chapter 9: An Engineering Approach to Computer Networking: ATM Networks, the Internet, and the Telephone Network

Srinivasan Keshav

1997

Reference 35 in Chapter 9: ATM Networks

Othmar Kyas

1995

Reference 36 in Chapter 9: Tales of the Tail: Hardware, OS, and Application-level Sources of Tail Latency

Jialin Li, Naveen Kr. Sharma, Dan R. K. Ports and Steven D. Gribble

ACM Symposium on Cloud Computing 2014

syslab.cs.washington.edu PDF doi

Reference 37 in Chapter 9: InfiniBand FAQ, Rev 1.3

Mellanox Technologies

2014

network.nvidia.com PDF

Reference 38 in Chapter 9: End-to-End Congestion Control for InfiniBand

Jose Renato Santos, Yoshio Turner and G. (John) Janakiraman

22nd Annual Joint Conference of the IEEE Computer and Communications Societies 2003

infocom2003.ieee-infocom.org PDF doi

Reference 39 in Chapter 9: The NTP FAQ and HOWTO

Ulrich Windl, David Dalton, Marc Martinec and Dale R. Worley

2006

ntp.org

Reference 40 in Chapter 9: How and Why the Leap Second Affected Cloudflare DNS

John Graham-Cumming

2017

blog.cloudflare.com

Reference 41 in Chapter 9: Inside the Hotspot VM: Clocks, Timers and Scheduling Events—Part I—Windows

David Holmes

blogs.oracle.com 2006

web.archive.org

Reference 42 in Chapter 9: Three Clocks Are Better than One

Joran Dirk Greef

2021

tigerbeetle.com

Reference 43 in Chapter 9: Pitfalls of TSC Usage

Oliver Yang

2015

oliveryang.net

Reference 44 in Chapter 9: Time on Multi-Core, Multi-Socket Servers

Steve Loughran

steveloughran.blogspot.co.uk 2015

steveloughran.blogspot.com

Reference 45 in Chapter 9: Spanner: Google’s Globally-Distributed Database

10th USENIX Symposium on Operating System Design and Implementation 2012

research.google

Reference 46 in Chapter 9: How Closely Can a Personal Computer Clock Track the UTC Timescale Via the Internet?

M. Caporaloni and R. Ambrosini

European Journal of Physics 2012

iopscience.iop.org doi

Reference 47 in Chapter 9: A Survey of the NTP Network

Nelson Minar

1999

alumni.media.mit.edu

Reference 48 in Chapter 9: Synchronizing Clocks in a Cassandra Cluster Pt. 1—The Problem

Viliam Holub

blog.rapid7.com 2014

rapid7.com

Reference 49 in Chapter 9: The One-Second War (What Time Will You Die?)

Poul-Henning Kamp

ACM Queue 2011

Reference 50 in Chapter 9: Leap Second Crashes Half the Internet

Nelson Minar

2012

somebits.com

Reference 51 in Chapter 9: Time, Technology and Leaping Seconds

Christopher Pascoe

googleblog.blogspot.co.uk 2011

googleblog.blogspot.com

Reference 52 in Chapter 9: Look Before You Leap—The Coming Leap Second and AWS

Mingxue Zhao and Jeff Barr

2015

aws.amazon.com

Reference 53 in Chapter 9: Network Timing and the 2015 Leap Second

Darryl Veitch and Kanthaiah Vijayalayan

17th International Conference on Passive and Active Measurement 2016

opus.lib.uts.edu.au PDF doi

Reference 54 in Chapter 9: Timekeeping in VMware Virtual Machines

VMware, Inc

2008

vmware.com PDF

Reference 55 in Chapter 9: Clock Synchronization in Finance and Beyond

Victor Yodaiken

2017

yodaiken.com PDF

Reference 56 in Chapter 9: Where the Wild Warnings Are: Root Causes of Chrome HTTPS Certificate Errors

Mustafa Emre Acer, Emily Stark, Adrienne Porter Felt and 6 others

ACM SIGSAC Conference on Computer and Communications Security 2017

acmccs.github.io PDF doi

Reference 57 in Chapter 9: MiFID II / MiFIR: Regulatory Technical and Implementing Standards—Annex I

European Securities and Markets Authority

2015

esma.europa.eu PDF

Reference 58 in Chapter 9: Solving MiFID II Clock Synchronisation with Minimum Spend (Part 1)

Luke Bigum

2015

catach.blogspot.com

Reference 59 in Chapter 9: How Precision Time Protocol Is Being Deployed at Meta

Oleg Obleukhov and Ahmad Byagowi

2022

engineering.fb.com

Reference 60 in Chapter 9: GPSJAM: Daily Maps of GPS Interference

John Wiseman

gpsjam.org

Reference 61 in Chapter 9: It’s About Time: Microsecond-Accurate Clocks on Amazon EC2 Instances

Josh Levinson, Julien Ridoux and Chris Munns

2023

aws.amazon.com

Reference 62 in Chapter 9: Jepsen: Cassandra

Kyle Kingsbury

2013

Reference 63 in Chapter 9: Clocks Are Bad, or, Welcome to the Wonderful World of Distributed Systems

John Daily

2013

riak.com

Reference 64 in Chapter 9: It’s About Time!

Marc Brooker

2023

brooker.co.za

Reference 65 in Chapter 9: The Trouble with Timestamps

Kyle Kingsbury

2013

Reference 66 in Chapter 9: Time, Clocks, and the Ordering of Events in a Distributed System

Leslie Lamport

Communications of the ACM 1978

microsoft.com doi

Reference 67 in Chapter 9: There Is No Now: Problems with Simultaneity in Distributed Systems

Justin Sheehy

ACM Queue 2015

muratbuffalo.blogspot.com

Reference 68 in Chapter 9: Spanner: Google's Globally-Distributed Database

Murat Demirbas

muratbuffalo.blogspot.co.uk 2013

Reference 69 in Chapter 9: Spanner's Concurrency Control

Dahlia Malkhi and Jean-Philippe Martin

ACM SIGACT News 2013

cs.cornell.edu PDF doi

Reference 70 in Chapter 9: Achieving Precise Clock Synchronization on AWS

Franck Pachot

2024

yugabyte.com

Reference 71 in Chapter 9: Living Without Atomic Clocks: Where CockroachDB and Spanner Diverge

Spencer Kimball

2022

cockroachlabs.com

Reference 72 in Chapter 9: Use of Time in Distributed Databases (Part 4): Synchronized Clocks in Production Databases

Murat Demirbas

2025

muratbuffalo.blogspot.com

Reference 73 in Chapter 9: Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency

Cary G. Gray and David R. Cheriton

12th ACM Symposium on Operating Systems Principles 1989

courses.cs.duke.edu PDF doi

Reference 74 in Chapter 9: Roblox Return to Service

Daniel Sturman, Scott Delap, Max Ross and et al

2022

corp.roblox.com

Reference 75 in Chapter 9: Avoiding Full GCs with MemStore-Local Allocation Buffers

Todd Lipcon

2011

slideshare.net

Reference 76 in Chapter 9: Live Migration of Virtual Machines

Christopher Clark, Keir Fraser, Steven Hand and 5 others

2nd USENIX Symposium on Symposium on Networked Systems Design & Implementation 2005

mechanical-sympathy.blogspot.com

Reference 77 in Chapter 9: fsyncers and Curveballs

Mike Shaver

shaver.off.net 2008

web.archive.org

Reference 78 in Chapter 9: Eliminating Large JVM GC Pauses Caused by Background IO Traffic

Zhenyun Zhuang and Cuong Tran

2016

engineering.linkedin.com

Reference 79 in Chapter 9: Java Garbage Collection Distilled

Martin Thompson

mechanical-sympathy.blogspot.co.uk 2013

Reference 80 in Chapter 9: Blade: A Data Center Garbage Collector

David Terei and Amit Levy

arXiv 2015

arxiv.org PDF

Reference 81 in Chapter 9: Trash Day: Coordinating Garbage Collection in Distributed Systems

Martin Maas, Tim Harris, Krste Asanović and John Kubiatowicz

15th USENIX Workshop on Hot Topics in Operating Systems 2015

timharris.uk PDF

Reference 82 in Chapter 9: The LMAX Architecture

Martin Fowler

2011

martinfowler.com

Reference 83 in Chapter 9: Knowledge and Common Knowledge in a Distributed Environment

Joseph Y. Halpern and Yoram Moses

Journal of the ACM 1990

groups.csail.mit.edu PDF doi

Reference 84 in Chapter 9: Ad Hoc Transactions in Web Applications: The Good, the Bad, and the Ugly

Chuzhe Tang, Zhaoguo Wang, Xiaodong Zhang and 4 others

ACM International Conference on Management of Data 2022

ipads.se.sjtu.edu.cn PDF doi

Reference 85 in Chapter 9: ZooKeeper: Distributed Process Coordination

Flavio P. Junqueira and Benjamin Reed

2013

oreilly.com

Reference 86 in Chapter 9: HBase and HDFS: Understanding Filesystem Usage in HBase

Enis Söztutar

HBaseCon 2013

slideshare.net

Reference 87 in Chapter 9: SUSE Linux Enterprise High Availability 15 SP6 Administration Guide, Section 12: Fencing and STONITH

SUSE LLC

2025

documentation.suse.com

Reference 88 in Chapter 9: The Chubby Lock Service for Loosely-Coupled Distributed Systems

Mike Burrows

7th USENIX Symposium on Operating System Design and Implementation 2006

research.google

Reference 89 in Chapter 9: etcd 3.4.3

Kyle Kingsbury

2020

jepsen.io

Reference 90 in Chapter 9: Distributed Locks Are Dead; Long Live Distributed Locks!

Ensar Basri Kahveci

2019

hazelcast.com

Reference 91 in Chapter 9: How to Do Distributed Locking

Martin Kleppmann

2016

martin.kleppmann.com

Reference 92 in Chapter 9: Is Redlock Safe?

Salvatore Sanfilippo

2016

antirez.com

Reference 93 in Chapter 9: Leader Election with S3 Conditional Writes

Gunnar Morling

2024

morling.dev

Reference 94 in Chapter 9: The Byzantine Generals Problem

Leslie Lamport, Robert Shostak and Marshall Pease

ACM Transactions on Programming Languages and Systems 1982

microsoft.com doi

Reference 95 in Chapter 9: Notes on Data Base Operating Systems

Jim N. Gray

Operating Systems: An Advanced Course 1978

jimgray.azurewebsites.net PDF

Reference 96 in Chapter 9: How Complicated Was the Byzantine Empire?

Brian Palmer

2011

slate.com

Reference 97 in Chapter 9: My Writings

Leslie Lamport

2014

lamport.azurewebsites.net

Reference 98 in Chapter 9: Bus Architectures for Safety-Critical Embedded Systems

John Rushby

1st International Workshop on Embedded Software 2001

csl.sri.com PDF doi

Reference 99 in Chapter 9: ELC: SpaceX Lessons Learned

Jake Edge

2013

lwn.net

Reference 100 in Chapter 9: SoK: Consensus in the Age of Blockchains

Shehar Bano, Alberto Sonnino, Mustafa Al-Bassam and 4 others

1st ACM Conference on Advances in Financial Technologies 2019

smeiklej.com PDF gone doi

Reference 101 in Chapter 9: Why We Should Train AI in Space

Ezra Feilden, Adi Oltean and Philip Johnston

2024

starcloud.com PDF

Reference 102 in Chapter 9: The Saddest Moment

James Mickens

USENIX ;login 2013

netflixtechblog.medium.com

Reference 103 in Chapter 9: Byzantine Eventual Consistency and the Fundamental Limits of Peer-to-Peer Databases

Martin Kleppmann and Heidi Howard

arXiv 2020

arxiv.org

Reference 104 in Chapter 9: Making CRDTs Byzantine Fault Tolerant

Martin Kleppmann

9th Workshop on Principles and Practice of Consistency for Distributed Data 2022

martin.kleppmann.com PDF doi

Reference 105 in Chapter 9: The Discovery of Apache ZooKeeper’s Poison Packet

Evan Gilman

pagerduty.com gone

Reference 106 in Chapter 9: When the CRC and TCP Checksum Disagree

Jonathan Stone and Craig Partridge

ACM Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication 2000

conferences2.sigcomm.org PDF doi

Reference 107 in Chapter 9: How Both TCP and Ethernet Checksums Fail

Evan Jones

2015

evanjones.ca

Reference 108 in Chapter 9: Consensus in the Presence of Partial Synchrony

Cynthia Dwork, Nancy Lynch and Larry Stockmeyer

Journal of the ACM 1988

groups.csail.mit.edu PDF doi

Reference 109 in Chapter 9: Fail-Stop Processors: An Approach to Designing Fault-Tolerant Computing Systems

Richard D. Schlichting and Fred B. Schneider

ACM Transactions on Computer Systems 1983

cs.cornell.edu PDF doi

Reference 110 in Chapter 9: Limplock: Understanding the Impact of Limpware on Scale-out Cloud Systems

Thanh Do, Mingzhe Hao, Tanakorn Leesatapornwongsa and 2 others

4th ACM Symposium on Cloud Computing 2013

ucare.cs.uchicago.edu PDF gone doi

Reference 111 in Chapter 9: Garbage Collecting Unhealthy JVMs, a Proactive Approach

Josh Snyder and Joseph Lynch

2019

Reference 112 in Chapter 9: Fail-Slow at Scale: Evidence of Hardware Performance Faults in Large Production Systems

Haryadi S. Gunawi, Riza O. Suminto, Russell Sears and 18 others

16th USENIX Conference on File and Storage Technologies 2018

Reference 113 in Chapter 9: Gray Failure: The Achilles' Heel of Cloud-Scale Systems

Peng Huang, Chuanxiong Guo, Lidong Zhou and 4 others

16th Workshop on Hot Topics in Operating Systems 2017

microsoft.com PDF doi

Reference 114 in Chapter 9: Understanding, Detecting and Localizing Partial Failures in Large System Software

Chang Lou, Peng Huang and Scott Smith

17th USENIX Symposium on Networked Systems Design and Implementation 2020

usenix.org

Reference 115 in Chapter 9: Eventual Consistency Today: Limitations, Extensions, and Beyond

Peter Bailis and Ali Ghodsi

ACM Queue 2013

Reference 116 in Chapter 9: Defining Liveness

Bowen Alpern and Fred B. Schneider

Information Processing Letters 1985

cs.cornell.edu PDF doi

Reference 117 in Chapter 9: Dude, Where’s My Metadata?

Flavio P. Junqueira

2015

fpj.me

Reference 118 in Chapter 9: January 28th Incident Report

Scott Sanders

2016

github.com

Reference 119 in Chapter 9: A Few Notes on Kafka and Jepsen

Jay Kreps

2013

blog.empathybox.com

Reference 120 in Chapter 9: Systems Correctness Practices at AWS

Marc Brooker and Ankush Desai

ACM Queue 2024

dl.acm.org doi

Reference 121 in Chapter 9: Testing Distributed Systems: Curated list of Resources on Testing Distributed Systems

Andrey Satarin

asatarin.github.io

Reference 122 in Chapter 9: We Put a Distributed Database in the Browser—And Made a Game of It!

Phil Eaton and Joran Dirk Greef

2023

tigerbeetle.com

Reference 123 in Chapter 9: FoundationDB—Simulation and Testing

Apple, Inc and the FoundationDB project authors

apple.github.io

Reference 124 in Chapter 9: Verifying Kafka Transactions—Diary Entry 2—Writing an Initial TLA+ Spec

Jack Vanlightly

2024

jack-vanlightly.com

Reference 125 in Chapter 9: From Chaos to Order—Tools and Techniques for Testing TiDB, A Distributed NewSQL Database

Siddon Tang

pingcap.com 2018

pingcap.co.jp

Reference 126 in Chapter 9: Parallel Commits: An Atomic Commit Protocol for Globally Distributed Transactions

Nathan VanBenschoten

2019

cockroachlabs.com

Reference 127 in Chapter 9: Paper: VR Revisited—State Transfer (Part 3)

Jack Vanlightly

2022

jack-vanlightly.com

Reference 128 in Chapter 9: What If the Spec Doesn't Match the Code?

Hillel Wayne

2024

buttondown.com

Reference 129 in Chapter 9: Multi-Grained Specifications for Distributed System Model Checking and Verification

Lingzhi Ouyang, Xudong Sun, Ruize Tang and 4 others

20th European Conference on Computer Systems 2025

arxiv.org doi

Reference 130 in Chapter 9: The Netflix Simian Army

Yury Izrailevsky and Ariel Tseitlin

2011

netflixtechblog.com

Reference 131 in Chapter 9: Jepsen: On the Perils of Network Partitions

Kyle Kingsbury

2013