About this site

Why I made this

The second edition of Designing Data-Intensive Applications cites over a thousand sources, and I kept wanting to follow them while reading on my phone. The list is published as Markdown, one file per chapter, which is fine on a laptop and hard on a phone. I wanted something I could search, and I wanted to know whether a link still worked before I tapped it.

So I built this. 1,000 of the 1,040 entries point at something you can read on the open web without paying, and 38 of the links no longer resolve. The site says which, and when it last checked.

Whose work this is

The reference list belongs to Martin Kleppmann and Chris Riccomini, the authors of Designing Data-Intensive Applications (O’Reilly Media). They maintain it at github.com/ept/ddia2-references and keep the links alive as the web changes. Each paper, book, post and talk in it belongs to its own authors and publishers, who are named in each entry. None of them are affiliated with this site.

The list is published under Creative Commons Attribution-NonCommercial 4.0, which is what lets me reproduce the citations here. It asks for attribution, which is in the footer of every page, and for non-commercial use, which is why there is no advertising, no tracking and no paywall. The code that builds the site is separate and MIT licensed.

What is hosted here

Nothing that I do not have permission to host. The site links to each work; it does not mirror it. A paper behind a paywall or a post with all rights reserved is a link and nothing more. Where a page publishes its own short description for link previews, I quote that on the reference’s page with the date I fetched it, so you can recognise the work before you follow the link.

That includes the PDFs. The crawler downloads them to check they are there and to record what they are, but they are not stored here or served from here. Of the 269 PDFs behind these references, not one declares a licence that would allow it.

The exception is a source that declares a licence permitting redistribution, such as a Creative Commons licence or a public-domain dedication. Those I republish as reading pages, with the author credited, the licence named, and the original linked as the authoritative copy. Anything without such a declaration I treat as all rights reserved.

Link checking

A crawler visits each link, records the HTTP status, content type and the page's own title and description, and commits that summary so the site builds without network access. It reads each host's robots.txt first and leaves alone whatever that asks it to. Across 1,040 references, as of the last crawl:

• 841 reachable
• 149 behind a bot block or paywall, which usually means fine in a browser
• 5 not checked, because the host's robots.txt asks crawlers to stay away
• 17 returning "not found"
• 21 failing to connect
• 7 with no link to check

26 of the linked pages declare a licence that permits redistribution, and 26 of those are republished here as reading pages, with the author credited and the original linked as the authoritative copy.

Colophon

Static HTML from a small TypeScript build, a Go crawler and a Python extractor, all in one repository shared with the other site built the same way. No framework, no web fonts, no analytics, no cookies. The type is whatever serif your device already has.

Found a broken link?

Fix it upstream, where everyone benefits: open a pull request against ept/ddia2-references. This site picks up the change when it next rebuilds.

The sibling site

The same engine runs sdi.l3.ai, which does the same job for the reference lists of System Design Interview.