My year 2015 in review

Published by Martin Kleppmann on 28 Dec 2015.

I’ve had a pretty busy and interesting year 2015. Inspired by Caitie McCaffrey and Julia Evans, who recently wrote up reflections on their past year, I will try the same.

A lot of my work this year has been to write and speak in public about technical topics, and I am fortunate to have received a very positive response. Judging from what people have told me, both online and in person, it seems that my ideas have been useful for people in designing their software. This is very gratifying: although I find these topics interesting and useful (and hence I wanted to talk about them), that doesn’t guarantee that other people will find them useful too.

Public speaking

I gave approximately 25 talks and lectures over the course of the year (I’m losing track of how many I did exactly). Many of these were at major software engineering and data conferences in the UK, US, Germany, Belgium, Sweden and Hungary, including Strange Loop, Code Mesh, Craft, Crunch, Berlin Buzzwords, All Your Base, Øredev, Strata New York and Strata London. A few were at internal events at various organisations. See my talks archive for the list of talks, and my YouTube playlist for the selection of talk recordings that made it onto YouTube.

Fairly few of those talks were verbatim repeats – I count at least 17 distinct talks over the course of the year. Some of the talks shared material, of course (I don’t have anywhere near enough new ideas to produce 17 entirely non-overlapping talks), but I like to think that there was at least a nugget of something new and interesting in every one. I would get bored if I kept giving the same talk over and over again, and it’s not much fun to listen to a speaker who’s bored of their own talk!

Preparing all those talks and travelling to the various conferences has been quite time-consuming and tiring. Thus, even though it has been a great experience, and I have enjoyed many great conversations with people at those conferences, I will be reducing my conference activity to a more sustainable level next year.

Book writing

In 2015, we released chapters 4, 7 and 8 of my work-in-progress book, Designing Data-Intensive Applications, as early release. This means that two thirds of the planned 12 chapters are now available. However, work on chapter 9 has been painfully slow, and I am painfully aware that it has now been over 6 months since we last released a chapter.

The reason for the delay is twofold: I was too distracted by giving lots of talks, and the topic of chapter 9 (consistency and consensus) is really hard. I wanted to make sure I really understood the topic all the way to the core, with no niggling uncertanties or doubts, and so I spent a huge amount of time researching the topic, reading many papers, and challenging myself to find intuitive and correct explanations for things.

Frankly, so much nonsense has been written about the topic of distributed consistency that it has been a bit of a challenge to find good source material. But the good news is that I think I have got it figured out now, and making good headway with writing it up, so the release of chapter 9 should not be too far away. And then the following chapters will be much easier, since they will cover more straightfoward topics.

And by scaling back my speaking activity next year, I hope to get the rest of the book out quickly.

New job

After leaving LinkedIn in 2014, I took a 14-month sabbatical/funemployment to focus full-time on book writing, speaking, blogging and open source. Since October 2015, I am once again employed, namely as a postdoc on a research project at the University of Cambridge Computer Laboratory.

The fact that I got a postdoc position may be surprising, since I actually don’t have a PhD. However, the university was happy to regard my industrial experience as “equivalent to a PhD”, whatever that means. Thumbs up to them for recognising my degree from the School of Hard Knocks.

So far I’m really enjoying being in academia, and the research project I’m on is super interesting. I’ll be blogging and talking about it lots more in future, but for now I’ll just say this: my area of focus is moving away from stream processing, and towards information security. There are big and important problems to be solved at the intersection of data systems, distributed systems, infosec and cryptography. Watch this space!

Non-book writing

As a way of procrastinating from book-writing, I published three research papers in 2015, in three different areas of computer science (namely distributed systems, cryptographic protocols, and data management):

• “A Critique of the CAP Theorem” explains in detail why I think the CAP theorem is terrible, and why we should all stop quoting it. In this paper I catalogue various common misunderstandings and ambiguities, prove a more precisely formulated theorem, and suggest a saner alternative to CAP.
• “Strengthening public key authentication against key theft” (with Conrad Irwin) describes a cryptographic authentication protocol that is somewhat robust against someone stealing your private key. We presented this paper at the 9th International Conference on Passwords.
• “Kafka, Samza and the Unix philosophy of distributed data” (with Jay Kreps) describes the design philosophy of Apache Kafka and stream processing framework Apache Samza by analogy to Unix pipes. This article is a refined version of my prior blog post on the topic. It was invited to appear in the December 2015 issue of the IEEE Data Engineering Bulletin.

I also wrote up transcripts of several of my conference talks as oversized blog posts:

• Stream processing, Event sourcing, Reactive, CEP… and making sense of it all
• Turning the database inside-out with Apache Samza
• Real-time full-text search with Luwak and Samza
• Bottled Water: Real-time integration of PostgreSQL and Kafka
• Using logs to build a solid data infrastructure (or: why dual writes are a bad idea)
• Kafka, Samza, and the Unix philosophy of distributed data

An edited and updated version of those blog posts is planned to be published as a report by O’Reilly in 2016. Complete with hand-drawn slides, of course.

Besides those transcripts, I published a few blog posts on topics other than stream processing:

• Please stop calling databases CP or AP was a precursor to my CAP theorem paper. It was a bit contentious, but I stand by my point that the CP/AP categorisation is more confusing than helpful.
• The Recurse Center and the joy of learning describes my experience of spending two weeks as a resident at the Recurse Center, an educational retreat for programmers in NYC.
• The Investigatory Powers Bill would increase cybercrime – see politics section below. Apologies for the “cyber”, but I decided it was important to speak the language of my target audience.

Interviews

• Werner Schuster interviewed me for InfoQ about my talk at Craft, and about the future of data systems design.
• Jim Brikman interviewed me for his book “Hello, Startup” regarding the pros and cons of startup life.
• PwC Technology Forecast used some quotes from me in their report on “Remapping the database landscape: the rise of immutable data stores”.
• The September 2015 issue of Linux Journal republished some of my older work on SSH private key encryption.

Politics

The UK government has been pursuing some disastrous information security policies. I have participated in the political process by adding a technical voice to the debate:

• Early in 2015, I warned members of the House of Lords about the draft Communications Data Bill (“Snoopers’ Charter”), which was eventually withdrawn. However, an even worse proposal called the Investigatory Powers Bill was introduced in November 2015, about which I contacted my MP.
• I wrote a blog post explaining why you should be worried about the Investigatory Powers Bill even if you don’t care about privacy, and you are happy with government services scanning all your communication. In a nutshell: the bill mandates backdoors and security holes in software. It is a bad idea to make systems insecure by design.
• I submitted written evidence to the House of Commons Science and Technology Committee, explaining the problems I see with the draft Investigatory Powers Bill.
• I also submitted similar written evidence to the House of Commons and House of Lords joint committee on the draft Investigatory Powers Bill.

The debate is still ongoing, but I hope the warnings from computer scientists and software developers will be heard.

Open Source

This year I released Bottled Water, a tool for Postgres that captures all the data written to a database, and replicates it to Kafka for use in other systems (e.g. building search indexes or maintaining caches). Lots of people have started playing with Bottled Water, and I have received several good pull requests and bug reports.

Bottled Water is still a very early-stage project, but I am optimistic about it. I plan to integrate it with the new Kafka Connect framework (pull requests welcome!), which will make it easier to deploy. Similar ideas have also been popping up for other databases, for example Maxwell for MySQL.

Besides Bottled Water, I have made some small open source contributions to Apache Avro and a few other projects. My WARC library is now used in Apache Nutch. There are probably some more things that I have forgotten.

Conclusion

Another year has passed, and it’s easy to forget everything one has done. So I find it motivating to write down the major things that have happened, and see that actually it was quite a lot.

If anything, I have spread myself too thin, trying to do too many different things. Certainly my progress with the book has not been as fast as I (or my editor!) would have liked. I am honoured that so many conferences want me to speak, but unfortunately I will have to dial down my number of talks, otherwise I’ll never get the book finished.

Thank you to Confluent, who sponsored a lot of my work in the last year on a freelance basis. Thank you also to all the great people I’ve met, who have shared their ideas and helped me understand things. Happy new year!

Update 7 Jan 2016: added link to evidence I submitted to parliament, which has now been published.