My year 2015 in review
Published by Martin Kleppmann on 28 Dec 2015.
I’ve had a pretty busy and interesting year 2015. Inspired by
Caitie McCaffrey and
who recently wrote up reflections on their past year, I will try the same.
A lot of my work this year has been to write and speak in public about technical topics, and I am
fortunate to have received a very positive response. Judging from what people have told me, both
online and in person, it seems that my ideas have been useful for people in designing their
software. This is very gratifying: although I find these topics interesting and useful (and hence
I wanted to talk about them), that doesn’t guarantee that other people will find them useful too.
I gave approximately 25 talks and lectures over the course of the year (I’m losing track of how many
I did exactly). Many of these were at major software engineering and data conferences in the UK, US,
Germany, Belgium, Sweden and Hungary, including
All Your Base,
Strata New York and
Strata London. A few were at internal events at various
organisations. See my talks archive for the list of talks, and my
YouTube playlist for
the selection of talk recordings that made it onto YouTube.
Fairly few of those talks were verbatim repeats – I count at least 17 distinct talks over the
course of the year. Some of the talks shared material, of course (I don’t have anywhere near enough
new ideas to produce 17 entirely non-overlapping talks), but I like to think that there was at least
a nugget of something new and interesting in every one. I would get bored if I kept giving the same
talk over and over again, and it’s not much fun to listen to a speaker who’s bored of their own
Preparing all those talks and travelling to the various conferences has been quite time-consuming
and tiring. Thus, even though it has been a great experience, and I have enjoyed many great
conversations with people at those conferences, I will be reducing my conference activity to a more
sustainable level next year.
In 2015, we released chapters 4, 7 and 8 of my work-in-progress book,
Designing Data-Intensive Applications, as early release. This means
that two thirds of the planned 12 chapters are now available. However, work on chapter 9 has been
painfully slow, and I am painfully aware that it has now been over 6 months since we last released
The reason for the delay is twofold: I was too distracted by giving lots of talks, and the topic of
chapter 9 (consistency and consensus) is really hard. I wanted to make sure I really understood the
topic all the way to the core, with no niggling uncertanties or doubts, and so I spent a huge amount
of time researching the topic, reading many papers, and challenging myself to find intuitive and
correct explanations for things.
Frankly, so much nonsense has been written about the topic of distributed consistency that it has
been a bit of a challenge to find good source material. But the good news is that I think I have got
it figured out now, and making good headway with writing it up, so the release of chapter 9 should
not be too far away. And then the following chapters will be much easier, since they will cover more
And by scaling back my speaking activity next year, I hope to get the rest of the book out quickly.
After leaving LinkedIn in 2014, I took a 14-month sabbatical/funemployment to focus full-time on
book writing, speaking, blogging and open source. Since October 2015, I am once again employed,
namely as a postdoc on a research project at the
University of Cambridge Computer Laboratory.
The fact that I got a postdoc position may be surprising, since I actually don’t have a PhD.
However, the university was happy to regard my industrial experience as “equivalent to a PhD”,
whatever that means. Thumbs up to them for recognising my degree from the School of Hard Knocks.
So far I’m really enjoying being in academia, and the research project I’m on is super interesting.
I’ll be blogging and talking about it lots more in future, but for now I’ll just say this: my area
of focus is moving away from stream processing, and towards information security. There are big and
important problems to be solved at the intersection of data systems, distributed systems, infosec
and cryptography. Watch this space!
As a way of procrastinating from book-writing, I published three research papers in 2015, in three
different areas of computer science (namely distributed systems, cryptographic protocols, and data
I also wrote up transcripts of several of my conference talks as oversized blog posts:
An edited and updated version of those blog posts is planned to be published as a report by O’Reilly
in 2016. Complete with hand-drawn slides, of course.
Besides those transcripts, I published a few blog posts on topics other than stream processing:
The UK government has been pursuing some disastrous information security policies. I have
participated in the political process by adding a technical voice to the debate:
The debate is still ongoing, but I hope the warnings from computer scientists and software
developers will be heard.
This year I released Bottled Water, a tool for
Postgres that captures all the data written to a database, and replicates it to Kafka for use in
other systems (e.g. building search indexes or maintaining caches). Lots of people have started
playing with Bottled Water, and I have received several good pull requests and bug reports.
Bottled Water is still a very early-stage project, but I am optimistic about it. I plan to integrate
it with the new Kafka Connect
framework (pull requests welcome!), which will make it easier to deploy. Similar ideas have also been
popping up for other databases, for example
Maxwell for MySQL.
Besides Bottled Water, I have made some small open source contributions to
Apache Avro and a few other projects. My
WARC library is now
used in Apache Nutch.
There are probably some more things that I have forgotten.
Another year has passed, and it’s easy to forget everything one has done. So I find it motivating
to write down the major things that have happened, and see that actually it was quite a lot.
If anything, I have spread myself too thin, trying to do too many different things. Certainly my
progress with the book has not been as fast as I (or my editor!) would have liked. I am honoured
that so many conferences want me to speak, but unfortunately I will have to dial down my number of
talks, otherwise I’ll never get the book finished.
Thank you to Confluent, who sponsored a lot of my work in the last year on
a freelance basis. Thank you also to all the great people I’ve met, who have shared their ideas and
helped me understand things. Happy new year!
Update 7 Jan 2016: added link to
evidence I submitted to parliament, which has now bene published.