Skip to content


My year 2015 in review

Published by Martin Kleppmann on 28 Dec 2015.

I’ve had a pretty busy and interesting year 2015. Inspired by Caitie McCaffrey and Julia Evans, who recently wrote up reflections on their past year, I will try the same.

A lot of my work this year has been to write and speak in public about technical topics, and I am fortunate to have received a very positive response. Judging from what people have told me, both online and in person, it seems that my ideas have been useful for people in designing their software. This is very gratifying: although I find these topics interesting and useful (and hence I wanted to talk about them), that doesn’t guarantee that other people will find them useful too.

Public speaking

I gave approximately 25 talks and lectures over the course of the year (I’m losing track of how many I did exactly). Many of these were at major software engineering and data conferences in the UK, US, Germany, Belgium, Sweden and Hungary, including Strange Loop, Code Mesh, Craft, Crunch, Berlin Buzzwords, All Your Base, Øredev, Strata New York and Strata London. A few were at internal events at various organisations. See my talks archive for the list of talks, and my YouTube playlist for the selection of talk recordings that made it onto YouTube.

Fairly few of those talks were verbatim repeats – I count at least 17 distinct talks over the course of the year. Some of the talks shared material, of course (I don’t have anywhere near enough new ideas to produce 17 entirely non-overlapping talks), but I like to think that there was at least a nugget of something new and interesting in every one. I would get bored if I kept giving the same talk over and over again, and it’s not much fun to listen to a speaker who’s bored of their own talk!

Preparing all those talks and travelling to the various conferences has been quite time-consuming and tiring. Thus, even though it has been a great experience, and I have enjoyed many great conversations with people at those conferences, I will be reducing my conference activity to a more sustainable level next year.

Book writing

In 2015, we released chapters 4, 7 and 8 of my work-in-progress book, Designing Data-Intensive Applications, as early release. This means that two thirds of the planned 12 chapters are now available. However, work on chapter 9 has been painfully slow, and I am painfully aware that it has now been over 6 months since we last released a chapter.

The reason for the delay is twofold: I was too distracted by giving lots of talks, and the topic of chapter 9 (consistency and consensus) is really hard. I wanted to make sure I really understood the topic all the way to the core, with no niggling uncertanties or doubts, and so I spent a huge amount of time researching the topic, reading many papers, and challenging myself to find intuitive and correct explanations for things.

Frankly, so much nonsense has been written about the topic of distributed consistency that it has been a bit of a challenge to find good source material. But the good news is that I think I have got it figured out now, and making good headway with writing it up, so the release of chapter 9 should not be too far away. And then the following chapters will be much easier, since they will cover more straightfoward topics.

And by scaling back my speaking activity next year, I hope to get the rest of the book out quickly.

New job

After leaving LinkedIn in 2014, I took a 14-month sabbatical/funemployment to focus full-time on book writing, speaking, blogging and open source. Since October 2015, I am once again employed, namely as a postdoc on a research project at the University of Cambridge Computer Laboratory.

The fact that I got a postdoc position may be surprising, since I actually don’t have a PhD. However, the university was happy to regard my industrial experience as “equivalent to a PhD”, whatever that means. Thumbs up to them for recognising my degree from the School of Hard Knocks.

So far I’m really enjoying being in academia, and the research project I’m on is super interesting. I’ll be blogging and talking about it lots more in future, but for now I’ll just say this: my area of focus is moving away from stream processing, and towards information security. There are big and important problems to be solved at the intersection of data systems, distributed systems, infosec and cryptography. Watch this space!

Non-book writing

As a way of procrastinating from book-writing, I published three research papers in 2015, in three different areas of computer science (namely distributed systems, cryptographic protocols, and data management):

I also wrote up transcripts of several of my conference talks as oversized blog posts:

An edited and updated version of those blog posts is planned to be published as a report by O’Reilly in 2016. Complete with hand-drawn slides, of course.

Besides those transcripts, I published a few blog posts on topics other than stream processing:

Interviews

Politics

The UK government has been pursuing some disastrous information security policies. I have participated in the political process by adding a technical voice to the debate:

The debate is still ongoing, but I hope the warnings from computer scientists and software developers will be heard.

Open Source

This year I released Bottled Water, a tool for Postgres that captures all the data written to a database, and replicates it to Kafka for use in other systems (e.g. building search indexes or maintaining caches). Lots of people have started playing with Bottled Water, and I have received several good pull requests and bug reports.

Bottled Water is still a very early-stage project, but I am optimistic about it. I plan to integrate it with the new Kafka Connect framework (pull requests welcome!), which will make it easier to deploy. Similar ideas have also been popping up for other databases, for example Maxwell for MySQL.

Besides Bottled Water, I have made some small open source contributions to Apache Avro and a few other projects. My WARC library is now used in Apache Nutch. There are probably some more things that I have forgotten.

Conclusion

Another year has passed, and it’s easy to forget everything one has done. So I find it motivating to write down the major things that have happened, and see that actually it was quite a lot.

If anything, I have spread myself too thin, trying to do too many different things. Certainly my progress with the book has not been as fast as I (or my editor!) would have liked. I am honoured that so many conferences want me to speak, but unfortunately I will have to dial down my number of talks, otherwise I’ll never get the book finished.

Thank you to Confluent, who sponsored a lot of my work in the last year on a freelance basis. Thank you also to all the great people I’ve met, who have shared their ideas and helped me understand things. Happy new year!

Update 7 Jan 2016: added link to evidence I submitted to parliament, which has now bene published.