Packt Publishing have a released a new book, Implementing Cloud Design Patterns for AWS, for which I acted as an official technical reviewer.
Continue reading “Reviewing Implementing Cloud Design Patterns for AWS”
Philip O'Toole
Software engineering, distributed systems, databases, and the teams that build them
Philip O'Toole
Software engineering, distributed systems, databases, and the teams that build them
Increasing bleve indexing performance with sharding
Search is everywhere. Once you’ve built search systems, you see its potential application in many places. So when I came across bleve, an open-source search library written in Go, I was interested in learning more about its feature set and its indexing performance. And I could see immediately one might be able to shard it to improve performance.
Continue reading “Increasing bleve indexing performance with sharding”
Reviewing Elasticsearch Cookbook
I recently acted as one of the official technical reviewers for ElasticSearch Cookbook – Second Edition by Alberto Paro. Published by Packt Publishing, the book contains a large number of “recipes” for elasticsearch.
Code reviews still rule
Recently at InfluxDB we discussed how code reviews fit in during the various stages of development. It’s great to see the team reach consensus about how we should develop software. It made me think more deeply about why I remain a big believer in the code review process.
History of Software Engineering
I recently came across a talk on YouTube titled History of Software Engineering, given by Paolo Perrotta. Normally I find online videos to have a low information-to-time ratio, but this one was excellent. It’s not too long, with plenty of humour, and makes many serious points that resonated with me.
Book Review: Cassandra High Availability
Packt recently asked me to review their new publication Cassandra High Availability, written by Robbie Strickland.
I’ve worked with Cassandra in the past — early designs of Loggly‘s 2nd generation Log analytics platform used Cassandra as its authoritative store for log data, but we ended up pulling it and using elasticsearch as both the store and search engine.
Software Development for Infrastructure
Bjarne Stroustrup
has another very interesting paper on his website. Titled Software Development for Infrastructure, it discusses some key ideas for building software that has “…more stringent correctness, reliability, efficiency, and maintainability requirements than non-essential applications.” It is not a long paper, but offers useful observations and guidelines for building such software systems.
Eudyptula Challenge
The Eudyptula Challenge is a series of programming tasks, with the goal of getting one up-to-speed on Linux kernel programming. When I first heard about it, it immediately intrigued me. I’ve written a few production Linux kernel modules in my time — mostly device drivers — so I started the challenge today.
Drop, Throttle, or Buffer
Real-time — or near real-time — data pipelines are all the rage these days. I’ve built one myself, and they are becoming key components of many SaaS platforms. SaaS Analytics, Operations, and Business Intelligence systems often involve moving large amounts of data, received over the public Internet, into complex backend systems. And managing the incoming flow of data to these pipelines is key.
Measure Everything
Tomorrow I join the team at InfluxDB, something I’m really excited about. I’m really looking forward to coding in Go full-time — it’s a language with real promise, a nice clean tool chain, and a very active community.
Continue reading “Measure Everything”
Is node.js just a stopgap?
Something just doesn’t feel right about node.js.
After coding in it for almost a year, it’s been fun, but I’ve decided it’s just a waypoint to somewhere better.
You come for the features, but you stay for the uptime
I’ve been thinking a lot recently about what makes computer services and products sticky — what makes users and customers come back again and again to what you’ve built. There are lots of ways to summarize it, but when it comes to systems that help technical people run their own systems, they come for the features, but they stay for the uptime.
Continue reading “You come for the features, but you stay for the uptime”
Evolving a language in and for the real world: C++ 1991-2006
Bjarne Stroustrup has great paper on his website titled Evolving a language in and for the real world: C++ 1991-2006. It provides fascinating insights on the development of the language, the challenges involved, and discusses interesting design ideas. If you have even a basic understanding of C++, it’s a such a worthwhile read.
Continue reading “Evolving a language in and for the real world: C++ 1991-2006”
Replicating SQLite using Raft Consensus
SQLite is a “self-contained, serverless, zero-configuration, transactional SQL database engine”. However, it doesn’t come with replication built in, so if you want to store mission-critical data in it, you better back it up. The usual approach is to continually copy the SQLite file on every change.
I wanted SQLite, I wanted it distributed, and I really wanted a more elegant solution for replication. So rqlite was born.
Wow, the Go Memory Model really threw me
So far coding in Go has been fun. It comes with nice functionality that lets you know that the Go team really have been writing system software (useful stuff like this, and this). And then I read about the Go Memory Model, and had my consciousness raised.
Writing a Syslog Collector in Go
I’ve started coding in Go (golang), and I received some advice recently from Robert Griesemer, whom I was fortunate enough to sit beside at a recent Go Meetup. To learn Go, Robert suggested that I code a solution in Go for a problem I had previously solved in a different language.
Call me Definitely
The creator of the network monitoring system Riemann, Kyle Kingsbury, has put together a comprehensive series of blog posts, on the fault-tolerance, high-availability, and general correctness of number of database and storage technologies. Of the technologies discussed I am most familiar with — elasticsearch and Apache Kafka — I found the posts to be a great read.
If you haven’t read them yet, you should check them out on his site.
A Prayer for Distributed Systems Developers
Over 16 years, I’ve written software up-and-down the entire stack. Earliest in my career I wrote boot ROM software for specialized embedded devices. This kind of programming taught me so much about how computers really work.
Continue reading “A Prayer for Distributed Systems Developers”
InfluxDB and Grafana HOWTO
This blog describes working with InfluxDB 0.8. InfluxDB 0.8 is no longer supported, and has been superseded by the 1.0 release.
I recently came across InfluxDB — it’s a time-series database built on LevelDB. It’s designed to support horizontal as well as vertical scaling and, best of all, it’s not written in Java — it’s written in Go. I was intrigued to say the least.
What I wish I’d been told about the JVM
Java is the predominant language of Big Data technologies. HBase, Lucene, elasticsearch, Cassandra – all are written in Java and, of course, run inside a Java Virtual Machine (JVM). There are some other important Big Data technologies, while not written in Java, also run inside a JVM.
Examples include Apache Storm, which is written in Clojure, and Apache Kafka, which is written in Scala. This makes basic knowledge of the JVM quite important when it comes to deploying and operating Big Data technologies.
How you should write software design documents
In my last blog post I explained why writing design documents is such a powerful approach to building well-engineered systems. But what should one document?
Continue reading “How you should write software design documents”
Why you should write software design documents
Many software engineers never write design documents. Design documentation takes time, and implementations often proceed so far without any documentation that if it happens, it’s an act of recording what has been done — a tedious task at the best times.
Many software engineers argue “the code exists, it’s running, it’s working, let’s move on and build the next thing.”
Continue reading “Why you should write software design documents”
Always thinking of the next guy
My father worked for many years in Quality Assurance at Beckman, an American medical instruments firm. His job was to ensure that newly-manufactured centrifuge rotors would hold up when spun at thousands of RPMs. He used to tell me that the Beckman philosophy could be summarised in one sentence — “There is no substitute for quality”.
Welcome to your data
After 2 years at Loggly, tomorrow I start a new role at Jut. While I will miss the team at Loggly very much, and the wonderful product we built during my team there, I’m looking forward very much to working again with some old colleagues from Riverbed Technology.
Distributed Systems for Fun and Profit
I came across a very readable paper on distributed systems — Distributed systems for fun and profit. I recommend it for anyone interested in learning more about distributed systems, and the challenges involved with designing, building, and operating distributed systems.