Measure Everything
Tomorrow I join the team at InfluxDB, something I’m really excited about. I’m really looking forward to coding in Go full-time — it’s a language with real promise, a nice clean tool chain, and a very active community.
Philip O'Toole
Software engineering, distributed systems, databases, and the teams that build them
Philip O'Toole
Software engineering, distributed systems, databases, and the teams that build them
Philip O'Toole
Is node.js just a stopgap?
Something just doesn’t feel right about node.js. After coding in it for almost a year, it’s been fun, but I’ve decided it’s just a waypoint to somewhere better.
You come for the features, but you stay for the uptime
I’ve been thinking a lot recently about what makes computer services and products sticky — what makes users and customers come back again and again to what you’ve built. There are lots of ways to summarize it, but when it…
Evolving a language in and for the real world: C++ 1991-2006
Bjarne Stroustrup has great paper on his website titled Evolving a language in and for the real world: C++ 1991-2006. It provides fascinating insights on the development of the language, the challenges involved, and discusses interesting design ideas. If you…
Replicating SQLite using Raft Consensus
SQLite is a “self-contained, serverless, zero-configuration, transactional SQL database engine”. However, it doesn’t come with replication built in, so if you want to store mission-critical data in it, you better back it up. The usual approach is to continually copy…
Wow, the Go Memory Model really threw me
So far coding in Go has been fun. It comes with nice functionality that lets you know that the Go team really have been writing system software (useful stuff like this, and this). And then I read about the Go…
Writing a Syslog Collector in Go
I’ve started coding in Go (golang), and I received some advice recently from Robert Griesemer, whom I was fortunate enough to sit beside at a recent Go Meetup. To learn Go, Robert suggested that I code a solution in Go…
Call me Definitely
The creator of the network monitoring system Riemann, Kyle Kingsbury, has put together a comprehensive series of blog posts, on the fault-tolerance, high-availability, and general correctness of number of database and storage technologies. Of the technologies discussed I am most…
A Prayer for Distributed Systems Developers
Over 16 years, I’ve written software up-and-down the entire stack. Earliest in my career I wrote boot ROM software for specialized embedded devices. This kind of programming taught me so much about how computers really work.
InfluxDB and Grafana HOWTO
This blog describes working with InfluxDB 0.8. InfluxDB 0.8 is no longer supported, and has been superseded by the 1.0 release. I recently came across InfluxDB — it’s a time-series database built on LevelDB. It’s designed to support horizontal as…
What I wish I’d been told about the JVM
Java is the predominant language of Big Data technologies. HBase, Lucene, elasticsearch, Cassandra – all are written in Java and, of course, run inside a Java Virtual Machine (JVM). There are some other important Big Data technologies, while not written in…
How you should write software design documents
In my last blog post I explained why writing design documents is such a powerful approach to building well-engineered systems. But what should one document?
Why you should write software design documents
Many software engineers never write design documents. Design documentation takes time, and implementations often proceed so far without any documentation that if it happens, it’s an act of recording what has been done — a tedious task at the best…
Always thinking of the next guy
My father worked for many years in Quality Assurance at Beckman, an American medical instruments firm. His job was to ensure that newly-manufactured centrifuge rotors would hold up when spun at thousands of RPMs. He used to tell me that…
Welcome to your data
After 2 years at Loggly, tomorrow I start a new role at Jut. While I will miss the team at Loggly very much, and the wonderful product we built during my team there, I’m looking forward very much to working…
Distributed Systems for Fun and Profit
I came across a very readable paper on distributed systems — Distributed systems for fun and profit. I recommend it for anyone interested in learning more about distributed systems, and the challenges involved with designing, building, and operating distributed systems.
Book Review: Mastering ElasticSearch
Packt recently asked me to review their new publication Mastering ElasticSearch by Rafał Kuć and Marek Rogoziński. Since most of my experience with elasticsearch has been from a systems points of view — index management, cluster maintenance, indexing performance — I…
Infrastructure at Scale: Apache Kafka, Twitter Storm and elasticsearch
AWS have posted the video online of Jim Nisbet’s and my talk at AWS:reinvent 2013. In it, Jim and I describe the system we built at Loggly, which uses Apache Kafka, Twitter Storm, and elasticseach, to build a high-performance log…
Speaking at AWS re:Invent 2013
This past week I had the opportunity to speak, with my colleague Jim Nisbet, at AWS re:Invent 2013. Titled “Unmeltable Infrastructure at Scale: Using Apache Kafka, Twitter Storm, and Elastic Search on AWS“, Jim and I described the architecture of…
Avoiding elasticsearch split-brain
Loggly recently held an elasticsearch meetup, which was a great success. One question that was repeatedly asked was how to ensure elasticsearch does not suffer a partition — known as a split-brain. This can be a particular problem in AWS…
Loggly Generation 2 Released!
After 14 months of hard work, the next generation of Loggly has been released. It’s been a great time to be part of the Software Infrastructure team at Loggly and we have put together a superb log aggregation & real-time…
Technical Leadership through Testing
As technical lead at Loggly, responsibility for a well-engineered infrastructure ends with me. And one way to ensure the system is designed and implemented well is to stay as close as possible to the code, ensuring that the team and…
Using the Source
I have written another post for the Loggly blog — all about our guidelines for choosing and integrating open-source software and technology in your next project. Check it out here.
If you love your logs, set them free
I recently wrote my first post for the Loggly blog. It illustrates why host machines are often the worst place to store the logs those machines are generating. You can check it out here.
Monitoring Storm Kafka Spouts using Python
When running a large real-time processing system, monitoring is critical. But it does more than allow you to keep an eye on your system. During development it allows you test hypotheses about how it works, how it performs when certain…