Stop asking "how much data do you have?"
In every field there is a question that, while it sounds interesting, betrays a naiveté and lack of sophistication. In my field — SaaS and data platforms — it’s how much data do you have?
Philip O'Toole
Software engineering, distributed systems, databases, and the teams that build them
Philip O'Toole
Software engineering, distributed systems, databases, and the teams that build them
Tag operations
rqlite v3: Globally replicating SQLite
rqlite is an open-source distributed relational database, which uses SQLite as its storage engine. rqlite is written in Go and uses Raft to achieve consensus across a set of SQLite databases. It gracefully handles leader election, and can tolerate machine…
InfluxDB and the Raft consensus protocol
I recently presented at the InfluxDB San Francisco Meetup, on InfluxDB and the Raft consensus protocol. My talk was about the fundamental problems of distributed systems, and how InfluxDB uses Raft to solve these issues.
Designing a search system for log data — part 3
This is the last part of a 3-part series “Designing and building a search system for log data”. Be sure to check out part 1 and part 2. In the last post we examined the design and implementation of Ekanite,…
Designing a search system for log data — part 2
This is the second part of a 3-part series “Designing and building a search system for log data”. Be sure to check out part 1. Part 3 follows this post. In the previous post I outlined some of the high-level…
Designing a search system for log data — part 1
This is the first part of a 3-part series “Designing and building a search system for log data”. Part 2 is here, and part 3 is here. For the past few years, I’ve been building indexing and search systems, for…
Who watches the watchers?
I’ve written my first post for the InfluxDB blog. In it I discuss the new statistics and monitoring system built into InfluxDB, starting with the 0.9.4 release. Functionality like this is critical when it comes to running a distributed database…
400 days of Go
It’s been 418 days since my first Github commit of Go code. In that time I’ve written a Syslog-to-Kafka producer, a Raft-based distributed SQLite database, a near real-time log search system, and become a core developer of InfluxDB.
Running services is hard
I’ve recently been thinking about why running Services is particularly hard. By Services I mean Software-as-a-Service platforms. During the years, I’ve written software for many different systems — embedded software, web services, databases, and distributed systems, but being involved with…
Drop, Throttle, or Buffer
Real-time — or near real-time — data pipelines are all the rage these days. I’ve built one myself, and they are becoming key components of many SaaS platforms. SaaS Analytics, Operations, and Business Intelligence systems often involve moving large amounts…
You come for the features, but you stay for the uptime
I’ve been thinking a lot recently about what makes computer services and products sticky — what makes users and customers come back again and again to what you’ve built. There are lots of ways to summarize it, but when it…
InfluxDB and Grafana HOWTO
This blog describes working with InfluxDB 0.8. InfluxDB 0.8 is no longer supported, and has been superseded by the 1.0 release. I recently came across InfluxDB — it’s a time-series database built on LevelDB. It’s designed to support horizontal as…
What I wish I’d been told about the JVM
Java is the predominant language of Big Data technologies. HBase, Lucene, elasticsearch, Cassandra – all are written in Java and, of course, run inside a Java Virtual Machine (JVM). There are some other important Big Data technologies, while not written in…
Avoiding elasticsearch split-brain
Loggly recently held an elasticsearch meetup, which was a great success. One question that was repeatedly asked was how to ensure elasticsearch does not suffer a partition — known as a split-brain. This can be a particular problem in AWS…
If you love your logs, set them free
I recently wrote my first post for the Loggly blog. It illustrates why host machines are often the worst place to store the logs those machines are generating. You can check it out here.
Monitoring Storm Kafka Spouts using Python
When running a large real-time processing system, monitoring is critical. But it does more than allow you to keep an eye on your system. During development it allows you test hypotheses about how it works, how it performs when certain…