I recently wrote my first post for the Loggly blog. It illustrates why host machines are often the worst place to store the logs those machines are generating.
You can check it out here.
Philip O'Toole
Software engineering, distributed systems, databases, and the teams that build them
Philip O'Toole
Software engineering, distributed systems, databases, and the teams that build them
Monitoring Storm Kafka Spouts using Python
When running a large real-time processing system, monitoring is critical. But it does more than allow you to keep an eye on your system. During development it allows you test hypotheses about how it works, how it performs when certain parameters are changed, and takes the guessing out of working with dynamic systems.
Storm, a real-time computational framework open-sourced by Twitter, is such a system and comes with a Spout, allowing messages to be streamed from a Kafka Broker.
Continue reading “Monitoring Storm Kafka Spouts using Python”
Boost ASIO timers — errors are never enough
The Boost ASIO Library is a wonderful piece of software. I’ve built high-performance event-driven IO C++ programs that just scream — it works very well. However, there is one subtlety when it comes to timers — specifically when it comes to cancelling expired timers.
Continue reading “Boost ASIO timers — errors are never enough”
Bootstrapping Cassandra
Cassandra is an open-source, distributed database, informally known as a NoSQL database. It is designed to store large amounts of data, offer high-write performance, and provide fault-tolerance. I recently needed some hands-on experience with Cassandra, and being relatively new to Java programming, needed a simple set-up with which I would experiment.
Generating Type-1 UUIDs using C++
I needed some C++ code to generate Type-1 time-based UUIDs. The Boost libraries, while offering support for other types, don’t have support for time-based UUIDs.
A cut of my code can be found in github.
mutt and Google Mail
I finally moved to mutt for my Loggly e-mail (which runs on Google Mail). After moving from e-mail client to e-mail client, I was keen to give it a try — the minimalist design and speed really appealed.
It took a little while to get it just right, but it’s up and running now. I’m pretty happy with it so far, and might consider using it for my personal Yahoo! Mail.
You can find my .muttrc file here.
New Challenges
After almost 5 years at Riverbed Technology, it’s time for new challenges. I’ve started a new development position at Loggly in San Francisco, helping to build their Cloud-based Logging-as-a-Service platform.
I spent significant time at building systems that needed comprehensive logging support. But it’s something that developers don’t need to worry about — let others do it for you.
Why not check out Loggly for your logging and monitoring needs? And if you like building scalable, distributed, software systems, why not join us?
Eating your own dogfood
Dogfood testing is an effective way to increase testing, and get valuable feedback, on one’s products. It can be especially effective in the earlier stages of a product’s development, when the user base can be small. Having a forgiving — and sometimes captive — audience provides very useful feedback.
I just wrote a post for the Riverbed Blog about Dogfood testing during development of the Riverbed Cloud Portal. You can check it out here.
Kubuntu 11.04 on the Chembook 2370VA
My experience with The old ideas are still the best
Another post for the Riverbed Technology Blog, on the value of looking back.
You can read it here.
A simple host system for a 6502 emulator
CPU emulation, particularly of older processors, is an interesting topic.
While emulation source code for various CPU cores is easily available, I wanted to better understand how to interface the emulated CPU with my host machine. Therefore I decided to write a simple example of a host system for an emulated MOS Technology 6502 microprocessor.
The goal would be to have the emulated 6502 write “Hello, world” to the console of my linux desktop machine.
Fedora Core 15 on the Chembook 2370VA
It’s been almost 18 months since I installed Fedora Core 12, so I decided to move onto Fedora 15.
While the install went OK, I am not (yet) convinced it was worth the hassle. While it’s nice to pick up bug fixes, I haven’t noticed much change to the feature set.
Gaining Development Perspective from Field Trials
Another post, written by me, for the Riverbed Technology blog — this time about the value of Alpha and Beta testing.
Software developers, such as myself, can get very involved with a single part of the new software on which we are working, sometimes losing perspective. Helping customers deploy and test pre-release software helps us to design better products when we are reminded we need to create a solution, not just a box running some clever software.
You can read my latest post here.
The Power of the Cloud
I recently wrote a entry for the Riverbed Technology blog, describing an interesting collaborative development experience I had with the AWS EC2 Cloud.
You can read it here.
My guidelines for reusable Django applications
I came to Django development from much lower-level development — embedded software, device drivers, and system software. What has impressed me most about Django (and python in general) is the manner in which it guides you to do the right thing in terms of code construction. The framework and language naturally make you think about better ways to express your designs.
Continue reading “My guidelines for reusable Django applications”
gdb, inline source, and stepping through your code
I really like having inline source when using gdb. Code Complete, by Steve Mcconnell has an entire chapter explaining how you should proactively step through all code you write — and not just when you’re actively debugging an issue. Having followed this practice for a few years now, I can testify that it increases your productivity enormously. I simply can’t imagine not doing so before committing any code.
Continue reading “gdb, inline source, and stepping through your code”
Do you use expect? You should
expect is a tool built using Tcl which allows you to automate many tasks that would otherwise mean tedious repetition at the command line. While many tools come with a command line interface, they don’t lend themselves well to scripting — telnet is the classic example. But with expect you can script these tools as easily as bash.
Installing Fedora Core 12 on Chembook 2370VA
After a good experience with Fedora Core 8, and a reasonable experience with Fedora Core 11, I decided to install Fedora Core 12 on my Chembook laptop.
In summary, while FC8 and FC11 worked out of the box, FC12 failed to provide me with sound. I discovered later that the KDE mixer had set the center volume to mute. Once I unset that, I had sound.
Other than that, it worked pretty well.
Continue reading “Installing Fedora Core 12 on Chembook 2370VA”
Valgrind, blessed be its name
Valgrind
comprises a bunch of very useful tools for detecting problems with your programs. I first came across it a couple of years back and find it to be excellent. In particular I use its memory profiler, which helps you catch errors such as memory leaks and invalid accesses.
My git workflow
I cannot praise the revision-control tool git highly enough, and often use it as a buffer between SVN and me. Much of my professional work flow involves fixing a bug here, fixing a bug there — lots and lots of small changes in many different branches. git is the perfect tool for this kind of work. And it is fast.
php and EXIF data
A colleague at work gave me the idea of storing metadata for each photo on my brother’s site inside its EXIF data. I liked this idea as I originally thought I would need another text file on disk, which described the photos. Tying data to an object by adding it to the object itself is also much more robust.
Yahoo! Mail bans apostrophes
I am a Yahoo! Mail Plus subscriber and recently noticed that my outbound e-mail rewrote my name as Philip O'Toole.
Seemingly it had been like this for weeks, but it only caught my attention when I sent an e-mail to my work account. Of course, if you’re a programmer it’s pretty obvious what is going on here.
Yellow Dog Linux 6.1 on the PS3
I got around to installing Yellow Dog 6.1 using a DVD of the full distro. The installation went OK, and the installer fired up in graphical mode. However it proceeded to create the swap partition almost immediately because of low-memory concerns.
When it completed YDL was quite zippy – a much, much better experience than I got from FC12. I even had audio.
I may actually use this – it depends if I can get particular media players running on it.
Fedora 12 on the PS3
In between bouts of Wipeout HD, I net-installed english-language 64-bit PowerPC Fedora Core 12 on my 80GB PS3.
Installation with PetitBoot didn’t present any problems, though audio didn’t seem to work. However FC12 is quite slow on my PS3, so I ain’t going to use it – it seems it’s paging to disk a lot.