adventures in configuration management with Puppet

I’ve started investigating higher-order system configuration management tools, in particular, Puppet, in order to help manage CBC‘s web infrastructure. About two years ago, the only player in this space was cfengine, which at the time struck me as quite functional, but also quite arcane. Puppet seems to be much easier to learn. But I’ve come up against a fundamental problem that I’m hoping more experienced Puppet users can help me with. How does one force synchronous updates to Puppet clients? In this entry, I’ll explain my use case, and hopefully get some answers as to whether Puppet is the right tool for the job. Continue reading →

Quicktime caching of Windows Media payloads

A while ago I wrote a post about how the Windows Media video experience is sub-optimal on non-Windows computers — in particular, Macintoshes — and why I think this will trigger a run towards Flash on-demand and eventually Flash live. Here’s a concrete example: over the last six months to a year, and perhaps longer, we’ve been dealing with a steady stream of user complaints that the nightly newscast of CBC’s The National (insert promotional tagline about “Canada’s most trusted news source, hosted by newly-announced Order of Canada member Peter Mansbridge”) is frequently “out of date”. While I haven’t totally nailed down why this might be the case, I do note that most complainants seem to be using Quicktime to play back the stream, with Flip4Mac (ugh) as the translation layer. I believe that with so many moving parts, something is inevitably going to go wrong. Continue reading →

another one bites the dust

Today I sent another SATA hard drive back to Seagate because it failed. You might recall that I have had a bad track record with SATA drives: since purchasing this PC about two years ago, I’d gone through about three Western Digital SATA drives (all replaced under warranty due to failure) until I finally got fed up about six months ago and bought a pair of Seagate Barracuda 250 GB drives. One of them failed after three months, and the other died just this past weekend. Fortunately, my PC is RAID-1 protected (and I have all the data backed up on DLT) – but seriously, why are SATA drives so prone to failure? Continue reading →

fedora 9 upgrade

In previous entries here I have described my unhappiness with the Highpoint series of RAID controllers. In particular I owned the 1740 4-port SATA RAID controller, but dis-satisfaction with the frequency of driver updates finally caused me to dump the 1740 for another controller. (Note that even though Fedora 9 is the current release, Highpoint has still not updated their drivers beyond Fedora 7, which is almost EOL.) Continue reading →

off to Streaming Media East next week

I’m taking a long-awaited vacation next week, in part to attend my friend Kristin’s wedding down in New Jersey, but also for the Streaming Media East conference in Manhattan. My work these days requires a great deal of knowledge about video (and audio) delivery workflows for online media, and I can see many aspects of our operation ramping up in near term. Flash-based players like the Maven Networks front-end are already in use, and I can see live Flash being only six months off. It seems like Flash is suddenly on everyone’s tongue, and at least at CBC, Windows Media, while still our standard, is no longer the market darling that it once was. Continue reading →

rebuilding VoIP PBX, part two

I decided I’d rebuild my Linksys NSLU2-based PBX tonight, since SlugOS/BE 4.x had been released back in December and I wasn’t getting new versions of Asterisk pushed through the ipkg channel. Since I’ve done it before, I figured it would only take about an hour, and I was right (okay, it took ten minutes longer than I predicted, but close enough.) Continue reading →

On Webhosting: there’s The Planet and then there are the copycats… literally

A few years ago, when I was still in charge of the Toronto Community Co-Location Project (a project that I’m pretty sure is defunct by now), I was approached by a fellow named Da Shi, who was just starting a company called 3z Canada. He provided some competitive rates for co-location, but we ultimately sublet space from Chris Kirby. Continue reading →

64-bit Xen considered harmful

Recently at work, we tried to implement Xen on Intel Xeon, running a 64-bit dom0/domU. I have to say that this failed horribly, so I’m writing this post to warn others off it. My colleague Gabriel worked hard to migrate everything back to a 32-bit environment, so kudos to him.

The specific symptoms we experienced while running 64-bit Xen is that the domU’s would crash and reboot randomly under (or after) high load. One of our domU’s is a development server, which also runs a CruiseControl, a continuous integration system. This means that every minute, CruiseControl wakes up, does a cvs update to see if there are any changes, and then recompiles the project(s) if needed. Periodically we started to see error messages like
Bad pte = 32971e067, process = cvs, vm_flags = 100077, vaddr = b7f34000 [] vm_normal_page+0xb7/0xd3 [] unmap_vmas+0x3d1/0x761 [] unmap_region+0x8a/0xf0 [] do_munmap+0x148/0x19b [] sys_munmap+0x33/0x41 [] syscall_call+0x7/0xb =======================

After a few of these, domU would reboot. It seems like others are having the same problem on 64-bit Xen. This user was running CentOS 5.1, which is basically what we’re running (we have the real deal Red Hat Enterprise LInux 5.1).

As I said, migrating the domU back to a 32-bit dom0 seemed to fix this, so let this be a fair warning to others thinking of running a 64-bit dom0.

computer equipment upgrade time

It’s that time of year again when I start to think about what computer upgrades I might want to do. I’ve had some annoying things happen with my desktop PC recently and have considered either replacing it entirely, or implementing some much-needed upgrades. Continue reading →

stupid Internet memes

April Fool’s Day is upon us again, and along with this are some truly stupid April Fool’s jokes. Chief among them was YouTube’s replacement of its featured videos with ones that actually link to Rick Astley’s Never Gonna Give You Up, a common Internet meme known as "rickrolling". Now characterize me as an old codger, but I have to say that rickrolling is one of the stupidest Internet memes I have ever come across. I really don’t see what’s so funny about this. Maybe I’m missing something? It’s been done many times over, even in xkcd cartoons, but really folks, it’s not funny, and it should just be put to rest.

I’ll tell you what I found really funny: the folks over at Atlassian who announced the JIRA Solver. Some of you may recall that Cenqua (now a wholly owned subsidiary of Atlassian) presented both the PairOn extreme-programming chair and the Commentator automated code-commenter (commentator?) on 1 April 2005, both of which were a riot. Now that’s creative… perpetuating a lame Internet meme is not.