Google for system logs

I’ve been playing around with Splunk recently, which I bill as “Google for your system logs.” It’s much more than just a simple search engine, but that’s the simplest way to describe what it does; it aggregates log data from multiple sources and allows you to search, correlate data in time, and also post (anonymized) snippets from your log data on Splunk Base for others to see.

For our little shop, Splunk is probably overkill; I have about 30 servers (physical and virtual) to manage, and I have not found myself needing the functionality, per se. But it’s still a neat tool. I wish we’d had something like this at my previous job, in particular to index log4j entries from misbehaving Java applications. Trying to sift through data from six Java servers and six webservers in real-time to try and find out why the site is tanking is nearly impossible and often led to live hacks on production to disable dumb ideas that were taking the site down.

Now that I’ve posted all those HREFs, I wonder if Google will take down the site when it next indexes my journal. 🙂

MTBF for Sun drives: 4 months or less

Boy, I’m glad I wrote down directions for replacing a drive in an LVM mirror, because c0t1d0 just died on me. That’s right, the drive that I didn’t replace last time.

Keep in mind that I purchased this Sun server less than 4 months ago. I wonder if the assembly line workers at Seagate were smoking pot 6 months ago when they put this batch together?

PowerPoint, meet HTML Slidy

I can’t claim to be the first one to think that maybe PowerPoint is far too heavyweight of a tool for making presentations. In fact, Edward Tufte has said this before and in fact argues that PowerPoint makes people dumber. Having been at the receiving end of many sleep-inducing PowerPoint-driven presentations, I can heartily agree.

From a technical perspective, though, PowerPoint really is a bloated product for a task that should be fairly mundane. Into the breach steps HTML Slidy, which comprises two simple pieces: a CSS stylesheet, and a bunch of Javascript, to make a PowerPoint slide-show workalike in XHTML. Each slide is contained in a <div class="slide"> tag, and the Javascript lets you use the familiar PowerPoint navigation to get through the slideshow (forward and back arrows, clicking the slide for the next one, etc.) The only downside — and one that is easily solved with custom editor plugins for any XML editor, in my view — is that one must code all the slides by hand. Some may argue that this may be a good thing (having to hand-bomb everything forces one to think about the content more carefully).

The other interesting presentation-related product I saw recently was TeamSlide. This is a YAW2 AJAX-fronted application (a PHP program) simulating the parts of Microsoft Live Communication Server or WebEx that allow you to share a presentation with other participants. Advantage of TeamSlide: it’s only USD $99, and you don’t need any fancy browser plugins or a thick client. The disadvantage is that it’s basically a distributed slideshow application; you can’t share your desktop or other windows with the other participants, or do anything fancy that you wouldn’t be able to do outside of a PowerPoint framework.

Still, I think there is promise for the World After PowerPoint. Thanks to John Udell for writing about these technologies.

connecting a Cisco 7960 IP Phone to Asterisk

(Don’t worry, the summaries from the 2nd day of BSDCan are coming. I’m 1/4 of the way through, but I got distracted by the following toy!)

I borrowed a Cisco 7960 IP phone from work to test the feasibility of making the existing telephony infrastructure operate with Asterisk instead of Call Manager. To do this, I had to first convert the phone’s firmware from the factory SCCP to SIP. (I could have tried to use Asterisk’s Skinny support, but I figured SIP is more widely supported and I also wanted to check out Cisco’s SIP firmware)

Conceptually this proceeds as follows:

  1. Set up DHCP server to boot the phone and instruct it what TFTP server to use
  2. Set up TFTP server and load it with the SIP firmware, as well as some helper files to ease the transition (files that the SCCP firmware would normally look for to load the second-stage bootloader)
  3. Add both default (SIPDefault.cnf) and phone-specific (SIP<mac-address>.cnf) configuration files to the TFTP server to tell the phone how to register itself to Asterisk
  4. Configure sip.conf in Asterisk to reflect the same settings

Continue reading

BSDCan 2006 Day 0xFFFE: VoIP and FreeBSD tutorial

I’m going to be posting a couple of journal entries where I will summarize the talks I attended at BSDCan 2006 last weekend. It might take me a little while to get through all the summaries, but eventually I’ll finish them all! I’m starting with Day -2 of BSDCan, i.e. 2 days before BSDCan really started I attended a tutorial on VoIP and FreeBSD.

VoIP and FreeBSD Tutorial

Presented by Massimiliano Stucchi

I attended this tutorial, presented by Massimiliano (Max), who works as a consultant for a major communications company in Italy. He mentioned that in his work he manages over 2000 channels — I took this to mean either simple FXO/FXS channels (single POTS lines) or PRIs. I took this to mean about 70 customers with PRIs since PRIs in Europe can carry up to 30 b-channels (whereas in North America, on the T-1 standard, one can carry 24 b-channels).

Primarily, VoIP in a free or open-source software environment means Asterisk, which is fine; that’s what I’m interested in. Max did briefly mention that there are other open-source PBXes out there such as VOCAL (which is probably not being actively maintained given that the "latest" release is for RedHat 7.3) and Bayonne, the GNU PBX which might show some promise. Still, when one thinks of open-source PBXes, one typically thinks of Asterisk.

Much of Max’s tutorial was a very basic introduction to Asterisk, which was fine although I felt like I didn’t learn too much more about the operation of the system. He did manage to clear up some terminology for me, though; in particular I think people use the word termination wrong. In the telephony world, termination is used to describe when calls leave your VoIP system and get into the PSTN. I think a lot of people, myself included, misuse the word termination to mean the completion of a call on a DID number, whereas this is actually termed origination. Confusing, yes. I guess it’s the same hurdle in understanding that you need to jump when thinking about what signalling to use for a FXO/FXS device: for an FXO device (which actually interfaces to an FXO, i.e. a CO) you need to use FXS signalling, because your FXO device is behaving as a "station" (like a regular analog phone is a station).

Another useful tip I got out of the tutorial was that faxing does not work well with Asterisk, and definitely not with pure IP. This is in part due to a number of factors: Asterisk does not have the capability to do the error control (ECM) necessary for faxing; although ECM is, strictly speaking, optional, most Class-1 fax devices expect to have it turned on. Moreover, the GSM codec does not handle faxing very well, therefore faxes will not work on IAX or SIP channels. Max felt that the T.38 protocol (faxing over UDP) is probably the way to go, but there is not yet wide support for this.

Max eventually got into some use cases of Asterisk, showing us parts of his dialplan, and discussing some of the syntax, but I felt that during this part of the talk he got a little distracted and started telling some stories & conversing semi-privately with some of the participants. To his credit, Max had some slides prepared, but he did not stick to them, particularly near the end of the tutorial. I think this would have been discouraging to any real newbie who came into the tutorial with absolutely no knowledge of Asterisk. At one point, a newbie asked him what it would take to implement a basic Asterisk setup just for testing — what hardware should be purchased? Max wasn’t able to answer this question directly, perhaps because he had never purchased small-scale hardware. I would have preferred to see an example of a starter setup, such as the use of cheap FXO clone cards to build an answering machine, for example.

Overall, though, I got some good tips from someone who has actually built a large-scale production Asterisk setup. I won’t try running a non-POTS fax line, that’s for sure.

Silicon Graphics files for Chapter 11

Silicon Graphics, Inc. has filed for Chapter 11 bankruptcy protection. It’s a sad day for all SGI fans; many have said that SGI started going downhill when they decided to start selling Windows workstations.

I have fond memories of SGI workstations. It was on an Indy that I first surfed the World Wide Web at the Ontario Science Centre back in about 1992 or 1993; I had no idea that one of the buttons on the front was the system shutdown button, so I pressed it out of curiosity. Nothing much seemed to happen except that the front panel LED started blinking, so I happily kept using the web browser (some version of Mosaic) only to have the system shut down on me about five minutes later.

Much later in life I worked for cbc.ca and in the storage room I came across another Indy, which I was told was originally used to host the entire cbc.ca website before it was moved to a Sun E450! My colleague Blake booted up the Indy and found that it still ran, happily starting up Apache 1.3.9 and a bunch of other software with lots of remote security holes, I’m sure.

I also used SGI servers while at the University of Toronto; the Engineering Computing Facility‘s main computing server was an 8-way Origin 3200 with 2 gigabytes of memory, an unfathomable amount of RAM in 1996.

So it’s very sad to see one of the pioneers of high-powered computing be brought to its knees like this. At least if they go down in history they will be remembered for bringing superior technologies like XFS to the world.

Linux WiFi improvements on the horizon

Wireless device support — and indeed, wireless reliability — has been frankly awful in Linux up to this point. Even among the devices that work (at least some of the time), there are frequent problems. For example, my IBM Thinkpad T42 laptop comes with an IPW2200BG adapter that mostly works — except after suspend, when it will refuse to function unless the driver is unloaded and reloaded. I’m using NetworkManager to "magically" manage my network connection; when it works, it works fabulously, but there are no docs. None. I challenge you to try and find a man page or any scrap of documentation about NetworkManager anywhere on the Internet.

At least on the driver side there might be some hope on the horizon. Devicescape, a WiFi software stack specialist, has just released their "Advanced Datapath" IEEE 802.11 driver stack under an open source license, and several kernel developers are trying to get it integrated into the Linux kernel. Of course, as with all integrations, this won’t happen overnight, but when it does, many wireless features such as WPA, WEP, software MAC, and so on that currently require add-ons like the userland wpa_supplicant for WPA could be directly run by this stack.

I’m looking forward to the day when I don’t have to do this magic incantation to get wireless working after suspend:

# /etc/init.d/NetworkManager stop
# /sbin/modprobe -r ipw2200
# /sbin/modprobe ipw2200
# sleep 10
# /etc/init.d/NetworkManager start

A complete non-sequitur: survey questions that make no sense.
how quickly?

[some] questions answered!

Some of my Asterisk-related questions (below) have been answered through my own digging:

  • There is a list of Asterisk commands on the VoIP Info Wiki.
  • The format of each exten => line is exten => name,priority,application().
  • I still don’t know what all the bundled sounds say, but the extra sound files package includes a readme that indicates what all those GSM files contain. This will be leading to the following hilarious dialplan, when I get around to it:
    carried-away-monkeys.gsm Nobody is available to take your call at the moment. They have been carried away by monkeys.
    lots-o-monkeys.gsm (sound effects of many angry monkeys)
    if-u-know-ext-dial.gsm If you know the extension of the party you wish to reach, dial it now.
    company-dir-411.gsm To access our company directory, dial 411 or dial zero to leave a message in our general mailbox.

Also, it turns out that the entire O’Reilly book on Asterisk is available for download here under a Creative Commons License. Not that I don’t want to pay for the book — I will, because I believe in such things — but for now, it’ll make a good reference.