moving from 1&1 to Dreamhost

I decided to move my shared hosting from 1&1 to DreamHost. I had some poor experiences with 1&1:

remapping domains to subdirectories of my $HOME didn’t work at first
excessively stringent RLimitCPU meant that certain operations, like trying to migrate from Gallery 1.x to 2.x would fail and time out
trying to use 1&1’s built-in photo gallery hosed my site for a day as it remapped all the virtual host to subdirectory mappings

I hope hosting with Dreamhost will help these issues. I would really love to have my own server in a co-lo (i.e. eating my own dog food by having one in the TCCP co-lo, which I run) but I can’t justify the expense.

authenticating Apache against Windows 2003 Active Directory

Devlin’s rebuilding its intranet and moving away from the old Lotus Domino-based directory service. One of the developers on the intranet project asked me if he could authenticate employees against Active Directory instead. He’ll be using the MODx CMS, and would like to authenticate using mod_auth_ldap.

We’ve done this before to authenticate Subversion SCM users, but just as a test. This time I decided to try and create a user in Active Directory that would be used solely to bind to LDAP when doing lookups. I called this user “LDAP User”.

Making this work required a lot of trial and error, and I still haven’t managed to figure out a few things (see below). The first problem I had was that I was confused as to what the CN actually is for this particular user: it’s going to be cn=LDAP User, cn=Users, dc=devlin, dc=ca rather than cn=ldapuser, cn=Users, dc=devlin, dc=ca. ldapuser is just the login ID of the account rather than the actual CN.

The other thing I did wrong is that I put quotes around the Require statement, so rather than having

Require group “cn=Devlin Employees,cn=Users,dc=devlin,dc=ca”

the correct syntax is just

Require group cn=Devlin Employees,cn=Users,dc=devlin,dc=ca

A few things are still broken:

I can’t figure out why LDAPS isn’t working. Doing searches from the command line using ldapsearch over SSL work fine, but the configuration of LDAP-SSL within Apache seems to be really tricky. I already have the directives
```
        LDAPTrustedCA certs/sf_issuing.crt
        LDAPTrustedCAType BASE64_FILE
```
in the configuration file, and Apache does say [notice] LDAP: SSL support available, but any attempt to actually use it gives an
```
[LDAP: ldap_simple_bind_s() failed][Can't contact LDAP server]
```
error.
I’m not particularly impressed that AuthLDAPBindPassword is stored in cleartext in the configuration file, but there doesn’t seem to be a way of hashing it or otherwise concealing it.
I haven’t figured out how to enable LDAPS on Domain Controllers that aren’t already HTTPS-enabled, so for now I’m not authenticating against them.

I should just get my parents a Mac Mini

I’ve been preparing an old IBM PC 300PL for my parents to replace their generic clone that died (RIP the machine formerly known as exodus.dreaming.org). They’re familiar with Windows, so I installed Windows 2000 Professional, ran Windows Update to download all the latest patches, and installed ClamWin as a virus scanner, and ZoneAlarm as a firewall. Fortunately they don’t have Internet access yet, but I worry about them clicking on some malware link and having some nasty spyware/trojan/virus take over their machine.

Despite all this, somewhere along the line I picked up some nasty trojan. This particular strain, TROJ_CONHOOK.AE, attached itself to WINLOGON.EXE so even booting in Safe Mode wouldn’t get rid of it. It saved itself as a randomly-named DLL (in my case, C:WINNTSYSTEM32pmnmlml.dll) and added itself as an AutoRun all over the place, a fact I was able to ascertain by using SysInternals‘ excellent Autoruns utility. Using another SysInternals utility, ProcessExplorer, I was also able to see that it was causing WINLOGON.EXE to run some routine inside the DLL file every second!

Call me a skeptic but I was still not totally convinced that pmnmlml.dll was not some legitimate Windows DLL. After all, just open your C:WINNTSYSTEM32 and half the stuff in there looks like a virus. (Quick: is dcomcnfg.exe a virus? How about dcomcfg.exe?) So I decided to copy the DLL to my Linux workstation and run strings(1) on it. Sure enough, the following text string was enough confirmation for me to see that it was a trojan:

http://82.98.235.63/cgi-bin/check/autoaff3

So I followed the procedure on Trend Micro’s site for getting rid of it — namely, booting off the Windows 2000 Professional CD and running the recovery console, then deleting the DLL.

Let’s step back for a second here. I am a professional system administrator, and my parents are not. How can I expect them to surf the Internet safely and not suck down one or more of these nasty trojans? Next thing I know, I’ll be getting a call from their ISP telling me that their little IBM is sending out 10,000 spams a minute, or is the control point for some DoS botnet.

I’m leaning more and more towards just getting them a Macintosh. I just have to convince them to part with their beloved Windows.

(By the way, dcomcnfg.exe is legitimate, while dcomcfg.exe is not. But how would one ever tell?

a quick reflection upon DemoCampToronto7

This evening I went to DemoCampToronto #7, a project of BarCamp Toronto. As BarCamp’s website says,

BarCamp is an ad-hoc gathering born from the desire for people to share and learn in an open environment. It is an intense event with discussions, demos, and interaction from attendees.

DemoCamp consists of a set of presentations totally no more than 15 minutes apiece (including questions) on up-and-coming software projects. It’s basically the same as a WiP session at any USENIX conference.

I don’t have enough time to summarize all of the presentations, but I’m sure others will (and I’ll try to link to some of the better summaries here). I just wanted to step back a moment and reflect on the fact that a room full of 150 passionate, articulate coders — in Toronto, no less — makes me think that we’re having a renaissance in the software development and IT industry. These are not coders who are just buzzword and Web 2.0-compliant; I sense that these folks are making real productive use of technologies like Ruby on Rails, AJAX, DHTML, Flash, and all the other gadgets that are revolutionizing the Internet by providing a true challenge to the classic thick application.

This renaissance is borne out by the increasing proliferation of jobs. Tucows just held a job fair, after which they hired a number of individuals fresh out of Computer Science at U of T (I know because two of them were sitting at my table). Exciting companies like Nurun and Critical Mass are hiring and expanding. I’ve personally been courted by one or two companies, unsolicited. Contrast this with the state of affairs five years ago, which is when I graduated from U of T. Jobs were scarce and I was lucky to land a position programming PHP for a firm that hadn’t blown its money in the dot-com crash.

It seems to be a great time to be in IT. The buzz is in the air again, and I have but one word of warning for many of the IT firms that have just barely stayed afloat for the last few years: You’d better do something to make sure you hang onto your technical staff — i.e. give them interesting, challenging work, and respect their talents — or you will lose them to other companies that are willing to make those tools available.

getting VLANs working between Cisco & HP gear

Ever since I started at Devlin, I’ve had one nagging problem with the network gear: the VLANs from the Cisco equipment (a triad of Catalyst 3550-24 switches) won’t propagate to the other gear we have (an HP ProCurve 2424M and a Linksys SRW2024). I read all I could about VLANs and tagging, but no matter what I did I couldn’t get the non-default VLANs to show up on anything but the Cisco gear. I figured I was missing some key information, particularly about when to tag and not tag VLAN traffic, that was preventing me from getting this working.

I finally did a search on Google about Cisco interoperability, and found this page which indirectly made everything clear. It turns out that the tagging on the HP (or any other switch being connected to the topology) needs to be done as follows:

set traffic on the trunk port to be tagged for every VLAN you want to propagate
allow access to the VLAN on the non-trunk ports but set them to be untagged

I was originally a little worried because the VLAN I’m interested in propagating is the voice VLAN (for our IP telephony setup) and I feared that the Catalyst would do something really weird with it (seeing as how you specify switchport voice vlan 2), but it seems to be just another VLAN. I assume the foregoing IOS directive is just for QoS or something on the Catalyst.

By the way, doesn’t the University of Wales’ IT department have an awesome name? I know it’s Gaelic, but I should start calling my department Gwasanaethau Gwybodaeth too. That would certainly cut down on the help tickets — I could start saying “please e-mail help-gwasanaethau@devlin.ca to open a ticket” 🙂

NetworkManager starts getting some docs

Looks like someone has started putting together some informal documentation for NetworkManager.

In a completely unrelated note, the upgrading of my Fedora Core 5 Thinkpad T42 to kernel 2.6.17-1.2139 has broken wireless (again). Any attempt to use NetworkManager with it causes ipw2200: Firmware error detected. Restarting. to be seen in the dmesg. However, if I run wpa_supplicant manually and then dhclient, it works.

I’m really looking forward to the day when all this is fixed, although I suspect wireless is such a bleeding edge problem space that the day won’t be coming soon.

who’s AFRAID of real hardware RAID?

Recently we bought a low-end IBM xSeries 306m server to handle generic IT utility tasks, such as hosting an installation of Request Tracker, Cacti and, in the near future, Nagios. The server came with a pair of 160GB SATA disks attached to a ServeRAID-8e HostRAID controller. I quickly discovered that HostRAID is an awful hack; it’s not real hardware RAID, but software-emulated RAID, utilizing the host system’s SATA controller to do the actual I/O to the disks, but with the RAID processing done in software using a proprietary driver, in my case, a driver called adpahci. In other words, it’s "A Fake RAID", which some pundits have noted collapses into the fitting acronym AFRAID.

Several admins have criticized HostRAID for a number of reasons:

Performance is terrible because the AFRAID controller must do polled I/O (PIO) through the CPU
The drivers are, by nature, proprietary, since the RAID logic is licensed from a third party
Limited sophistication in array rebuilds, since the controller has a minimal BIOS and online rebuilds are not possible
Disks in an AFRAID array are probably unusable outside of the array, given that the driver is chipset-specific

Although I don’t really care about performance for such a low-end utility box, I have been seriously bitten by the second point. We use RedHat Enterprise Linux 4 Update 3 on all production servers like my utility box. IBM only provides binary HostRAID drivers up to RHEL4 Update 2. You can allegedly rebuild the drivers using a SHIM from Adaptec, but it doesn’t work; although the SHIM package contains C drivers for all the Adaptec HostRAID controllers (aar81xx, adp94xx, adpahci, adpsata, etc.) the only binary blob you can obtain is the one for the aar81xx. Ergo, I am S.O.L. I’m stuck with a RHEL 4 Update 3 userland on a RHEL 4 Update 2 kernel.

I guess the appropriate solution if you’re going to buy this model of server (with SATA) is to ditch the on-board AFRAID and buy a ServeRAID-7t SATA controller, which has a real 80302 processor and 64MB of cache memory, or any of the other ServeRAID products which fit in the server.

On a final note, what the heck is with IBM’s insane naming schemes for all of its ServeRAID products? I can’t keep the 6i+, 7t, 7k, 7e, 8e, 8i, 6M, etc. straight — can you? Have a look at this driver matrix and your eyes will glaze over. Why don’t they name the controllers something meaningful?

memories of Farallon PhoneNet

My 10-year high school reunion is happening over the August long weekend this year, and the event got me thinking about some of the technology we used during those years.

Every Ontario elementary school and high school student of a certain vintage will remember the ubiquitous Unisys ICON terminals, a topic that I will actually leave to a later entry (we had a lot of fun with those ICONs, especially upon discovering that one could write a C program fork() from any PID on the system, including /sbin/init, with extremely useful results). However, I started thinking about Farallon PhoneNet, a fabulous networking technology for Macintoshes back in the day, and I thought I should record for posterity what kind of equipment it took to produce the Mackenzie High Times back in the day.

Continue reading →

new computer woes

So my trusty 6 year-old desktop, jupiter, died after a power outage a couple of weeks ago. I suspect the motherboard got fried, because trying to power on the system did nothing, although a monitor plugged into the back of the PSU could still power up.

I’d been thinking of getting a new computer for some time, because the Pentium III 800 MHz processor and 768 MB of PC133 RAM wasn’t really cutting it for running VMWare Workstation, so this failure pushed me into action. I decided to purchase a standard "template" system from Canada Computers with the following specifications:

ASUS P5LD2-VM motherboard
Pentium 4 3.2 GHz CPU
512 MB of DDR400 PC3200 RAM
250 GB Western Digital SATA hard disk
LG 16x dual-layer DVD writer

The P5LD2-VM has onboard sound, video (using an Intel 945G chipset) and Intel Gigabit Ethernet, so I decided to just use those.

To this system I added another 512 MB of RAM, a second 250GB SATA hard disk (for a software RAID-1 mirror), and an APC Back-UPS CS 500 uninterruptible power supply.

I picked up the system on Saturday, took it home and powered it up. Immediately I saw a problem: one of the SATA hard disks wasn’t being properly detected. After fiddling around with the connections on the motherboard, I was able to get both disks to show up, but only if I used SATA ports 1 and 3, rather than 1 and 2. Plugging any device into ports 3 and 4 caused them to not show up in the BIOS.

I resolved to take the system back to have Canada Computers’ technicians diagnose the issue (eventually they reset the BIOS and everything was fine) but in the meantime I could still install Fedora Core 5 on it. Or so I thought.

I started by installing the i386 version of Fedora, which succeeded, but then I realized that the Pentium 4 is an EM64T CPU, so I should install x86_64 Fedora. Trying to do so, however, caused the installer to lock up right before the first boot, and resulted in a corrupted system — for example, /etc/inittab would be missing. I observed other weird behaviour, like the fact that the primary software RAID partition, /dev/md2, would be in a rebuilding state immediately after the install, even though the installer said to reboot the system.

I subsequently tried to install the Fedora Unity Re-Spin of x86_64 Fedora Core 5, with similar results; at least I was able to get through the installer and onto first boot, but when starting up X the system would lock up hard. SUSE Linux 10.1, which I tried just to see if it would behave differently, had the same issues.

I came to the conclusion that the on-board Intel 945G video chipset is no good, at least with the 64-bit drivers in X. So I ran out to Canada Computers again and bought the cheapest PCI Express video card I could find: an ASUS Extreme AX300SE (basically an ATI Radeon X300SE). Then I tried to reinstall Fedora Core 5, and it worked perfectly! So I would advise everyone to stay away from the Intel 945G chipset for on-board video.

By the way, the UPS was broken too — I opened a support case with APC and they are planning courier me a replacement. I guess I just have bad luck with computer equipment.

why has CORBA failed?

There’s a great article in this month’s ACM Queue entitled The Rise and Fall of CORBA. Since it’s authored by Michi Henning, who worked on CORBA as part of the OMG’s architecture board, and subsequently became an ORB implementer, consultant, and author of a book on CORBA Programming with C++, I had to take notice. The article itself isn’t available online, so I’m sorry I can’t suggest that you read it — instead you’ll just have to put up with my opinions, peppered with some quotes from the article.

Continue reading →