the Zen of Communication.

It may just be a USB drive…

I was working on a client’s Small Business Server today. It had been some time since they’d had updates run, so I installed Server 2003 SP1, SBS 2003 SP1, then rebooted to prepare for Server 2003 SP2. After I’d rebooted, the server posted, loaded Server 2k3, and just before the expected ‘Preparing network connections’ I got a message that ‘The Active Directory is Rebuilding Indices’. Assuming it was something to do with the Service Packs I’d just loaded, I let it sit for a short while. Well, 20 minutes later it hadn’t made any progress, the disks were still idle, and my client was understandably wondering how much more time they’d be billed for this screen saver equivalent.

Rather than hardboot the server while it was still loading, I began to research exactly what the problem could be. I found several posts about the message having to do with VSS, others explaining that pulling the power cable to shut down the server isn’t the most advantageous method (nice), others offering that it may be the AD catalog has gone belly up and would need restoring (makes sense, but I really hoped not), and some suggesting that RAID hardware may be failing – either the controller or a drive in the array.

Well, this server was on a RAID array, which made my pulse quicken. But I’d seen no other signs of failed hard drives – the controller booted fine, no errors with the disks, chkdsk wasn’t offering itself up for sacrifice upon every boot. Didn’t seem like a failing array.

This post made mention of a corrupted USB driver, and how the poster’s server rebooted fine when there wasn’t a USB hard drive. I instantly thought back to the time I gave myself a heart attack when a clients server displayed ‘OS Not Found’ after I tried to reboot it, inadvertantly with a USB hard drive attached. (I turned off Booting from a USB Device shortly after that).

Well, I looked at the server I was working on, and sure enough there was a USB hard drive attached that my client used for backing up data. The disks weren’t being read (either the RAID or the USB) while it was stalled, so I thought what the hell. I turned off the power to the USB drive, disconnected the USB cable, and within seconds the server moved on to ‘Preparing Network Connections’.

So in conclusion, the ‘Active Directory is Rebuilding Indices’ message during boot may be indicative of a stalled drive or accessory. Check for a USB peripheral, an external SCSI disk drive, or maybe something using firewire. If you have any such devices, try turning them off (little bit tougher with external SCSI drives, but feasible if there’s nothing critical), and then try rebooting. I didn’t run into the message again, and it saved me a complete AD restore from tape, let alone uninstalling and reinstalling service packs and hotfixes.

I recently had a need to manually create some 400 AD-integrated stub zones for a client (~40 zones on 11 domains, and it was ten to midnight on a friday night). Because each domain was in a separate forest, AD integrated replication between forests and domains wasn’t an option, and secondary transfers would occupy too much bandwidth between sites, nor would they be as promptly updated as would be required. So I had two choices: 1) Create each record, specify the master servers, and configure replication, one by one, on each domain, all by hand (I think not). Or 2) Find some way to script it.

Off to Google I go.

Several searches for DNS scripting, server 2003 DNS command line, etc, later, and I’d been able to find references to dnscmd.exe (part of the Server Resource Kit – available from Microsoft here). There were several guides about the syntax, but it took several tries to get the CLI to work as I needed to. I figured I’d put this up to shave some time off the next poor sod’s research.

To create a record, open the Support Tools CLI. From the CLI, this string is the basic format:

dnscmd . /zoneadd domain1.com /DsStub 192.168.5.1 192.168.5.2 /DP /forest

This will open the local server you’re working on, create a new zone called domain1.com, store it as a stub in AD (/DsStub), use 192.168.5.1 and 192.168.5.2 as the master servers, put the zone in the directory partition of AD (the /DP switch), and set the zone to replicate to all DNS servers in the forest (the /forest switch). You can substitute another hostname for the “.” if you want to add records to another server, so “dnscmd dnsserver1 /zoneadd …”

That’s the basic CLI for creating a forward lookup zone. But I was actually interested in making reverse lookup zones, which has one more little trick to it. You have to put in the full in-addr.arpa addy of the reverse zone (you can see these in the DNS MMC console by right-clicking on your reverse zones, selecting ‘View’, and choosing ‘Advanced’). This made my string look like:

dnscmd . /zoneadd 5.168.192.in-addr.arpa /DsStub 192.168.5.1 192.168.5.2 /DP /forest

So this command is opening the local server (the ‘.’), creating a new zone (/zoneadd) that’s a reverse zone of 192.168.5.x (5.168.192.in-addr.arpa), specifying a Stub zone stored in AD (/DsStub) with master servers of 192.168.5.1 and 192.168.5.2, stored in the Directory Partition (/DP), with replication across all DNS servers in the forest (/forest).

After perfecting the syntax, I edited a text file with the appropriate IP addresses of the zones I was creating, which lead to a batch file with some 40 or 50 instances of dnscmd being called, each with different zones and IPs. I’m sure there’s some very clever way to run a perl script that calls dnscmd and pulls variables from a csv or text file, but that’s way beyond my abilities here and now.

So to point you in some other directions, there are many resources out there which cover dnscmd and its many capabilities – that’s beyond this post. Petri.co.il has a page here, Microsoft’s official TechNet page is here, and Google has many others. It’s a very powerful tool that can shave hours off of larger infrastructure management tasks, or simply allow you to bash in commands instead of clicking a mouse (fun for the old-schoolers).