the Zen of Communication.

Running some performance tuning, the app needed to know how many flops (floating point operations) per cycle the system could handle.

I used SiSoftware’s Sandra benchmarking app. It told me that my Intel Pentium D Dual Core 1.8ghz proc was producing 10.86gflops, but not the flops per clock cycle.

From this we know: a) the total gigaflops (10.86), b) the number of cores (2), and c) the number of clock cycles per second (1.8ghz)

Example of the standard formula:
The formula to determine total gigaflops is:
Flops per cycle x # of Cores x Clock speed.

This involves four values:

a = flop per clock cycle
b = clock speed (ghz)
c = cores
n = gflops

For a dual core 3ghz system with 4 flops per cycle, we can deduce 24gflops (a x c x b = n, or 4 x 2 x 3 = 24) . But I only have the total gflops, clock speed, and number of cores.

Reverse algebra:

a = n / b / c

Or in my case:
10.86 gflops / 1.8ghz / 2 cores = 3.01 flops per cycle (per core). So the E2610 chip at 1.8ghz produces 3 flops per cycle per core, or 6 flops total. Ta da.

Note: It’s worth mentioning that in this case, 10.86 gflops and 1.8ghz seem like closely related numbers, and that it would be quick to figure out how many gflops a system can handle by its clock speed (i.e. 1.8ghz equals 10.86gflops).  This is not the case.  In the first example of a dual core 3ghz proc producing 24gflops, you can’t deduce the one from the other.  It was just a coincidence in my case, so don’t do that.

Installed cacti from the “easy” installer – http://forums.cacti.net/about14946-0-asc-0.html – which does simplify a lot, but there are a lot of other hacks i’ve had to implement to make it work.

First and foremost, any time the poller runs i was getting “Cannot find module (IP-MIB): At line 0 in (none)”

Repeat about a dozen times for various mibs, and you obviously have a problem.

Under System Properties, Advanced, Environment Variables, there’s a new variable called MIBDIRS .  It’s pointed to c:\php\extras\mibs which in my case, after running the installer, was empty.

TO FIX: Either update this to point to your actual mib directory (mine was c:\usr\mibs , i’ve also seen d:\usr\mibs) or copy your populated mib directory (with IP-Mib and about a dozen others) to c:\php\extras\mibs .  Presto, the poller now runs as it should.

Loaded MRTG, followed as many how-tos as I could get my hands on, configured a WMI script, and all I get was:

C:\Program Files\mrtg-2.16.0\bin>perl mrtg mrtgwmi.cfg
Daemonizing MRTG …
Do Not close this window. Or MRTG will die
2009-12-17 13:06:38: ERROR: Target[my.monitor][_IN_] ‘cscript //nologo mymonitor.vbs myserver’ (kill): Search pattern not terminated at (eval 18) line 1.
2009-12-17 13:06:38: ERROR: Target[my.monitor][_OUT_] ‘cscript //nologo mymonitor.vbs myserver’ (kill): Search pattern not terminated at (eval 19) line 1.
Terminating on signal SIGINT(2)

Turns out in my config, the line calling the script:
cscript //nologo “c:\program files\mrtg-2.16.0\scripts\mymonitor.vbs” myserver

I wasn’t wrapping it in the right apostrophe. The line should read:

Target[my.monitor]: `cscript //nologo “c:\program files\mrtg-2.16.0\scripts\mymonitor.vbs” myserver`

Using the apostrophe left of the 1 (`) and NOT the apostrophe next to the return key (‘) . I love perl, really.

Setting up a new pool (persistent linked clone, in this instance) for VMWare View 4. Had the template built, took a snapshot, then tried to create the pool. Got all the way through setup to select the template and snapshot, but voila. Snapshot wasn’t there.

VMWare View templates require the snapshot to be taken WITHOUT the VM memory state. If the memory state is taken with the snap, the snapshot isn’t available – except it won’t tell you why.

VMWare also highly recommends/suggests that the VM template be powered off. This makes a lot of sense, and you really should, but I had two snaps taken with the system powered on which I built VDI Persistent Linked Pools from, and 10 desktops worked fine.

Warranty service required on a PowerEdge (m600 specifically, but pick your poison). Replacement motherboard shipped, with no service tag burned in. Requires “asset.com” (DOS .com file).

Note: Some people have reported Asset.com running in a dos box under Windows.  All I know is it doesn’t work under x64.

How to update the service tag:
- Download the Dell Diagnostics CD (R212797 – Extracts to an exe which then makes an ISO or USB stick.  http://support.dell.com/support/downloads/driverslist.aspx?os=LIN45&catid=13&dateid=-1&impid=-1&osl=EN&servicetag=&SystemID=PWE_2900&hidos=NW&hidlang=EN&TabIndex=).
- Create the ISO. Load up your DRAC/iDRAC interface. Mount the ISO as the virtual CD ROM drive.  (Path will be different, wherever it expanded to.  I moved mine.)

- Reboot your server. From the console (iDRAC or physical) hit the boot menu, select Virtual CD. It will boot to the diagnostics.

> ALTERNATIVELY: Burn the ISO to a CD, boot your CD from that instead.

- Once booted, Pick option 4 – quit. Drops you to a D:> dos prompt.

- change to C:, cd to UTIL . dir should reveal asset.com

- Run asset.com to see your service tag number (or if it’s blank).

- Run asset.com /s AB12345 to update the service tag number (AB12345 will be the number you want to burn in).

- Run asset.com to check that the change has taken. If so, reboot your machine, disconnect your Diags ISO, and walk away from a job well done.

Ta da.

Last time (here) I introduced an OID tree for the descriptions of Storage.  I hope you copied and pasted your results into a new window, because we’ll need them.

If not, here’s mine:

c:\SNMP>snmputil walk 127.0.0.1 public .1.3.6.1.2.1.25.2.3.1.3
Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.1
Value    = String A:\

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.2
Value    = String C:\ Label:  Serial Number 2053422

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.3
Value    = String D:\ Label:Data  Serial Number c9d83a42

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.4
Value    = String E:\

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.5
Value    = String F:\

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.6
Value    = String G:\ Label:FileDump  Serial Number 0dc359f2

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.7
Value    = String Virtual Memory

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.8
Value    = String Physical Memory

End of MIB subtree.

c:\SNMP>

So these are all the OIBs we can pull from the branch .1.3.6.1.2.1.25.2.3.1.3, which is the “Host Resource Storage Description” or hrStorageDescr for short. 

In MRTG’s case, we can’t set up a chart to poll .1.3.6.1.2.1.25.2.3.1.3, because it has subvalues.  So how do we get there?  Well, first – keep track of what number goes to what drive.  .2 is my C: drive, .3 is my D: drive, and .5 is my F: drive.  .4 and .5 have no descriptions because they’re CD-ROM drives.  I want to stress:  Your values may be different!  Your C: drive may be .3, .4, or any other number depending on your system configuration.

Now just for kicks, try entering this (one line):

c:\SNMP>snmputil get 127.0.0.1 public .1.3.6.1.2.1.25.2.3.1.5.2

You’ll notice the subtle change – we’re not ..2.3.1.3.2 anymore, we just switched to ..2.3.1.5.2 .  We stayed with .2 at the end because that’s the value for the C: drive (on my system – yours may be different!).  But the .5 is now a different OID tree.  Want to find out what?  Type (again, one line):

c:\SNMP>snmputil walk 127.0.0.1 public .1.3.6.1.2.1.25.2.3.1.5

You should get a listing of all the total storage sizes for all volumes on your system, including Virtual and Physical memory spaces.  But it’s not going to look sensible, because it’s giving us the number of blocks on each volume – not Kilobytes, Megabytes, or Gigabytes.  [For a complete explanation, read this post from the MRTG Mailing List.]

Remember what the value was for .2, or whatever your C:\ drive was.

So we have the number of blocks, but that doesn’t tell us much.  We want to know in megs or gigs what the total storage space is.  Next we need to find out what the block sizes are.  Lucky for us, there’s an SNMP OID for that.

c:\SNMP>snmputil walk 127.0.0.1 public .1.3.6.1.2.1.25.2.3.1.4

This will ‘walk’ the OID tree of block sizes for each volume on our system.  Lets say we just wanted to find the C: drive block size.  We’d use:

c:\SNMP>snmputil get 127.0.0.1 public .1.3.6.1.2.1.25.2.3.1.4.2

to which my system responded:

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageAllocationUnits.2
Value = Integer32 4096

Which tells me that my C: drive, has a block size of 4096 bytes.  So, for our own math here – My C:\ drive has a total number of 8958237 blocks.  We multiply that by 4096 to get 36,692,938,752 bytes.  To reduce that to Gigabytes, we divide by 1024, three times.

36,692,938,752 / 1024 = 35,832,948 Kilobytes.
35,832,948 Kb / 1024 = 34,993 Megabytes
34,993 Mb / 1024 = 34.17 Gigabytes.

Which, if I’ve done my math correctly, is exactly what I should see when you pull up the properties on my C: drive.  And sure enough -

So that’s a very basic introduction-by-example to an OID tree, and it’s uses.  I’ll write another example about calculating the percentage of used space on a drive with MRTG, which will use more OIDs. 

For more information on MRTG, SNMP, and MIBs, see this post.

A client is using a mixed environment, with several VMWare ESXi clusters, and several ESX clusters.  They’ve been running a number of maintenance tasks through SSH on the ESX clusters, but believed one could not get to a console under ESXi 3.5.  I found a quick howto:  http://www.vm-help.com/esx/esx3i/ESXi_enable_SSH.php

The short of it is you have to enter a debug mode in the console.  From the main console window, press ALT+F1 to get to the console terminal.  You’ll start out with:

(names have been changed to protect the innocent).  And after hitting ALT+F1, you’ll get:

Whereupon you’ll type “unsupported” (no quotes) and hit enter, and it will not be visible.  Just trust me, it’s there.  You may need to try it a couple of times if the console has had any keypresses still in the buffer.  Afterwards, you’ll get:

Enter your root console password here.  You’ll get dumped to a linux prompt.  Know this: Yes, it’s “linux”, but it’s stripped.  Many rudimentary functions are not present. 

From here, edit /etc/inetd.conf (using vi).  Scroll down until you find the line with “#ssh” .  Remove the # to enable the line .  (the vm-help.com page has detailed vi instructions.  I won’t go into those here.  But here’s more help) 

Once you’ve uncommented the ssh service, write and quit.  Then run /sbin/services.sh restart. 

Now, every other howto out there would leave you believing you should be all set.  If you’re anything like me, you’ll reboot your host, and then wonder where you went wrong.  In life.  After all, 5 or 10 how-tos have the same instruction set, and everyone else said “hai this rox kthxbye!”.  And yet I followed the instructions and I still had no joy.  I killed the inetd process numerous times, and had no joy.  What was a frustrated sysadmin who enjoyed such problems as this to do?  Get cracking.

I eventually hit the logs and discovered a couple of parameters were missing for their implementation of the ssh server, dropbear.  The path in the /etc/inetd.conf file was simply /sbin/dropbear .  For kicks, I tried to run ./sbin/dropbear.  It, in a world of generosity, spit out a list of symlinks I needed to create.

If I’m remembering this right, I did what it asked but it still didn’t work.  So, the actual path I ended up using in /etc/initd.conf was:

/sbin/dropbearmulti [tab] dropbear ++min=0,swap,group=shell –i

This calls the dropbearmulti app, and instead of using the symlink method it’s asking for, just tells it “here, run the server, and here’s your arguments.” .  It seems to be working, because several weeks and a few reboots later I’m SSH’d into the server to pull up the details for this post. 

Back with the MRTG and SNMP series, I spent a good number of hours trying to get otherwise-working configurations to work on my server with rare success.  Every once in a while, I’d get a completely different value than what I expected, and other times I’d get no value whatsoever because that OID couldn’t be found, even though it worked on another (live!) config!  This will be an overview about how to find the OID value you’re looking for, specifically Hard Drives and Memory.

I’m going to presume you’ve already become vaguely familiar with what SNMP is as a concept, you’ve installed the Windows SNMP server, and you’ve configured a community name and set allowed hosts.  If you haven’t, start here.

Once you have the basics done, get a copy of SNMPUtil.exe .  If you have a Windows NT4 cd laying around, how handy.  If not, go here and get it.  Note: That site also has an app called SNMP-Informant available.  I’ve heard it simplifies things considerably, but I’m not using it at present, I like the pain.  Once you’ve downloaded SNMPUtil, move the .exe to your windowssystem32 directory that way you can use it from a command line without specifying a full path. 

[ For those still wondering, the windowssystem32 directory is already in the “Path” portion of your Environmental Variables, which means we can call a program in the dir from anywhere.  While we could have added whatever directory you placed snmputil.exe in to the Path variable, my way was easier.  Google Environmental Variables for more help. ]

Once you have snmputil placed, open a command prompt and run it with no flags/arguments.  You should get a response like:

c:SNMP>snmputil
Error:  Incorrect number of arguments specified.

usage:  snmputil [get|getnext|walk] agent community oid [oid ...]
        snmputil trap

c:SNMP>

That tells us that there are three options – get, getnext, and walk – when we’re using the app.  The rest of the arguments are agent (which is the device/server you’re trying to poll, in my case localhost or 127.0.0.1), community (uhh… Google.) and oid which is the number found in a MIB. 

Quick explanation: A MIB is a “database” (big text file) with individual OIDs in them.  An OID is a specific resource with a value.  So if I’m looking for an OID that will poll my Windows Server’s Processor utilization, I want the Windows NT Performance MIB (http://www.mibdepot.com/cgi-bin/vendor_index.cgi?r=microsoft&id=144151), and the corresponding OID.

Back to SNMPUtil…

Those arguments (get, getnext, walk) will do three related but different things.  Get will get the value from a specific OID (such as “total hard disk space”).  getnext will get the NEXT OID in line (don’t worry about this yet).  And walk will follow  a an OID tree to show you every value you can get.

Now the configuration sections that I found and used (such as from snmpboy.msft.net) referenced specific OIDs.  What I’ve discovered: NOT ALL OIDS ARE THE SAME FROM SYSTEM TO SYSTEM.  Let me make that perfectly clear, because nowhere did I find this written, and it’s taken me days to suss out.  An OID that polls Virtual Memory for one system will NOT be the same OID on another!  The same holds true for drives.  Where the config demonstrated may find free space on drive C, that same OID for you could be polling drive A, B, D, or anything else.

So how do we find the OIDs for our specific system? 

I’m going to introduce the rather disturbingly simple hierarchy that is in place with SNMP.  The OID we’re going to start with is .1.3.6.1.2.1.25.2.3.1.3 .  This is not a final value (such as “total hard disk space”, but one branch in a tree (such as: “storage resources”).  These examples will assume you’re using an SNMP Server from your local machine, and your community string is public – Substitute Accordingly.  From your console, type (all one line):

c:SNMP>snmputil walk 127.0.0.1 public .1.3.6.1.2.1.25.2.3.1.3

which, if your system is anywhere like mine, will return something similar to:

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.1
Value    = String A:

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.2
Value    = String C: Label:  Serial Number 2053422

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.3
Value    = String D: Label:Data  Serial Number c9d83a42

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.4
Value    = String E:

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.5
Value    = String F:

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.6
Value    = String G: Label:FileDump  Serial Number 0dc359f2

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.7
Value    = String Virtual Memory

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.8
Value    = String Physical Memory

End of MIB subtree.

c:SNMP>

Now at first glance, these results might be somewhat confusing.  They were to me.  Here’s the down and dirty:

That OIB we entered (.1.3.6.1.2.1.25.2.3.1.3) was the “Host Storage Description” tree/branch.  The values underneath it pull the descriptions of storage objects that SNMP is mapping.  In my case, .1 goes to my floppy, .2 to my C: drive, .3 to my D: drive, etc.  This also includes .7 for Virtual Memory, and .8 for Physical Memory.  (Remember that).  Do yourself a favor and copy and paste those results into a text file for easy viewing and reference.

What we’ll do with these will be in the next post.

The following is a compilation of the resources I’ve been taking advantage of in configuring MRTG and SNMP on my system.  Given how long they took me to put together and the number of hours I spent trawling, hopefully it will be of some use and a bit of a timesaver.

The App itself -

- MRTG itself: http://oss.oetiker.ch/mrtg/
(the documentation section is mostly your friend, but can be unhelpful, especially with a WinNT implementation)

- SNMPBoy: http://snmpboy.msft.net/ . A live demonstration of MRTG for capturing both WS-Man information and SNMP information.  View in IE to use the drop-down menus on the left (that took me days).  His MRTG config is at http://snmpboy.msft.net/pub/mrtg.cfg.txt .  He also has all of the Windows MIBs available.

Get MRTG up and running -

- Install and Setup MRTG on Windows and IIS: http://www.amset.info/netadmin/mrtg.asp .  A basic installation guide for monitoring an SNMP enabled device. Not much detail, but it got me started. 

- MRTG for Intrusion Detection w/ IIS 6: http://www.securityfocus.com/infocus/1721 .  A very good article, but he jumped from “Here’s how to configure MRTG” to “Here are the VBScripts I use for pulling WMI data”.  I had a lot of in-between work to do.

- Burnett (the author of the above article) posted his config files at: http://www.securityfocus.com/microsoft/images/burnett_MRTG_files.zip .  I used them as a starting point, but what he fails to mention is that OIDs can be different from system to system.  I also bailed on the WMI scripts. 

- Another MRTG/SNMP/Windows config guide: http://www.syslog.gr/content/view/10/99 .  This one had much more detail, but you have to remember configs are unique to the system.

Other helpful stuff:

- Configure MRTG as a service: http://forums.firedaemon.com/showthread.php?t=45 

- SNMP4tPC: http://www.wtcs.org/snmp4tpc/testing.htm#SNMPUTIL .  An older page, but search results just kept coming back to it.  A lot of the theory is still sound, but be sure you discover the OIDs yourself.  The page also has Microsoft’s legacy SNMPUtil available for download – get it.  (His app turned into SNMP Informant, which I’ve read is very useful.)

- MIBDepot: http://www.mibdepot.com  .  A database of MIBs and OIDs.  Although comprehensive, I have yet to find a way to download a MIB.

- NetSNMP tools compiled for Windows.  http://www.elifulkerson.com/articles/net-snmp-windows-binary-unofficial.php .  The *nix SNMP utils compiled for Windows.  Nice job. 

The more I find the more I will add.  Good hunting.

I’ve undertaken the task of installing MRTG on one of my servers.  I plan on using it for the usual – network load, memory usage, drive space, etc – but at some point I’m going to find a way to poll event logs for things like Failed Login Attempts (to look for a brute force attack), or a high number of disk errors to indicate a dying drive, things like that.

What I’m going to write up in a series of posts will not be a how-to guide to get MRTG up and running under 2003.  Plenty of sites (referenced below) already give you that, some of which I used.  The biggest problems I’ve had are:
A) once I get MRTG installed and running, what then?  and
B) What MIB or OID do I use and for what, and how do I find them?

A number of the scripts, howtos, and other get-started pages include entries that you can copy and paste into your MRTG config.  Some of them work straight off the bat.  Others leave you scratching your head.  And yet others look like they should work, but can produce insane errors.  Such as this one:

(from a config section to poll memory stats)

YLegend[localhost.memoryUsed]: % Memory Used
Options[localhost.memoryUsed]: growright,gauge
Target[localhost.memoryUsed]: .1.3.6.1.2.1.25.2.3.1.6.3&.1.3.6.1.2.1.25.2.3.1.6.4:public@localhost / .1.3.6.1.2.1.25.2.3.1.5.3&.1.3.6.1.2.1.25.2.3.1.5.4:public@localhost * 100
MaxBytes[localhost.memoryUsed]: 523444000
Title[localhost.memoryUsed]: Server: Memory Used
ShortLegend[localhost.memoryUsed]: %
Legend1[localhost.memoryUsed]: Vir in next minute
Legend2[localhost.memoryUsed]: Phy in next minute
Legend3[localhost.memoryUsed]: Maximal 5 Minute Vir
Legend4[localhost.memoryUsed]: Maximal 5 Minute Phy
LegendI[localhost.memoryUsed]:  Vir
LegendO[localhost.memoryUsed]:  Phy
PageTop[localhost.memoryUsed]: <H1>Memory Utilization</H1>
    <TABLE>
       <TR><TD>System:</TD>     <TD>Server</TD></TR>
        <TR><TD>Vir</TD><TD>.1.3.6.1.2.1.25.2.3.1.6.3</TD></TR>
        <TR><TD>Phy</TD><TD>.1.3.6.1.2.1.25.2.3.1.6.4</TD></TR>
    </TABLE>

Looks relatively straight forward.  And given that I copied and pasted quite blatently from a functioning config (see http://snmpboy.msft.net, but make sure you use Internet Explorer – that “bug” took me days to work out.) you’d think it would work just fine.  But it produced two sets of errors – the first was difficulty with the OIDs.  They didn’t poll my RAM and pagefile.  The second, once I fixed that, was a set of errors:

SNMP Error:
no response received
SNMPv1_Session (remote host: "localhost" [127.0.0.1].161)
                  community: "public"
                 request ID: 780985927
                PDU bufsize: 8000 bytes
                    timeout: 2s
                    retries: 5
                    backoff: 1)

followed by:

Use of uninitialized value in division (/) at (eval 30) line 1.

The fun part was that the second error then seemed to hose up the rest of an already functioning config, apparently by deciding that it would no longer poll localhost for any info.  As localhost is the only server I’m polling, that obviously presents a bit of a problem.

Since I was adding new sections into the config one at a time and testing them, I knew exactly which one had the problem – the memory usage stats.  Something I found interesting was that when I moved the failing section to the very end of the cfg file, it worked.  More intrigued, I did some poking around.  Google turned up some not so helpful results which were mostly mailing lists where people asked “what does this uninitialized value mean?” but received no answer.

Flashback (6 hours): I’d been playing around with SNMPUtil’s get function earlier today when I was figuring out what OIBs mapped to what components (another post will follow on that), and I noticed that some polls were taking quite a while to respond – 4 to 5 seconds sometimes.  That first error tells me that the timeout is 2 seconds.  If the poll was taking longer than that, MRTG would fail it and move onto the next operation – in this case, dividing another poll against the one that just failed.  Essentially, a 0 divided-by error. 

“Ok, so how do you turn up the timeout value?” you may ask.  Good question!

http://oss.oetiker.ch/mrtg/doc/mrtg-reference.en.html (the official MRTG reference guide) has a section on SNMPOptions, one of which includes setting the timeout value.  The exact syntax is:

snmpoptions[cfgname]: timeout => (x)

I used 5 seconds, so I set it to timeout => 5 .  In my implementation, this looks like:

####
#
# Memory Utilization (SNMP)
#
####

YLegend[localhost.memoryUsed]: % Memory Used
Options[localhost.memoryUsed]: growright,gauge
SnmpOptions[localhost.memoryUsed]:timeout => 5
Target[localhost.memoryUsed]: .1.3.6.1.2.1.25.2.3.1.6.7&.1.3.6.1.2.1.25.2.3.1.6.8:public@localhost / .1.3.6.1.2.1.25.2.3.1.5.7&.1.3.6.1.2.1.25.2.3.1.5.8:public@localhost * 100
MaxBytes[localhost.memoryUsed]: 523444000
Title[localhost.memoryUsed]: Stargate: Memory Used
ShortLegend[localhost.memoryUsed]: %
Legend1[localhost.memoryUsed]: Vir in next minute
Legend2[localhost.memoryUsed]: Phy in next minute
Legend3[localhost.memoryUsed]: Maximal 5 Minute Vir
Legend4[localhost.memoryUsed]: Maximal 5 Minute Phy
LegendI[localhost.memoryUsed]: &nbsp;Vir
LegendO[localhost.memoryUsed]: &nbsp;Phy
PageTop[localhost.memoryUsed]: <H1>Memory Utilization</H1>
    <TABLE>
       <TR><TD>System:</TD>     <TD>Server</TD></TR>
        <TR><TD>Vir</TD><TD>.1.3.6.1.2.1.25.2.3.1.6.7</TD></TR>
        <TR><TD>Phy</TD><TD>.1.3.6.1.2.1.25.2.3.1.6.8</TD></TR>
    </TABLE>

(more will come about why the OIBs changed).  Sure enough, though, those errors ceased and a previously hosed configuration now worked fine.  So if you receive an uninitialized value error, or a no response received error, try increasing the timeout. 

Just bought a series of HP DL360s from a firesale.  AOL was refreshing a datacenter, these things were dirt cheap.  I’ve outfitted one with a pair of UW320 hotswap 72gb drives, but I’m figuring out a mod to put SATA drives in – cheaper, readily available, and don’t require ubercash and ebay to pick up if one dies.  It looks straight forward enough.  From my initial pokings, the SCSI backplane just lifts out.  The power connector is a single molex to the backplane that distributes it to the two drives, so I need a y-cable.

Empty drive trays are $10 a pop, so I’ll use those to lock the drives in place.  I have one outfitted with 4gb ram already which should be enough for most ops.  They’re dual-proc Xeons (single core, no VT) at 2.8ghz, and should make nice dogfood boxes for labs w/ VMWare GSX server.  Once I load test them, know they’re stable and won’t crap out on me, I may replace my dual p3-800 server which hosts my critical stuff.  At the very least, with a cheap SATA raid array, great media and file server.

www.nautilusnet.com was the company.  The chassis were $20 a pop, came with the procs and 1gb ram, no drives.  They had a 3.04ghz model for $30, same outfit (240 mhz doesn’t make that much diff. to me).  The company has everything else needed to outfit them – drives (36 or 72gb), memory kits, and redundant power supplies.  I’ve put in three orders with them now, and they’re awesome to work with.

Disclaimer: I’m not paid nor do I benefit in any way from this plug.  I don’t get kickbacks, gift cards, taken out for free drinks by the sales staff, free gear, logo’d jackets, tote bags, bumper stickers, or any special sale price on equipment.  They’re good people that do a good job and are worth a mention.

I also just bought a Linksys WMP300N PCI Wifi N card, and a WRT600N “Ultra RangePlus Dual-Band Wireless-N Gigabit Router” .  I installed them last night in hopes of getting better “reception” with my home-theater box (which I use for Netflix On Demand, and MS Flight Simulator – rest in peace – on my TV), but ran into a few hurdles configuring the 5ghz N signal.  A post will be forthcoming about that and some mild insanity once I stop foaming at the mouth about the Linksys wifi manager software.

I’m writing this up now so I don’t forget it – I’ve got about a dozen posts that I’ve started writing, but haven’t finished, and I want to get this one out there…

I’m preparing for an SCCM deployment for a client.  Instead of going through (yet again) the 10 prep worksheets from TechNet, I decided to combine them into an Excel spreadsheet.  It’s got pretty colors and nifty layouts.  I just wish I could get it to do bar graphs, as it would be perfect for executive presentations then…

The SCCM Planning Worksheet Compilation (link is the file) is a combination of all the worksheets, and will (hopefully) streamline the process of planning and preparing from an SCCM deployment in your org.  Feel free to post comments, let me know how it worked for you, or if there’s anything you’d suggest be added or changed.

Thanks and good luck

(If you missed it the first time, here’s the file: SCCM Planning Worksheet Compilation)

Here’s the scenario:

We’re attempting to set up a trust between two domains – Source.com, and Domain-Baker.com.  But there’s a catch.  We’ll exclude the how’s and why’s, but creating the trust is more complicated than usual because of the routing on the network.  Source.com and Domain-Baker.com can’t ping each other directly, however a domain controller (BridgeDC.Domain-Baker.com) has been placed on a subnet segment that can reach both.  The picture below draws this out a bit – (green = ping, black = network route, red = can not ping)

So the DCs in Source.com can’t reach the DCs at the main site across the map, but they can reach the Bridge DC in the middle.  The BridgeDC can, in turn, reach both sites.

Typically, when you’re creating a trust, you’d create a secondary DNS zone in Domain A (in this case the zone Domain-Baker.com hosted on the DNS server for Source.com, with a master of DNS.Domain-Baker.com), and vice versa (zone Source.com hosted on the DNS server of Domain-Baker.com).  (Diagram:)

This way both domains see the domain controllers available on either side of the networks, the trust gets established using one of the DCs located through DNS, and the trust comes up happily.

Except it fails.  It says it can’t find the domain.  In fact, the error itself was:

The New Trust Wizard cannot continue because the specified domain cannot be contacted.  Either the domain does not exist, or network or other problems are preventing the connection.

But DNS is in place, both zones are replicated, why did it fail?

Only one DC of Domain-Baker.com is reachable by Source.com -  BridgeDC.Domain-Baker.com (see the first diagram).  When Source.com queries “domain-baker.com” from the records that it’s just copied down, it gets a whole slew of resource records and DNS servers that simply aren’t available to it.  It looks something like -

But we know we have the BridgeDC for Domain-Baker.com sitting right there, why won’t it talk to it?  The answer is, it might – it’s a shot in the dark.  One out of 20 or 30 times it might work, because that would be the DC that gets resolved for Domain-Baker.com .  But the timing would have to be just right, and you’d never know when it was lined up.  Obviously a trust that only works one in 30 (or more) doesn’t help anyone.

So how do we force Source.com to only use BridgeDC.domain-baker.com to create the trust?  DNS.

Editing DNS Records to Specify Domain Controller

The goal will be to edit the DNS records hosted in Source.com so that it only knows about BridgeDC.domain-baker.com .  Now before you go jumping into the DNS console and start deleting records, hold on!  Things can get pretty ugly if they’re not handled right.

First you have to make sure you’re editing the right records.  Deleting all the other servers for the zone Domain-Baker.com in your production Active Directory (of Domain-Baker.com) would obviously wipe out half your network.  Yes, Source.com would transfer the updated zone with only the records for BridgeDC, but the ends don’t justify the means in that case.

So we have to make Domain-Baker.com editable within Source.com .  The easiest thing to do:

[I use my demonstration domain names here, obviously you'll have to substitute your domains in their place]

1) From the DNS console of a DNS server in Source.com, create a new secondary zone (yes, secondary) called Domain-Baker.com .

2) When it asks for the IP address to copy the zone from, put in the IP address of BridgeDC (or any other DNS server that has a copy of the target zone).

3) Confirm the zone, and watch it populate.

Now, this is going to be a complete copy of the zone, full of records for servers we can’t use.  So, lets get rid of them.

1) Open the properties of the zone.  Change the type of zone from Secondary to Primary (making the zone AD Integrated is your choice).

2) Under the Zone Transfers tab, turn OFF zone transfers.

3) Under the Name Servers tab, pull out the DNS servers listed for Domain-Baker.com.  We don’t want our other servers making queries on the complete copy of the zone!

Now, we’ve just made the zone Domain-Baker.com editable within Source.com, we’ve made sure our changes will NOT replicate to Domain-Baker.com (and thus screw up the network quite royally), and we’ve made sure that this will be the only server to host this record.  So lets get to wiping the other servers.

Expand the zone, and you’ll probably see a great number of records and servers.  Because of the way Active Directory works, there’s a lot of information in there about all of the AD Sites, servers at each site, workstations, etc etc.  DNS, you may not know, is one of the most crucial parts of Active Directory.  Because of this unique issue, we need to sort through those records and servers, and pull out the ones we don’t want. 

Go through each folder from the top down, and delete any Name Server (NS), Service Location (SRV) record, or a No Name A record (“Same as parent folder”), that is NOT the domain controller we want to use (in this case we want BridgeDC.domain-baker.com / IP 172.16.56.4, and don’t want the others).  You’ll have to go through each and every folder inside the Domain-Baker.com zone to delete the extraneous records.  The end result should look like:

Multiple DNS Servers in Source.com

Now, if you have multiple DNS servers within Source.com, you’ll need to set them to only query your DNS server which has the Domain-Baker.com zone, any time they run a lookup (unless your zone is AD Integrated, in which case skip this step).  On each other DNS server:

1) Open the server properties (DNS console, right click the Server, click Properties),

2) Open the Forwarders tab,

3) Create a new forwarder for Domain-Baker.com,

4) Put the IP address of the DNS server which we just edited the records on.

This ensures that only your edited copy of Domain-Baker.com is queried from your source domain.  After making changes to DNS servers, especially when they’re across the board, its good practice to restart each DNS service, and on each DC run

ipconfig /flushdns
and
ipconfig /registerdns

Creating the Trust

DNS should now be configured so that from any server in Source.com, you can ping Domain-Baker.com and only reach BridgeDC.domain-baker.com (again, substitute your hostnames/domains here).  You should also be able to nslookup domain-baker.com and only receive BridgeDC.domain-baker.com or its IP address.

Once that’s confirmed, create the trust.  I suggest you do this from the Source.com side – I tried to run it from Domain-Baker.com and discovered that it was asking one of the back-end DCs that couldn’t reach Source.com to establish the trust.  Once I ran it from Source, everything worked great.

You may receive an error indicating the trust could not be validated, because there isn’t a workstation or computer account created.  (Or words to that effect).

Open Microsoft Support Article 246264 (http://support.microsoft.com/kb/246264).  It says that the Trust may validate itself, even though it throws this error.  Sure enough, once I told it to validate, then got the error, had it reset the password, then told it to validate again (only reset the trust password once), it returned successful.

Presto, trust completed, using one DC as the “gateway”.

Recap

I don’t normally do this, but in this case it’s easy to get confused.  So here’s the bullet points.

• Attempting to set a trust between Source.com and Domain-Baker.com using BridgeDC.domain-baker.com as a “gateway” or “bridge”
• First created a secondary DNS zone in Source.com of Domain-Baker.com, replicated a complete copy of the zone from BridgeDC.
• Changed Secondary Zone to Primary zone.  Disabled Zone transfers and updates.
• Deleted any other SRV, NS, or NoName-A records that weren’t BridgeDC.domain-baker.com
• This makes Source.com think that the only server in Domain-Baker.com is BridgeDC.
• Set up the trust, Source.com now uses Domain-Baker.com for the trust (because it doesn’t know any better).

And that oughta do it.

Recently, I did a test run of a migration from Live Communications Server (LCS) 2005 to OCS 2007. Some things we found out:

1) OCS 2007 doesn’t like SQL 2000 SP4. There’s nothing indicating that it would be unsupported, but we couldn’t get OCS to create the pool with a SQL 2000 SP4 backend for love or money. We eventually ditched it and went straight to SQL 2005, which worked seamlessly.

2) You can make a single-server deployment of OCS Enterprise. Microsoft doesn’t support it, technically. With Enterprise, the pool and the server have to be on separate IPs. The pool DNS record is intended to be pointed at the load balancer you’re using for all those servers (right?). Well, a load balancer for a single server is a bit silly. So, assign the server a second IP address, and change the pool A record to point to that second IP. Clients have to be able to resolve the IP of the pool, and the client MUST see this as a separate IP from the server). Presto – an Enterprise deployment on a single server. That begs the question, if you’re only using one box, why run Enterprise? Well, like I said earlier – it was a test run.

3) OCS works under VMWare Server. Microsoft swear up and down that OCS isn’t supported under a virtualized environment. Sure, they may not help you if you get in trouble, but it worked fine for us. We set up a 2 server pool and the SQL server, and connected several clients to it without any trouble, all under VMWare Server. Even the LCS 2005 pool we were migrating from (in the test lab) was under VMWare. No trouble. We didn’t try Microsoft’s Virtual Server, so that may have some quirks as well.

4) Beware of enhanced presence (EP). If you’re migrating, and looking to run your users concurrently on LCS 2005 and OCS 2007, DO NOT deploy Office Communicator 2007 until the last step. OC2007 when connected to OCS 2007 activates enhanced presence in Active Directory. EP USERS CAN NOT INTERACT WITH NON-EP USERS. This is true even if the users are on the same pool or server. One special quirk we found was that EP can be ticked in the Communications tab in ADUC, but unless the user connects with OC 2007, EP is never activated. Different wording: If you only use OC 2005, nobody uses EP, so you’re fine. So, when you’re doing your migration, save the rollout of OC 2007 till the very end, and everyone can talk to everyone else.

Some resources I found valuable during our testing:

http://forums.microsoft.com/unifiedcommunications/ – Microsoft’s Unified Comms forums. Sometimes slow to reply, but the search engine always has good info.

http://communicationsserverteam.com/ – The OCS team’s blog.

http://blogs.technet.com/toml/archive/2008/01/28/lcs-ocs-coexistence-and-migration-series-allow-block-list-not-in-mmc.aspx – LCSKid’s blog. Great content, helped us tremendously.

It may just be a USB drive…

I was working on a client’s Small Business Server today. It had been some time since they’d had updates run, so I installed Server 2003 SP1, SBS 2003 SP1, then rebooted to prepare for Server 2003 SP2. After I’d rebooted, the server posted, loaded Server 2k3, and just before the expected ‘Preparing network connections’ I got a message that ‘The Active Directory is Rebuilding Indices’. Assuming it was something to do with the Service Packs I’d just loaded, I let it sit for a short while. Well, 20 minutes later it hadn’t made any progress, the disks were still idle, and my client was understandably wondering how much more time they’d be billed for this screen saver equivalent.

Rather than hardboot the server while it was still loading, I began to research exactly what the problem could be. I found several posts about the message having to do with VSS, others explaining that pulling the power cable to shut down the server isn’t the most advantageous method (nice), others offering that it may be the AD catalog has gone belly up and would need restoring (makes sense, but I really hoped not), and some suggesting that RAID hardware may be failing – either the controller or a drive in the array.

Well, this server was on a RAID array, which made my pulse quicken. But I’d seen no other signs of failed hard drives – the controller booted fine, no errors with the disks, chkdsk wasn’t offering itself up for sacrifice upon every boot. Didn’t seem like a failing array.

This post made mention of a corrupted USB driver, and how the poster’s server rebooted fine when there wasn’t a USB hard drive. I instantly thought back to the time I gave myself a heart attack when a clients server displayed ‘OS Not Found’ after I tried to reboot it, inadvertantly with a USB hard drive attached. (I turned off Booting from a USB Device shortly after that).

Well, I looked at the server I was working on, and sure enough there was a USB hard drive attached that my client used for backing up data. The disks weren’t being read (either the RAID or the USB) while it was stalled, so I thought what the hell. I turned off the power to the USB drive, disconnected the USB cable, and within seconds the server moved on to ‘Preparing Network Connections’.

So in conclusion, the ‘Active Directory is Rebuilding Indices’ message during boot may be indicative of a stalled drive or accessory. Check for a USB peripheral, an external SCSI disk drive, or maybe something using firewire. If you have any such devices, try turning them off (little bit tougher with external SCSI drives, but feasible if there’s nothing critical), and then try rebooting. I didn’t run into the message again, and it saved me a complete AD restore from tape, let alone uninstalling and reinstalling service packs and hotfixes.

I recently had a need to manually create some 400 AD-integrated stub zones for a client (~40 zones on 11 domains, and it was ten to midnight on a friday night). Because each domain was in a separate forest, AD integrated replication between forests and domains wasn’t an option, and secondary transfers would occupy too much bandwidth between sites, nor would they be as promptly updated as would be required. So I had two choices: 1) Create each record, specify the master servers, and configure replication, one by one, on each domain, all by hand (I think not). Or 2) Find some way to script it.

Off to Google I go.

Several searches for DNS scripting, server 2003 DNS command line, etc, later, and I’d been able to find references to dnscmd.exe (part of the Server Resource Kit – available from Microsoft here). There were several guides about the syntax, but it took several tries to get the CLI to work as I needed to. I figured I’d put this up to shave some time off the next poor sod’s research.

To create a record, open the Support Tools CLI. From the CLI, this string is the basic format:

dnscmd . /zoneadd domain1.com /DsStub 192.168.5.1 192.168.5.2 /DP /forest

This will open the local server you’re working on, create a new zone called domain1.com, store it as a stub in AD (/DsStub), use 192.168.5.1 and 192.168.5.2 as the master servers, put the zone in the directory partition of AD (the /DP switch), and set the zone to replicate to all DNS servers in the forest (the /forest switch). You can substitute another hostname for the “.” if you want to add records to another server, so “dnscmd dnsserver1 /zoneadd …”

That’s the basic CLI for creating a forward lookup zone. But I was actually interested in making reverse lookup zones, which has one more little trick to it. You have to put in the full in-addr.arpa addy of the reverse zone (you can see these in the DNS MMC console by right-clicking on your reverse zones, selecting ‘View’, and choosing ‘Advanced’). This made my string look like:

dnscmd . /zoneadd 5.168.192.in-addr.arpa /DsStub 192.168.5.1 192.168.5.2 /DP /forest

So this command is opening the local server (the ‘.’), creating a new zone (/zoneadd) that’s a reverse zone of 192.168.5.x (5.168.192.in-addr.arpa), specifying a Stub zone stored in AD (/DsStub) with master servers of 192.168.5.1 and 192.168.5.2, stored in the Directory Partition (/DP), with replication across all DNS servers in the forest (/forest).

After perfecting the syntax, I edited a text file with the appropriate IP addresses of the zones I was creating, which lead to a batch file with some 40 or 50 instances of dnscmd being called, each with different zones and IPs. I’m sure there’s some very clever way to run a perl script that calls dnscmd and pulls variables from a csv or text file, but that’s way beyond my abilities here and now.

So to point you in some other directions, there are many resources out there which cover dnscmd and its many capabilities – that’s beyond this post. Petri.co.il has a page here, Microsoft’s official TechNet page is here, and Google has many others. It’s a very powerful tool that can shave hours off of larger infrastructure management tasks, or simply allow you to bash in commands instead of clicking a mouse (fun for the old-schoolers).