the Zen of Communication.

Installed cacti from the “easy” installer – http://forums.cacti.net/about14946-0-asc-0.html – which does simplify a lot, but there are a lot of other hacks i’ve had to implement to make it work.

First and foremost, any time the poller runs i was getting “Cannot find module (IP-MIB): At line 0 in (none)”

Repeat about a dozen times for various mibs, and you obviously have a problem.

Under System Properties, Advanced, Environment Variables, there’s a new variable called MIBDIRS .  It’s pointed to c:\php\extras\mibs which in my case, after running the installer, was empty.

TO FIX: Either update this to point to your actual mib directory (mine was c:\usr\mibs , i’ve also seen d:\usr\mibs) or copy your populated mib directory (with IP-Mib and about a dozen others) to c:\php\extras\mibs .  Presto, the poller now runs as it should.

Loaded MRTG, followed as many how-tos as I could get my hands on, configured a WMI script, and all I get was:

C:\Program Files\mrtg-2.16.0\bin>perl mrtg mrtgwmi.cfg
Daemonizing MRTG …
Do Not close this window. Or MRTG will die
2009-12-17 13:06:38: ERROR: Target[my.monitor][_IN_] ‘cscript //nologo mymonitor.vbs myserver’ (kill): Search pattern not terminated at (eval 18) line 1.
2009-12-17 13:06:38: ERROR: Target[my.monitor][_OUT_] ‘cscript //nologo mymonitor.vbs myserver’ (kill): Search pattern not terminated at (eval 19) line 1.
Terminating on signal SIGINT(2)

Turns out in my config, the line calling the script:
cscript //nologo “c:\program files\mrtg-2.16.0\scripts\mymonitor.vbs” myserver

I wasn’t wrapping it in the right apostrophe. The line should read:

Target[my.monitor]: `cscript //nologo “c:\program files\mrtg-2.16.0\scripts\mymonitor.vbs” myserver`

Using the apostrophe left of the 1 (`) and NOT the apostrophe next to the return key (‘) . I love perl, really.

Last time (here) I introduced an OID tree for the descriptions of Storage.  I hope you copied and pasted your results into a new window, because we’ll need them.

If not, here’s mine:

c:\SNMP>snmputil walk 127.0.0.1 public .1.3.6.1.2.1.25.2.3.1.3
Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.1
Value    = String A:\

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.2
Value    = String C:\ Label:  Serial Number 2053422

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.3
Value    = String D:\ Label:Data  Serial Number c9d83a42

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.4
Value    = String E:\

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.5
Value    = String F:\

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.6
Value    = String G:\ Label:FileDump  Serial Number 0dc359f2

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.7
Value    = String Virtual Memory

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.8
Value    = String Physical Memory

End of MIB subtree.

c:\SNMP>

So these are all the OIBs we can pull from the branch .1.3.6.1.2.1.25.2.3.1.3, which is the “Host Resource Storage Description” or hrStorageDescr for short. 

In MRTG’s case, we can’t set up a chart to poll .1.3.6.1.2.1.25.2.3.1.3, because it has subvalues.  So how do we get there?  Well, first – keep track of what number goes to what drive.  .2 is my C: drive, .3 is my D: drive, and .5 is my F: drive.  .4 and .5 have no descriptions because they’re CD-ROM drives.  I want to stress:  Your values may be different!  Your C: drive may be .3, .4, or any other number depending on your system configuration.

Now just for kicks, try entering this (one line):

c:\SNMP>snmputil get 127.0.0.1 public .1.3.6.1.2.1.25.2.3.1.5.2

You’ll notice the subtle change – we’re not ..2.3.1.3.2 anymore, we just switched to ..2.3.1.5.2 .  We stayed with .2 at the end because that’s the value for the C: drive (on my system – yours may be different!).  But the .5 is now a different OID tree.  Want to find out what?  Type (again, one line):

c:\SNMP>snmputil walk 127.0.0.1 public .1.3.6.1.2.1.25.2.3.1.5

You should get a listing of all the total storage sizes for all volumes on your system, including Virtual and Physical memory spaces.  But it’s not going to look sensible, because it’s giving us the number of blocks on each volume – not Kilobytes, Megabytes, or Gigabytes.  [For a complete explanation, read this post from the MRTG Mailing List.]

Remember what the value was for .2, or whatever your C:\ drive was.

So we have the number of blocks, but that doesn’t tell us much.  We want to know in megs or gigs what the total storage space is.  Next we need to find out what the block sizes are.  Lucky for us, there’s an SNMP OID for that.

c:\SNMP>snmputil walk 127.0.0.1 public .1.3.6.1.2.1.25.2.3.1.4

This will ‘walk’ the OID tree of block sizes for each volume on our system.  Lets say we just wanted to find the C: drive block size.  We’d use:

c:\SNMP>snmputil get 127.0.0.1 public .1.3.6.1.2.1.25.2.3.1.4.2

to which my system responded:

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageAllocationUnits.2
Value = Integer32 4096

Which tells me that my C: drive, has a block size of 4096 bytes.  So, for our own math here – My C:\ drive has a total number of 8958237 blocks.  We multiply that by 4096 to get 36,692,938,752 bytes.  To reduce that to Gigabytes, we divide by 1024, three times.

36,692,938,752 / 1024 = 35,832,948 Kilobytes.
35,832,948 Kb / 1024 = 34,993 Megabytes
34,993 Mb / 1024 = 34.17 Gigabytes.

Which, if I’ve done my math correctly, is exactly what I should see when you pull up the properties on my C: drive.  And sure enough -

So that’s a very basic introduction-by-example to an OID tree, and it’s uses.  I’ll write another example about calculating the percentage of used space on a drive with MRTG, which will use more OIDs. 

For more information on MRTG, SNMP, and MIBs, see this post.

Back with the MRTG and SNMP series, I spent a good number of hours trying to get otherwise-working configurations to work on my server with rare success.  Every once in a while, I’d get a completely different value than what I expected, and other times I’d get no value whatsoever because that OID couldn’t be found, even though it worked on another (live!) config!  This will be an overview about how to find the OID value you’re looking for, specifically Hard Drives and Memory.

I’m going to presume you’ve already become vaguely familiar with what SNMP is as a concept, you’ve installed the Windows SNMP server, and you’ve configured a community name and set allowed hosts.  If you haven’t, start here.

Once you have the basics done, get a copy of SNMPUtil.exe .  If you have a Windows NT4 cd laying around, how handy.  If not, go here and get it.  Note: That site also has an app called SNMP-Informant available.  I’ve heard it simplifies things considerably, but I’m not using it at present, I like the pain.  Once you’ve downloaded SNMPUtil, move the .exe to your windowssystem32 directory that way you can use it from a command line without specifying a full path. 

[ For those still wondering, the windowssystem32 directory is already in the “Path” portion of your Environmental Variables, which means we can call a program in the dir from anywhere.  While we could have added whatever directory you placed snmputil.exe in to the Path variable, my way was easier.  Google Environmental Variables for more help. ]

Once you have snmputil placed, open a command prompt and run it with no flags/arguments.  You should get a response like:

c:SNMP>snmputil
Error:  Incorrect number of arguments specified.

usage:  snmputil [get|getnext|walk] agent community oid [oid ...]
        snmputil trap

c:SNMP>

That tells us that there are three options – get, getnext, and walk – when we’re using the app.  The rest of the arguments are agent (which is the device/server you’re trying to poll, in my case localhost or 127.0.0.1), community (uhh… Google.) and oid which is the number found in a MIB. 

Quick explanation: A MIB is a “database” (big text file) with individual OIDs in them.  An OID is a specific resource with a value.  So if I’m looking for an OID that will poll my Windows Server’s Processor utilization, I want the Windows NT Performance MIB (http://www.mibdepot.com/cgi-bin/vendor_index.cgi?r=microsoft&id=144151), and the corresponding OID.

Back to SNMPUtil…

Those arguments (get, getnext, walk) will do three related but different things.  Get will get the value from a specific OID (such as “total hard disk space”).  getnext will get the NEXT OID in line (don’t worry about this yet).  And walk will follow  a an OID tree to show you every value you can get.

Now the configuration sections that I found and used (such as from snmpboy.msft.net) referenced specific OIDs.  What I’ve discovered: NOT ALL OIDS ARE THE SAME FROM SYSTEM TO SYSTEM.  Let me make that perfectly clear, because nowhere did I find this written, and it’s taken me days to suss out.  An OID that polls Virtual Memory for one system will NOT be the same OID on another!  The same holds true for drives.  Where the config demonstrated may find free space on drive C, that same OID for you could be polling drive A, B, D, or anything else.

So how do we find the OIDs for our specific system? 

I’m going to introduce the rather disturbingly simple hierarchy that is in place with SNMP.  The OID we’re going to start with is .1.3.6.1.2.1.25.2.3.1.3 .  This is not a final value (such as “total hard disk space”, but one branch in a tree (such as: “storage resources”).  These examples will assume you’re using an SNMP Server from your local machine, and your community string is public – Substitute Accordingly.  From your console, type (all one line):

c:SNMP>snmputil walk 127.0.0.1 public .1.3.6.1.2.1.25.2.3.1.3

which, if your system is anywhere like mine, will return something similar to:

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.1
Value    = String A:

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.2
Value    = String C: Label:  Serial Number 2053422

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.3
Value    = String D: Label:Data  Serial Number c9d83a42

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.4
Value    = String E:

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.5
Value    = String F:

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.6
Value    = String G: Label:FileDump  Serial Number 0dc359f2

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.7
Value    = String Virtual Memory

Variable = host.hrStorage.hrStorageTable.hrStorageEntry.hrStorageDescr.8
Value    = String Physical Memory

End of MIB subtree.

c:SNMP>

Now at first glance, these results might be somewhat confusing.  They were to me.  Here’s the down and dirty:

That OIB we entered (.1.3.6.1.2.1.25.2.3.1.3) was the “Host Storage Description” tree/branch.  The values underneath it pull the descriptions of storage objects that SNMP is mapping.  In my case, .1 goes to my floppy, .2 to my C: drive, .3 to my D: drive, etc.  This also includes .7 for Virtual Memory, and .8 for Physical Memory.  (Remember that).  Do yourself a favor and copy and paste those results into a text file for easy viewing and reference.

What we’ll do with these will be in the next post.

The following is a compilation of the resources I’ve been taking advantage of in configuring MRTG and SNMP on my system.  Given how long they took me to put together and the number of hours I spent trawling, hopefully it will be of some use and a bit of a timesaver.

The App itself -

- MRTG itself: http://oss.oetiker.ch/mrtg/
(the documentation section is mostly your friend, but can be unhelpful, especially with a WinNT implementation)

- SNMPBoy: http://snmpboy.msft.net/ . A live demonstration of MRTG for capturing both WS-Man information and SNMP information.  View in IE to use the drop-down menus on the left (that took me days).  His MRTG config is at http://snmpboy.msft.net/pub/mrtg.cfg.txt .  He also has all of the Windows MIBs available.

Get MRTG up and running -

- Install and Setup MRTG on Windows and IIS: http://www.amset.info/netadmin/mrtg.asp .  A basic installation guide for monitoring an SNMP enabled device. Not much detail, but it got me started. 

- MRTG for Intrusion Detection w/ IIS 6: http://www.securityfocus.com/infocus/1721 .  A very good article, but he jumped from “Here’s how to configure MRTG” to “Here are the VBScripts I use for pulling WMI data”.  I had a lot of in-between work to do.

- Burnett (the author of the above article) posted his config files at: http://www.securityfocus.com/microsoft/images/burnett_MRTG_files.zip .  I used them as a starting point, but what he fails to mention is that OIDs can be different from system to system.  I also bailed on the WMI scripts. 

- Another MRTG/SNMP/Windows config guide: http://www.syslog.gr/content/view/10/99 .  This one had much more detail, but you have to remember configs are unique to the system.

Other helpful stuff:

- Configure MRTG as a service: http://forums.firedaemon.com/showthread.php?t=45 

- SNMP4tPC: http://www.wtcs.org/snmp4tpc/testing.htm#SNMPUTIL .  An older page, but search results just kept coming back to it.  A lot of the theory is still sound, but be sure you discover the OIDs yourself.  The page also has Microsoft’s legacy SNMPUtil available for download – get it.  (His app turned into SNMP Informant, which I’ve read is very useful.)

- MIBDepot: http://www.mibdepot.com  .  A database of MIBs and OIDs.  Although comprehensive, I have yet to find a way to download a MIB.

- NetSNMP tools compiled for Windows.  http://www.elifulkerson.com/articles/net-snmp-windows-binary-unofficial.php .  The *nix SNMP utils compiled for Windows.  Nice job. 

The more I find the more I will add.  Good hunting.

I’ve undertaken the task of installing MRTG on one of my servers.  I plan on using it for the usual – network load, memory usage, drive space, etc – but at some point I’m going to find a way to poll event logs for things like Failed Login Attempts (to look for a brute force attack), or a high number of disk errors to indicate a dying drive, things like that.

What I’m going to write up in a series of posts will not be a how-to guide to get MRTG up and running under 2003.  Plenty of sites (referenced below) already give you that, some of which I used.  The biggest problems I’ve had are:
A) once I get MRTG installed and running, what then?  and
B) What MIB or OID do I use and for what, and how do I find them?

A number of the scripts, howtos, and other get-started pages include entries that you can copy and paste into your MRTG config.  Some of them work straight off the bat.  Others leave you scratching your head.  And yet others look like they should work, but can produce insane errors.  Such as this one:

(from a config section to poll memory stats)

YLegend[localhost.memoryUsed]: % Memory Used
Options[localhost.memoryUsed]: growright,gauge
Target[localhost.memoryUsed]: .1.3.6.1.2.1.25.2.3.1.6.3&.1.3.6.1.2.1.25.2.3.1.6.4:public@localhost / .1.3.6.1.2.1.25.2.3.1.5.3&.1.3.6.1.2.1.25.2.3.1.5.4:public@localhost * 100
MaxBytes[localhost.memoryUsed]: 523444000
Title[localhost.memoryUsed]: Server: Memory Used
ShortLegend[localhost.memoryUsed]: %
Legend1[localhost.memoryUsed]: Vir in next minute
Legend2[localhost.memoryUsed]: Phy in next minute
Legend3[localhost.memoryUsed]: Maximal 5 Minute Vir
Legend4[localhost.memoryUsed]: Maximal 5 Minute Phy
LegendI[localhost.memoryUsed]:  Vir
LegendO[localhost.memoryUsed]:  Phy
PageTop[localhost.memoryUsed]: <H1>Memory Utilization</H1>
    <TABLE>
       <TR><TD>System:</TD>     <TD>Server</TD></TR>
        <TR><TD>Vir</TD><TD>.1.3.6.1.2.1.25.2.3.1.6.3</TD></TR>
        <TR><TD>Phy</TD><TD>.1.3.6.1.2.1.25.2.3.1.6.4</TD></TR>
    </TABLE>

Looks relatively straight forward.  And given that I copied and pasted quite blatently from a functioning config (see http://snmpboy.msft.net, but make sure you use Internet Explorer – that “bug” took me days to work out.) you’d think it would work just fine.  But it produced two sets of errors – the first was difficulty with the OIDs.  They didn’t poll my RAM and pagefile.  The second, once I fixed that, was a set of errors:

SNMP Error:
no response received
SNMPv1_Session (remote host: "localhost" [127.0.0.1].161)
                  community: "public"
                 request ID: 780985927
                PDU bufsize: 8000 bytes
                    timeout: 2s
                    retries: 5
                    backoff: 1)

followed by:

Use of uninitialized value in division (/) at (eval 30) line 1.

The fun part was that the second error then seemed to hose up the rest of an already functioning config, apparently by deciding that it would no longer poll localhost for any info.  As localhost is the only server I’m polling, that obviously presents a bit of a problem.

Since I was adding new sections into the config one at a time and testing them, I knew exactly which one had the problem – the memory usage stats.  Something I found interesting was that when I moved the failing section to the very end of the cfg file, it worked.  More intrigued, I did some poking around.  Google turned up some not so helpful results which were mostly mailing lists where people asked “what does this uninitialized value mean?” but received no answer.

Flashback (6 hours): I’d been playing around with SNMPUtil’s get function earlier today when I was figuring out what OIBs mapped to what components (another post will follow on that), and I noticed that some polls were taking quite a while to respond – 4 to 5 seconds sometimes.  That first error tells me that the timeout is 2 seconds.  If the poll was taking longer than that, MRTG would fail it and move onto the next operation – in this case, dividing another poll against the one that just failed.  Essentially, a 0 divided-by error. 

“Ok, so how do you turn up the timeout value?” you may ask.  Good question!

http://oss.oetiker.ch/mrtg/doc/mrtg-reference.en.html (the official MRTG reference guide) has a section on SNMPOptions, one of which includes setting the timeout value.  The exact syntax is:

snmpoptions[cfgname]: timeout => (x)

I used 5 seconds, so I set it to timeout => 5 .  In my implementation, this looks like:

####
#
# Memory Utilization (SNMP)
#
####

YLegend[localhost.memoryUsed]: % Memory Used
Options[localhost.memoryUsed]: growright,gauge
SnmpOptions[localhost.memoryUsed]:timeout => 5
Target[localhost.memoryUsed]: .1.3.6.1.2.1.25.2.3.1.6.7&.1.3.6.1.2.1.25.2.3.1.6.8:public@localhost / .1.3.6.1.2.1.25.2.3.1.5.7&.1.3.6.1.2.1.25.2.3.1.5.8:public@localhost * 100
MaxBytes[localhost.memoryUsed]: 523444000
Title[localhost.memoryUsed]: Stargate: Memory Used
ShortLegend[localhost.memoryUsed]: %
Legend1[localhost.memoryUsed]: Vir in next minute
Legend2[localhost.memoryUsed]: Phy in next minute
Legend3[localhost.memoryUsed]: Maximal 5 Minute Vir
Legend4[localhost.memoryUsed]: Maximal 5 Minute Phy
LegendI[localhost.memoryUsed]: &nbsp;Vir
LegendO[localhost.memoryUsed]: &nbsp;Phy
PageTop[localhost.memoryUsed]: <H1>Memory Utilization</H1>
    <TABLE>
       <TR><TD>System:</TD>     <TD>Server</TD></TR>
        <TR><TD>Vir</TD><TD>.1.3.6.1.2.1.25.2.3.1.6.7</TD></TR>
        <TR><TD>Phy</TD><TD>.1.3.6.1.2.1.25.2.3.1.6.8</TD></TR>
    </TABLE>

(more will come about why the OIBs changed).  Sure enough, though, those errors ceased and a previously hosed configuration now worked fine.  So if you receive an uninitialized value error, or a no response received error, try increasing the timeout.