 

b r o n c    v0.60
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
by chris snell <chris@bikeworld.com>
http://bronc.blueaspen.com

everything (c) 2000 chris snell except for badger.pl, which is
(c) 2000 ticketmaster online - citysearch, inc. and chris snell


Disclaimer
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Please read the "COPYING" and "COPYRIGHT" files for more information.

In short, if this software trashes your production servers or causes
your co-workers to break out with oozing skin lesions, we ain't liable.  

If it breaks, you can have both peices.


What is bronc?
. . . . . . . . . . . . . . . . . .

Bronc is a collection of perl scripts, mod_perl scripts, and Mason templates
that are designed to provide a highly flexible, extensible front-end to
Tobi Oetiker's RRDtool (http://ee-staff.ethz.ch/~oetiker/webtools/rrdtool/).

While there are other RRDtool front-ends available, I believe that this
implementation has several advantages over other front-ends.  First of
all, it uses a wonderful templating/caching engine called Mason 
(http://masonhq.com) to powerful definitions of the graphs you want
to create. Secondly, the graphing engine which interfaces with RRDtool
is written in Perl and designed to be run from mod_perl makes it 
quite fast with very little system overhead.  


What is required of you, the installer
. . . . . . . . . . . . . . . . . .

You *must* have an understanding of Perl.  Read the Learning Perl
and Programming Perl books from O'Reilly & Associates if you haven't
already.

You must have a basic understanding of RRDtool and have read the 
tutorials found off of the RRDtool home page at:
http://ee-staff.ethz.ch/~oetiker/webtools/rrdtool/

You must have a clue.  This is powerful software.  It's also very
"alpha" and buggy.  I hope you like debugging problems.  :-)
I should note, however, that I have it up and running nicely on
my site.  It doesn't crash.  It doesn't spew errors.  Your milage
may vary.



What you get
. . . . . . . . . . . . . . . . . .

- BRONC::Grapher, which is the interface between bronc and RRDtool.
  This component runs under mod_perl.

- chef, a Mason component which assembles Grapher requests based on 
  "definitions" (see below)

- definitions, a Mason component where you define your graphs--how you
  want them to look, what they are to contain, etc.

- badger, a perl script that fetches SNMP values (via Net::SNMP) and stores
  them in RRDtool RRD (which means Round-Robin Database, by the way) files.

- badger.conf, which you use to configure badger.

- and if you use Alteon Web Systems equipment, you get makedefs.pl, which
  walks your switches' SNMP tree and creates bronc's configuration files
  for you automatically!


How to install
. . . . . . . . . . . . . . . . . . . .

0a) Build a working installation of Apache with mod_perl support.
	http://www.apache.org/httpd.html   [for Apache]
	http://perl.apache.org   [for mod_perl]

0b) Install the Time::HiRes and URI::Escape perl modules. For these, go to:
	http://search.cpan.org

0c) Install Mason (which requires the MLDBM and Data::Dumper perl
    modules as well).
	http://www.masonhq.com   [for Mason]
	http://www.perl.com/CPAN/modules/by-authors/id/GSAR/
			         [for MLDBM and Data::Dumper]

0d) Install the latest version of RRDtool.
	http://ee-staff.ethz.ch/~oetiker/webtools/rrdtool/

0e) Install the perl modules that came with RRDtool by running:
	make site-perl-install
    in the RRDtool source directory.

0f) Make sure you have a working version of Perl, 5.004 or later

0g) If you want to be able to make neat little thumbnails of
    your graphs, you are going to need to install ImageMagick
    and it's accompanying Perl libs (called PerlMagick).
    ImageMagick is available here:

    http://freshmeat.net/appindex/1998/01/11/884574457.html

    Also, see the note below about uncommenting the line in
    Grapher.pm when you've installed ImageMagick.

1) Unpack the distribution into a temporary directory. (duh)

2) Decide on a home for bronc.  I suggest creating a new user
   account called 'bronc' and making it's home directory /home/bronc
   or /opt/bronc.  Allow for about 1Mb per item monitored.  Monitoring
   the input and output packet rates on one router is two items, thus
   the databases will take up about 2 megabytes total.  I have about 100Mb
   of databases in my monitoring installation.

3) Once you've chosen a home for bronc (from now on, we'll call it "BRONCHOME")
   do this:

	su - bronc (or login as bronc)
	cd /home/bronc  (or whatver you're using for BRONCHOME)
	mkdir rrd_files
	mkdir etc
	mkdir bin
	mkdir logs      (for the web server logs)
	mkdir data      (for Mason's cache)

4) Open up badger.pl in your favorite text editor and do the following:
	- set the first line to the location of your perl binary
	- change the lines under "C O N F I G U R A T I O N" to reflect
	  the directories that you just made for bronc.
	- save.

5) copy badger.pl to (BRONCHOME)/bin

6) Edit the community_map file.  This file contains the hostnames and the
   SNMP public (read-only) community names of all the things that you are
   going to monitor.  Please don't be a fool and set your public community
   string to the default "public".  Choose something a little less easy to
   guess.  :-)  While you're at it, make sure that you have SNMP enabled on
   all the devices that you want to monitor.  Look at your equipment
   documentation if you need more assistance.

7) copy community_map to (BRONCHOME)/etc

8) Edit the badger.conf file.  Each line represents one measurement that you
   will be querying.  For example, if you are querying the input and output
   bitrates of a router, you'll want to split these over two lines, one for
   input and another for output.  Each line contains four fields.  Look at
   my example as a reference.  The first field is the frequency that your
   devices with be queried, in seconds.  I've chosen 30 seconds for my 
   setup.  It provides high resolution for things like routers but doesn't
   tax system resources too much.  The second field is the name of the
   round-robin database file (RRD) that will hold the data gathered from this
   host.  Choose a short but descriptive name.  The name must end in '.rrd'.
   See my example for a reference.  The third field is the hostname or IP that 
   will be queried via SNMP.  Finally, the fourth field contains the OID of
   the parameter that you will be querying.  It is important that you use a
   numeric OID instead of a character string.  Use 'snmptranslate' from the
   UCD-SNMP package to find the OID if you don't already know it.  If you don't
   know what an OID is, I suggest reading up on SNMP at the UCD-SNMP site or
   the MRTG site.  They are:
		http://ucd-snmp.ucdavis.edu/
		http://ee-staff.ethz.ch/~oetiker/webtools/mrtg/mrtg.html

   IMPORTANT NOTE:
   ---------------
   While badger can accept several OIDs on one line and store their data in
   a single RRD file, bronc does not support this so don't do it.

9) copy badger.conf to (BRONCHOME)/etc

10) Edit the Grapher.pm file.  Go to the "C O N F I G U R A T I O N" section
    and set the path to your (BRONCHOME)/rrd_files directory.
    
    *** IF YOU'VE INSTALLED THE IMAGEMAGICK PERL LIBRARIES, READ ON ... ***
 
    If you have ImageMagick and it's Perl libraries properly installed,
    you can uncomment the two lines at the top of Grapher.pm that enable
    ImageMagick support.  If you don't have this enabled, thumbnails will
    not be scaled down.

11) Assuming Apache is installed in /usr/local/apache ...
    mkdir /usr/local/apache/lib/perl/BRONC
    cp Grapher.pm /usr/local/apache/lib/perl/BRONC
    chown -R bronc:bronc /usr/local/apache/lib/perl/BRONC
       [or whatever bronc's username and group are]

12) Look at the apache httpd.conf under apache_stuff/ and
    add the contents of httpd.conf to your httpd.conf, adjusting 
    hostnames and paths to reflect your site.

13) Edit apache_stuff/mason_handler.pl and look in the 
    "C O N F I G U R A T I O N" section for the pathnames.
    Change them if necessary to reflect your site's setup.

14) Copy apache_stuff/mason_handler.pl to /usr/local/apache/lib/perl/
    [adjusting for the path of your apache installation if necessary]

15) If you've never set up mod_perl before, copy apache_stuff/startup.pl
    to /usr/local/apache/lib/perl [or wherever you install apache's perl
    stuff].  If you *have* set up mod_perl before, compare my startup.pl
    to yours and make changes as necessary.

16) chmod 755 /usr/local/apache/lib/perl/*.pl

17) copy breakdown.html to (BRONCHOME)/htdocs

18) Edit "chef" and look under "C O N F I G U R A T I O N".  Set URL_BASE
    to reflect the location of Grapher.pm as specified in your httpd.conf.
    If you used my httpd.conf suggestions, the default is probably ok.

19) copy chef to (BRONCHOME)/htdocs

20) /usr/local/apache/bin/apachectl stop ;
    /usr/local/apache/bin/apachectl start
           and make sure it comes back up ok...

20) Now it's time to make the RRD files...



Setting up the RRD files
. . . . . . . . . . . . . . . . . . . . .

Bronc is a front-end to RRDtool.  To make the purty graphs, RRDtool
relies on data stored in round-robin databases called RRD's.  The neat
thing about the RRD's is that once you create them, they never grow in
size.  As new values are added, the oldest values go away.  

We need to make a RRD for each SNMP value (OID) that we are going to monitor.
Part of creating an RRD is choosing a "target" name.  A target name is a
keyword that identifies the data you are storing.  If you are going to be
storing the temperature of a router in your RRD file, I'd choose "temp"
or "temperature" as the target name.  Target names need to be one word, without
spaces or punctuation of any kind.  The more straight-forward, the better.

**IMPORTANT*****************************************************************
At this point, it is critical that you have read the RRDtool Tutorial
by Alex van den Bogaerdt.  It can be found off of the RRDtool web site at:

http://ee-staff.ethz.ch/~oetiker/webtools/rrdtool/tutorial/rrdtutorial.html

If you write me with a question that is answered by this tutorial, I will
*ignore* you.
****************************************************************************

To make an RRD, we use rrdtool with the 'create' method.  Here's how:

First, run the epochtime script that came with Bronc:

$ ./epochtime.pl

and jot down (or cut-n-paste) the number that it spits out.

Next, run a command like this (without the []'s)

$ rrdtool create [insert rrd filename here] --step 30 \
  --start [insert the timestamp you just generated here] \
  DS:[insert your target name here]:[GAUGE|COUNTER|DERIVE|ABSOLUTE]:30:0:U \
  RRA:AVERAGE:0.5:1:5760 \
  RRA:MAX:0.5:1:5760 \
  RRA:MIN:0.5:1:5760 \ 
  RRA:AVERAGE:0.5:10:4032 \
  RRA:AVERAGE:0.5:60:5376 \ 
  RRA:AVERAGE:0.5:120:8064 \
  RRA:MAX:0.5:120:8064 \
  RRA:MIN:0.5:120:8064 

On the first line, you need to specify the full path to the RRD file that
you are creating.  If you keep your rrd files in /home/bronc/rrd_files and
you are going to store inbound packet counts for your "core" router, you might
call your rrd file "/home/bronc/rrd_files/core-inbound.rrd".

The DS: line is what we need to focus on next.  You substitute your target
name where instructed.  Next, you need to pick your data type.  Your data
type (also called "DST") can be one of four values:

	GAUGE  - which is used for counting things like temperature or
                 the number of people in a room, etc.

	COUNTER - which is used when an SNMP query returns an ever-increasing
		  value.  Most routers measure throughput by incrementing a
	          packet count.  When you use COUNTER, RRDtool will look
	          at the previous value and determine a rate of change.  This
		  is how RRDtool determines bits/sec sent through a router that
	          only returns the total number of bits sent. 

	DERIVE - used for rate-of-change calculations.  See the rrdcreate
	         manpage and the RRDtool tutorial for more info on DERIVE.

	ABSOLUTE - used for counters that get reset to 0 every time you read
                   them.

Most of the time, you'll probably want to use GAUGE or COUNTER.

The RRA lines define a reasonable storage setup.  They store data for up
to two years and give you access to the minimum and maximum values for
the two year timespan.  Please see the RRDtool tutorial.  You might want
to modify these to suit your needs.



Now that you have created RRD files for every target in your badger.conf,
it's time to create your definitons file.


Creating the "definitions" file
. . . . . . . . . . . . . . . . . . . . . . . . . . .

This part sorta sucks if you've never done it before.  Please look at my example
definitions file for help.

The most important thing here is to be VERY, VERY careful not to make any
syntax errors.  If you make a syntax error, you will get broken graphs and
the error log that "Grapher" spits out will be of little or no use in 
identifying the problem.  If you do have problems, double-check your syntax
against mine.  Make sure that you have trailing commas after all }'s, )'s
and ]'s.  Make sure you close all quotations.

For every type of graph that you generate, you will need one definition in the
definitions file.  You may have noticed that a definition is a Perl data 
structure.  If you want to graph the temperature of your router versus
the inbound and outbound traffic, you will need a definition for this graph.
If you also want to graph the temperature by itself, you'll need another
definition.  

Please see my sample definitions file for examples and documentation.

*** IMPORTANT ***
Once you are satisfied with your definitions file, copy it to (BRONCBASE)/htdocs



Creating a page of graphs
. . . . . . . . . . . . . . . . . . . . . . . . . . .
 
Now that you are happy with your defintions file and have proofread it at least
five times and said a prayer to your supreme being of choice, it's time to set
up an HTML page to contain some graphs. 

Please look at the "myrouter.html" file for example.

These HTML pages that you create need to go in the same directory as the rest
of Bronc's HTML and Mason components... (BRONCHOME)/htdocs

Basically, to call a graph, you use this tag in the HTML:

<& chef, ipass => $conf_ref, definition => 0 &>

You change the number after the "defintion =>" to reflect the
definition "number" of the graph you want.  The defintion "number"
is simply the order of the definition in the definitions file.  The
first graph definition is always '0', the second is '1' and so on...

I've set up a "breakdown" template called breakdown.html.  You link to 
this file in an <A> tag and it will automatically give you a breakdown
of your graph definition over a several useful timespans.  To make the graph
you just called link to a break down page, enclose your call like this:

<A HREF="http://bronc.blueaspen.com/bronc/breakdown.html?definition=0&dfile=definitions">
<& chef, ipass => $conf_ref, definition => 0 &></a>

Notice the "?defintion=0" in the HREF.  Change the definition number to match
the definition of the graph whose breakdown you want.

***IMPORTANT***
both "myrouter.html" and "breakdown.html" contain hard-coded references to
my web server.  To get these files to work, you must change the references
to point to your web server.


------------------------------------------------------------
That's it!  You should be set at this point!!!!
------------------------------------------------------------



Troubleshooting
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Chances are, you're encountering some problems with broken graph images.
Here's a list of things to check:

In your definitions file...
        - ***IMPORTANT*** make sure that you have no "&", "+", "?", or "="
          symbols in any of your "title", "label", or "text" strings.
	- Did you remember to terminate all quotation mark with another
	  quotation mark?
	- Did you remember to put commas after each }, ], and ) ?
	- Make sure that for every { there is a corresponding }
	  and do the same for [ ], and ( ).
	- Make sure that the target names specified in your definition
	  match targets in your RRD file.  To view the contents of your
	  RRD file, do this:
		rrdtool dump hazzard-inbytes.rrd | more
	  and look for the target name between <name> and </name>.
	- Make sure that the .rrd filenames specified in the definitions
	  file are really present in the (BRONCHOME)/rrd_files directory.
	- if you are doing any math with composite_defs, make sure that
	  the targets referenced in the RPN expression actually exist in
	  the rrdfile and that your RPN is valid
	- check /tmp/GRAPHER.PM-ERROR.log for possible descriptions of
	  the problem.
	- double and triple-check your syntax.  
