http://bit.ly/c2UA4O

Challenges:

1. The Xen environment utilizes root SUID
2. The “xm” command used to list the Xen clients is also used for other activities with the Xen environment, including destroying Xen clients.
3. SOX concerns in regard to Xen client functionality.
4. Maintaining enterprise standards in regard to deploying scripts.
5. Using current Nagios monitoring and notification standards.
6. Do not send notifications because the location of the Xen client is informational.

Steps towards Solution:

1. Deploy the Nagios NSCA daemon for passive checks.
2. Create the script and deploy to the Nagios libexec directory.
3. Place the script within the Root crontab.

Breakdown:

1. Make sure the nagios-plugins and NRPE daemon are deployed on the Xen client.
2. Install mcrypt packages.
3. Install the NSCA daemon.
4. Add the passive check to the Nagios host.
5. For more information on setting up the NSCA client please refer to: http://community.nagios.org/2009/06/11/nagios-setting-up-the-nsca-addon-for-passive-checks/
6. Create the monitoring script. Here is a sample KISS script written in Perl:

    $vhosts = $virtmach[0] . "  " . $vhosts .
   } } open (FH, ">$tempfile") or die $!;
   print FH "$host\t$svc\t$state\tVMs RUNNING: $vhosts\n" or die $!;
   my $send_to_nagios=`/opt/nagios/bin/send_nsca nagcorp1 -c
   /opt/nagios/etc/send_nsca.cfg < $tempfile`;
   close(FH);
   exit(0);
   }
   else {
   #no
   exit(0);
   }

   #!/usr/local/bin/perl -w
   use strict;
   use warnings;                                                                                         
   
   #Let us see if this server is running a virtual environment
   if (-e "/usr/sbin/xm"){
   #yes
   #Variables
   my @xm_cmd = `/usr/sbin/xm list`;
   my $host = `/bin/hostname -f`;
   my $svc = "nsca_check_vhosts";
   my $state =0;
   my $tempfile = "/opt/nagios/libexec/nsca_check_vhosts.tmp";
   my $xm_cmd;
   my @host;
   my $vhosts = " ";
   my @virtmach;
   my $virtmach;
   chomp($host);
   shift(@xm_cmd);
   foreach $xm_cmd(@xm_cmd){
           chomp($xm_cmd);
           @virtmach = split(/\s* /,$xm_cmd,6);
                   if($virtmach[1] > 0  && $virtmach[1] < 512){

7. Crontab entry on the Xen host:

   #Nagios check of virtual machines
   15,30,45,02 * * * * /opt/nagios/libexec/
   #

8. Done

This is pretty straight forward. There are other examples of how to accomplish this task on the Internet. My primary concern was security. Large entrerprised have security standards, so exceptions are a no-no. This setup takes into consideration the SUID nature of the “xm” command and does not provide the Nagios user with additional privilages that could be exploited by outside individuals.

Thoughts and or comments?

Mike Kniaziewicz, MIS

Topics include:  The Nagios future and Nagios and the Community.

http://bit.ly/c9s7PY

• Heavy I/O on /var/log/nagios
• Heavy reads on /etc/nagios during starts and restarts.
• Bandwidth: consider connection bonding if you have more than one network interface. Nagios will use bandwidth, but you also need to consider the users connecting to the instance via a web browser.
• Memory: Nagios will run with 32 MB of memory. However, you need to consider Nagios is a web-based application and has a caching mechanism and Apache’s caching mechanism.
• Revision Control System (RCS): You want to be able to catalog changes to the production Nagios system.

Nagios Installation

1. Open yast2. Open: Software → Software Management
   • Install Packages:
     Apache2
     Apache2-prefork
     Apache2-mod_php5
     nagios
     nagios-nsca
     nagios-nsca-client
     nagios-plugins
     nagios-plugins-extras
     nagios-www
     rcs – if the package was not installed during the operating system installation.
   • Press: “Continue” to accept additional packages. You should see packages like fping and libmcrypt to be added to the RPM configurations.
   • After the packages have installed, select: “No” to “install or remove more packages?”
   • Quit out of yast2.
   • The installation is complete. At this time you can start Nagios (/etc/init.d/nagios start) and it will work. You can also start Apache (/etc/init.d/apache2 start) and it will work as well. However, you will still need to “tweak” Nagios and Apache to receive the desired results.

Where are the files?
In order to have your system become fully operational you need to know where all the pieces are located. Here is a listing of where the Nagios components are located:

Host, Service and Nagios Configurations: /etc/nagios/
Web Interface Files: /usr/share/nagios/
Nagios Log and Archives: /var/log/nagios/
Nagios Lock File: /var/run/nagios/
Nagios Log Rotating: /etc/logrotate.d/nagios
Nagios Data Files: /var/lib/nagios/
Nagios Plugins: /usr/lib/nagios/plugins/
CGI Files: /usr/lib/nagios/cgi/

Configuring Nagios and Web Interface

1. Add a nagcmd group to the system. sudo /usr/sbin/groupadd -g 5544 nagcmd
2. Add a nagios user. sudo /usr/sbin/useradd -u 1209 -g 5544 -c “Nagios” -d /home/nagios -s /bin/bash -m nagios.
3. You will also need to add the user nagios to the wwwrun group. Sudo /usr/sbin/usermod -g wwwrun nagios
4. Give the nagios user a password. sudo passwd nagios (write this down, because you will need to know it later).
5. Start Nagios: sudo /etc/init.d/nagios start.
6. In /etc/apache2/conf.d/nagios.conf, comment out the Directive lines “deny from all.”
7. Restart apache2: sudo /etc/init.d/apache2 restart.
8. Open a web browser and go to your host’s URL: http://your-host-name.com/nagios. You will be presented with the basic Nagios screen. However, if you select any of the “Monitoring” links you will receive a permissions error message.
9. Create a .htaccess_users file under /usr/share/nagios and add the nagiosadmin user and create a password: sudo htpasswd2 -c /usr/share/nagios/.htaccess_users nagiosadmin
10. Add the .htaccess file path to /etc/apache2/conf.d/nagios.conf under each “Directive” category:
    AuthName “Nagios Access”
    AuthType Basic
    AuthUserFile /usr/share/nagios/.htaccess_users
    Require valid-user
11. Restart apache: sudo /etc/init.d/apache2 restart
12. Uncomment the following lines in /etc/nagios/cgi.cfg:
    authorized_for_system_information
    authorized_for_configuration_information
    authorized_for_system_commands
    authorized_for_all_services
    authorized_for_all_hosts
    authorized_for_all_service_commands
    authorized_for_all_host_commands
13. Restart Nagios: /etc/init.d/nagios restart
14. Refresh your browser window and you will be prompted to login. Login using the nagiosadmin user name and the password you created in step 9.
15. SECURITY CONSIDERATIONS: Change the ownership of /usr/share/nagios/.htaccess_users to wwwrun (apache user) and the permissions to 400. This will ensure Apache can read the file. Root will always be able to read and append to the file. Restart apache for the change to take effect.

That is all there is to installing Nagios on SLES 10 SP3 using Novell’s Nagios bundle. I am sorry to say that Novell is bundling Nagios version 2.6, but it is a start.

Enjoy and please provide comments and or insight.

Our scenario involves two physical hosts (in my case I have created two virtual hosts using Xen). One host will house the master (node1) and the other the slave (node2). If you already have an instance of MySQL running in support of Nagios, this procedure will enable you to create a slave without interrupting your current installation and established permissions.

NOTE: Read through these instructions to make sure you understand them, because you will want to perform this procedure as quickly as possible. Nagios is going to be trying to writing to the master server while the instance is stopped during our data migration process.

1. Identify node1.
2. On node1 log into MySQL: sudo mysql (provide password).
3. Create a replication user: CREATE USER ‘repl’@'%.your domain.com’ IDENTIFIED BY ‘your password’; . Make sure you set up a separate account for the replication user, because the userid and password will be in plain text in your master.info and my.cnf files.
4. Grant the repl user rights: GRANT REPLICATION ON ‘*.*’ TO ‘repl’@'%.your domain.com’; The reason why we use the ‘*.*’ is to allow replication of all databases and hosts on the master instance. We use ‘%.your domain.com’ so we can create a slave anywhere on our networks.
5. Once that has been completed we need to proceed to node2 in order to modify the my.cnf file. If MySQL is running on the potential slave instance, stop MySQL.
6. Set the server-id = 2 (or the next number in your series of slaves).
7. Set the master-host = node1 hostname (I use the IP address of the master host).
8. Set the master-user = repl.
9. Set the master-password = your password (remember, I mentioned earlier that the userid and password for replication will be in plain text).
10. Set the log-bin = mysql-bin (you need to set this flag in order to have the ability to fail-over to this MySQL instance).
11. Now you want to start node2′s instance of MySQL. You will receive some warnings, but the instance should start and try to begin replication.
12. Log into node2′s instance of MySQL: sudo mysql and run: “SHOW SLAVE STATUS;”  You will see that the MySQL instance is now a slave.
13. Now, run “STOP SLAVE;” This will stop the slave on node2 in preparation for our next steps. You can quit the MySQL session on node2 and return to a command prompt.
14. On node1′s r instance of MySQL run: “FLUSH TABLES WITH READ LOCK;” This will prevent the tables from updating so we can obtain a snapshot.
15. Log into another session on node1 so we can perform a backup of the databases. From the command prompt run: mysqldump –all-databases –master-data > mydbdump.db. The “–master-data” flag will capture all our required log markers.
16. We will now restore the data from node1 to node2: sudo mysql < mydbdump.db. Remember, we froze node1, so we are restoring up-to-date data to node2.
17. Once the restore has completed, log into the MySQL instance on node2 and run: “START SLAVE;”
18. On node1 run: “UNLOCK TABLES;” This will begin the master’s data processing once more.

That’s all there is too it. I use a restore because I want my slave (node2) to spend as little time as possible updating or syncing to node1. This will also save on server load and network bandwidth. To test if replication is working properly perform this test:

• node1: Log into MySQL and run: “CREATE DATABASE testing;”, then “SHOW DATABASES;” and you should see the new database “testing”.
• node2: Log into MySQL and run: “SHOW DATABASES;” if replication is working you will now see the “testing” database.

I hope that helps with your MySQL instance on Nagios or Nagios XI. For more information on MySQL replication please visit: MySQL Replication .

Comments and or other solutions? You can always shoot me an email for questions at mikhail@ebusinessjuncture.com.

Using check_bigip_pool is also straight forward. You will need a hostname (hostname for F5), community (generally public), software version and pool name. The check_bigip_pool will return the number of nodes found running on the F5. So, you will need to work with the warning and critical percentages. HINT: If check_bigip_pool finds one of two nodes running and you only require one node running, set the warning to 49 and critical to 25. When both nodes are down you will receive a critical alert and notification.

Here are the configuration steps for this setup:

Define what is to be monitored (NOTE: this is just an outline, my organizational setup monitors 5 clients and 7 ports per client):

Client					URL				Port	HTTP Response
webstore.<your domain>.com	http://webstore			        80	 OK
webstore.<your domain>.com	http://webstore.com/WatchDog.tem 	80	 OK – I am here
f5host.<your domain>.com		your pool name			count of nodes up

*Service definition are placed within a template file. I prefer to place service definitions within a file that is descriptive of the function. You will also need to create a separate service definition for each service check, because you need to create distinct service_descriptions section. You can leave blanks in the service description; however, I do not because it will remove your ability to acknowledge problems via email. I also used the “hosts” flag instead of hostgroups, because I have 4 webstore servers, but not all of them are looking at the same URLs and ports. Also include a notes section, we will be using it later for our notification email.

Service Definition (nagios_host:/path-to-nagios/etc/objects/services/srv_check_http.cfg):

define service{
        use                     generic-service
        hosts                   hostname
        notes                   description of the service being monitored.
        service_description    	check_webstore
        check_command        	check_nrpe!check_webstore
        max_check_attempts      3
        normal_check_interval   5
        retry_check_interval    1
        check_period            24x7
        contact_groups          webstore_admins
        }

*Notice the variation in the check_command definition? This is so you can run the command
on the Nagios host and be listed under the defined host. 

F5 Service Definition (nagios_host:/path-to-nagios/etc/objects/services/f5_checks.cfg):

define service{
        use                      generic-service
        hosts                    f5host
        service_description      webstore_business_station
        notes                    There are 0 nodes online for http://business-station.coat.com.
        check_command            check_webstore_business_station_pool!
        max_check_attempts       3
        normal_check_interval    5
        retry_check_interval     1
        notification_period      24x7
        check_period             24x7
        contact_groups           webstore_admins
        }

*Host definition is placed within a template. I have created a separate template for each classification of hosts, which makes it easier to find and update hosts. The host definition would be the same for the f5 server.

Host Definition (nagios_host:/path_to_nagios/etc/objects/hosts/webstore_servers.cfg):

define host{
        use                      webstore-servers   ; Name of host template to use
        host_name                your host's name
        address                  ip_address
        alias                    short name for your host
        notes                    What does the host do <br>LOCATION:<CONSOLE: >
        icon_image               sun_logo.gif
        icon_image_alt           Sun Host
        hostgroups               Add your host groups for standard monitoring and specialty
                                 monitoring
        notification_period      24x7
        }

*Here is how I defined the commands. For separation purposes, I prefer to the definitions within a distinct configuration file and give each command name a distinct definition. This way if one port is not accessible, you can address the one port issue and not assume the entire web application is down.

Command Definition (nagios_host:/path_to_nagios/etc/objects/commands/cmd_http.cfg):

define command{
        command_name    	check_webstore
        command_line    	$USER1$/check_http  -H $ARG1$ -u $ARG2$ -p $ARG3$
        }

F5 Command Definition (nagios_host:/path_to_nagios/etc/objects/commands/cmd_check_bigip_pool.cfg):

define command{
        command_name    	check_webstore_pool
        command_line    	$USER1$/check_bigip_pool -H $HOSTADDRESS$ -C $ARG1$ -S $ARG2$ -P $ARG3$
                                 -w $ARG4$ -c $ARG5$
        }

*I combine the contactgroup with the contact definition in its own file. You may not have the luxury of email aliases, so this helps me manage the contacts for a specific host or service.

Contact Definition (nagios_host:/path_to_nagios/etc/objects/contacts/webstore_admins.cfg):

define contactgroup{
        contactgroup_name          webstore_admins
        alias                      WEBSTORE ADMINS
        members                    webstore_admins
        }
define contact{
        contact_name               webstore_admins
        alias                      WEBSTORE Admins
        contact_groups             webstore_admins
        host_notifications_enabled      0
        service_notifications_enabled  	1
        service_notification_period     24x7
        host_notification_period        24x7
        service_notification_options   	w,u,c,r
        host_notification_options      	d,r
        service_notification_commands   notify-webstore-service-by-email
        host_notification_commands     	notify-linux-host-by-email
        email                           webstore_support@your_domain.com
        can_submit_commands             1
        }

*Notifications are fun. I keep separate command files, because different groups need to receive different information. Notice that the service_notification_commands in the contacts matches the notification definition command name. Within the definition, you will find the notes section after the “-s” flag and also the method I use for enabling problem acknowledgments via a link within email.

Notification Definition (nagios_host:/path_to_nagios/etc/objects/commands/
webstore_admins_email_notification.cfg):

# Linux 'notify-webstore-service-by-email' command definition
define command{
        command_name    notify-webstore-service-by-email
        command_line    /usr/bin/printf "%b" "If this is a CRITICAL problem, please insure the
ecommerce on-call person is notified.\n\nHostname:\t\t$HOSTNAME$\nService:\t\t$SERVICEDESC$\n
State:\t\t\t$SERVICESTATE$\nDate/Time:\t\t$LONGDATETIME$\nAdditional Info:\t$SERVICEOUTPUT$
\nAcknowledge:\t\thttp://nagios_host.your_domain.com/nagios/cgi-bin/cmd.cgi?cmd_typ=34&host=
$HOSTNAME$&service=$SERVICEDESC$\nAuthor:\t\t$SERVICEACKAUTHOR$\nAcknowledgement:\t$SERVICEACKCOMMENT$"
| /usr/bin/mailx -s "$HOSTALIAS$:$SERVICENOTES$" $CONTACTEMAIL$ -r core.admin@coat.com
        }

Email Notification Message Received:

Subject: <Your service notes>
Body:
If this is a CRITICAL problem, please insure the ecommerce on-call person is notified.

Hostname:               webstore1.coat.com
Service:                check_webstore_8150
State:                  CRITICAL
Date/Time:              Wed Jan 6 02:24:39 EST 2010
Additional Info:        CRITICAL - Socket timeout after 10 seconds
Acknowledge:	        http://nagios_host.your_domain.com/nagios/cgi-bin/cmd.cgi?cmd_typ=34
                        &host=$service=service_name
Author:
Acknowledgement:

Hopefully this helps with your monitoring of web sites. Any questions please feel free to email me at mikhail@ebusinessjuncture.com. Here is a link to the document HTTP Monitoring Write-up

If anyone has comments or another method to perform this task, please leave a comment.