Hopefully this will clarify some problems users were having with the check_cluster command. The check_cluster command is great for checking all types of clusters. However, I have found the online documentation (gathered by a Google search) to only contain the same information.

Once again thanks for the clarification and hopefully members of the community benefit from this exchange.

You have fallen foul to the standard threshold definition. -c 1:3 means “give me a critical when the value falls outside 1 to 3″. Therefore, having 0 data points in a non-OK state will give a critical alert.

You can either run
* -c 0:2, so it will alert when it gets to 3
* -c @3:, where the @ means to alert from 3 onwards

My tests:


$ ./check_cluster -s -d 0,0,0 -c 0:2
CLUSTER OK: Service cluster: 3 ok, 0 warning, 0 unknown, 0 critical
$ ./check_cluster -s -d 0,0,2 -c 0:2
CLUSTER OK: Service cluster: 2 ok, 0 warning, 0 unknown, 1 critical
$ ./check_cluster -s -d 2,0,2 -c 0:2
CLUSTER OK: Service cluster: 1 ok, 0 warning, 0 unknown, 2 critical
$ ./check_cluster -s -d 2,2,2 -c 0:2
CLUSTER CRITICAL: Service cluster: 0 ok, 0 warning, 0 unknown, 3 critical

$ ./check_cluster -s -d 0,0,0 -c @3:
CLUSTER OK: Service cluster: 3 ok, 0 warning, 0 unknown, 0 critical
$ ./check_cluster -s -d 0,0,2 -c @3:
CLUSTER OK: Service cluster: 2 ok, 0 warning, 0 unknown, 1 critical
$ ./check_cluster -s -d 2,0,2 -c @3:
CLUSTER OK: Service cluster: 1 ok, 0 warning, 0 unknown, 2 critical
$ ./check_cluster -s -d 2,2,2 -c @3:
CLUSTER CRITICAL: Service cluster: 0 ok, 0 warning, 0 unknown, 3 critical


I propose adding this to the help output for check_cluster:

Examples:
check_cluster -s -d 2,0,2,0 -c @3:
Will alert critical if there are 3 or more data points in a non-OK state


Would that help?

Ton

Ben uses “lsof” to identify directories being impacted the hardest on the Nagios host. lsof is a great tool and seeing how Ben uses it is worth the read. He also mentions how to improve performance with proper scheduling of tasks.

I am currently using a ping command to check a system’s health, and you can read the article here: Nagios: How to check a host, that for security reasons has ping disabled. When you process a ping command locally by pinging a loopback address you are engaging the operating system. What we are concentrating on here is the Nagios host’s ability to connect to a client, run a command and receive a response.

A “ping” from the Nagios host to a client does not engage the operating system. It engages the hardware, I do believe at the Networking layer of the OSI model.

System Administrators do have a lot of process to check throughout the enterprise. Using a service check for a check_command reduces the number of commands running against a host by 1. So, by eliminating just one command you have in reality reduced the network load by 2000 commands in a large host environment.

Thanks for your question and I hope my thoughts in regard to the “ping” command are a little clearer.

Thanks for your reply too.

I understand your point but still, for me it was quite obvious that you can’t say “processes are running because your ping is OK”.

As a SysAdmin you have so much more things to check.

Yes.. I like the way the Nagios Community is moving ! So I’m quite happy to participate.

Now, if you want to use the “ping” command to check network latency, then that is fine; however, “ping” does not tell you if the operating system if functioning properly. As a Linux/Unix Systems Administrator, I have seen countless times when the hardware will return an “OK” status, but the software is not functioning properly using the ping command. To see this point in action, launch a continuous “ping” command against a host and bounce the host. I can assure you the ping will reply back much sooner than you are able to log into the host. Even longer if the host has databases to start.

My concerns rest with the end user being able to perform his or her tasks on the server. Using a service check will show a problem with the operating system. If the NRPE daemon times out with a service check, then I need to investigate deeper the problem. Many times I have discovered the xinetd daemon was not running after a system restart. Other times I have discovered a process consuming enough resources to slow the system down considerably. Using a “ping” command to check the host would have never revealed the problem.

In regard to consumption of bandwidth, well the comment points and basic mathematics are valid. However, I am considering the other traffic flowing to and from servers. My organization’s stores have very narrow bandwidth through which must flow credit card validations, pricing and inventory updates as well as other Nagios service checks that are occurring throughout the course of a business cycle. If I can obtain valuable monitoring information, like checking directory size or checking the “sendmail” process, wouldn’t that be more advantageous for checking a host’s up/down status?

The article focuses on maximization of Nagios’s service check ability. The “ping” command is very valuable; however, I would recommend only using it on systems that require minimal checks. If you really want the entire value of “ping” I would suggest using “tracert,” so you can identify where the latency is on the network. You also do not want Nagios checks to become an issue on the enterprise network or systems.

Thank you both for responding to this article. GREAT insight and both I and the community appreciate your posts.

I have exactly the same concerns as smetj when I read your article.

Pinging host is obviously important to check the network stack and less heavy than NRPE on the Bandwidth front.

Of course if you want to check resources on the OS, NRPE is a good choice but it’s an other question.

I also would like to mention that NRPE is running fine on SunOS

I’m sorry but I don’t really see the point of this article, except discovering NRPE.

Arnaud

After reading your article I have a couple of concerns I want to share:

• Not using a ping initiated from a monitoring server to a remote host makes you lose important information such as network response speed between a monitoring server and a remote host.
You will not be able to see degradation in the network link quality.

• You loose the ability to check if a remote host is available through the network from a monitoring perspective. You’re not testing the network path.
Granted when the host can’t be reached over network, your nrpe check will fail too, … But is this because of the network or because the nrpe daemon is down or malfunctioning on the remote host?
This way, its clunky and unclear to create a dependency between your host executed checks and the network connectivity check.

• *quote*In reality a ping will only tell you if (assuming you do not have any firewalls between the “pinger” and “pingee”) whether the network interface card (NIC) has power and is able to respond.
Ping tells nothing about the state of the operating system.*quote*
I agree you can’t determine the complete state of an OS through a simple ping, but it’s the OS network stack which “assigns” the IP address to your NIC which on it’s turn is used by a remote host to ping to. Merely powering a NIC will not suffice (for example have a look at http://en.wikipedia.org/wiki/Wake-on-LAN#How_it_works) to get an ICMP reply. The OS network stack must be able to reply.

• Following your philosophy you will end up in 2 ways of determining a remote system is available on the network as you can’t implement the same approach for routers, switches, firewall, whatever device which can’t run the NRPE daemon. Although this doesn’t look like a big deal, in a very large network you want to make sure your tests results should have the same meaning across devices.

• *quote*Ping also consumes bandwidth. Bandwidth usage may appear to be negligible in regard to a small enterprise ( less than 100 host). However, what about larger enterprises?
In order to be proactive in regard to host issues, the enterprise should perform a host-alive check on every server, switch and router every five minutes. Now we can start to see the bandwidth problem.*quote*

The default ping check in Nagios is: check_ping -H ‘remote_host’ -w 5000,100% -c 5000,100% -p 1
When executed, this check consumes 196 bytes of data.
When executed every 5 minutes, 1 host will consume 94080 bytes per day.
Having a fairly large setup of let’s say 2000 hosts this makes 188160000 bytes on icmp checks per day, which equals to merely 179.4 megabyte per day.

If you send 4 packets instead of one for the check_ping check in Nagios you end up with 717.6 Mb per day, which still is NOTHING if you have an infrastructure capable of hosting 2000 devices!

I didn’t try out how much bandwidth your proposal consumes but it must be most definitely more, as you’re working with a tcp session check_nrpe doing sending a request to the nrpe_daemon which on its turn replies back. That kind of defeats a big chunk of your purpose.

-= Stefan

(For those interested: I used OpenID to log into this site, which seems to result in a very ugly username. I cannot change it.)