Ryfujk

I have a NagiosXi server monitoring 631 services on 63 hosts. Every seven hours the load on the server spikes up to 20ish and then gradually falls back to near-0.

There are no cron jobs running every 7 hours.

The server has 8 cores and 2GB RAM. The RAM is not an issue, it still sits at 1GB free during the spikes, and upping it to 4GB makes no difference. The server was also migrated to a new host a week or so ago with no changes.

We also have scheduled downtime on 17 of the hosts being monitored so they are only monitored during 6am-6pm Mon-Fri, this seems to make no difference to the load spikes.

Most checks are done on Windows servers, using check_wmi_plus.

During load spikes, I tend to see 5-8 instances of check_wmi_plus.pl using 2-3% cpu, and a handful of httpd processes using the same, but nothing stands out as using a lot of cpu. Those processes also roll over quite fast so they are not hung or taking an unusual long period of time. The Service Check Execution Time in NagiosXi Performance Monitor tends to peak at ~5.5s with averages around 1s.

Can anyone suggest a possible cause, or how I can further troubleshoot this?

A high load does NOT necessarily mean that you are using high levels of CPU only it only provides the number of process at a snapshot in time that are ready to run and receive CPU time but not how much of it.

Nagios does spin off a lot of processes rapidly depending on how you have set its monitoring schedules and at times will cause a spike as it starts a lot of processes running as fast as possible, but they might not require very much CPU or go immediately into a sleep/wait state.

BTW, if you disable NOTIFICATIONS in Nagios, this does not stop it from continuing to monitor a given host or service.

Lower the rhel/centos defaults prefork settings in the default /etc/httpd/conf/httpd.conf to something more realistic.

Use tools like apachebuddy.pl & apachetuner.sh to do the math on memory per process fork. allow more memory for other process on the system (mysql/postgresql/php) and reduce the MaxClient and MaxRequestChild.

I experienced this after the upgrade to 2014R1.1 from 2012R2.9. not sure if the latest version of XI2014 requires more resources for the web frontend.

This morning after lowering my settings, I noticed my load spikes are smaller, and navigating through the interface doesn't give me the grey unhappy face screen using forward and back buttons in browser. does this weirdness in the interface seem similar?

One last item, I'm looking at now, is what rhel modules in this default httpd.conf file are required. I see no sense in loading default modules if not needed. This server is a PROD enterprise server at my place of business with thousands of checks, so it needs to be solid.

UPDATE:

run

# service mysqld stop

# sh /usr/local/nagiosxi/scripts/repair_databases.sh 

# service mysqld start

or optimize tables while online via

# mysql -u root -p

mysql> use nagios;

list your tables

mysql> show tables;

then

mysql> optimize table $TABLENAME;

mysql> optimize table $TABLENAME;

mysql> optimize table $TABLENAME;

...

mysql> use nagiosql;



**list your tables**



mysql> show tables;

then

mysql> optimize table $TABLENAME;

mysql> optimize table $TABLENAME;

mysql> optimize table $TABLENAME;

...

do this for all tables.

If you can stop the service for the couple of minutes, then do it via nagiosxi script. if you can't until a later time... do it online, but expect the interface to be a bit slow until queries are re-ran. It maybe also beneficial to flush your query cache

mysql> FLUSH QUERY CACHE;

http://assets.nagios.com/downloads/nagiosxi/docs/Repairing_The_Nagios_XI_Database.pdf

this is due to how kernel calculates load. see the source:
https://github.com/torvalds/linux/blob/master/include/linux/sched/loadavg.h
and you will get something like this: #define LOAD_FREQ (5*HZ+1)

LOAD_FREQ is the interval the kernel collects cpu load. Note that there is a minor shift with the value of 0.001s. So it take 5* 1000 *5.001 seconds to drift back to a multiple of 5 seconds. 25005/ 3600 is around 7 hours.

so I bet the system forks shourt tasks periodically and just gets "caught" by the kernel every 7 hours.