Automate Nagios configuration with Puppet – Part 3

In this third post about Automate Nagios configuration with Puppet, we will add services to our managed hosts, and streamline the monitoring of them with new Nagios checks.
It’s a continuation of the previous labs: Automate Nagios Configuration with Puppet – Part 1 and Part 2

First we will deploy the Apache daemon to our managed hosts. For this let’s create the httpd module, with 2 sub-classes, one for installing and enabling the service, and the other to copy a test file from the Puppet Fileserver.

class httpd {
    include httpd::install
    include httpd::testfile
}
class httpd::install {
    package { [ httpd, php ]:
        ensure => installed, }
    service { httpd:
        ensure => running,
        enable => true,
        require => Package[httpd],
    }
}
class httpd {
    include httpd::install
    include httpd::testfile
}
class httpd::testfile {
    file { "/var/www/html/test.html":
        mode => 440,
        owner => apache,
        group => apache,
        source => "puppet:///modules/httpd/test.html"
    }
}
test page</pre>
<h1>Test Page</h1>
<pre>

To define the new Nagios checks, we will use a sub class on the Nagios modile already built in the previous posts. We are using a check_http command to connect remotely to the test page, and a check_procs command run via NRPE to verify that the httpd processes are present.

class nagios::target::httpd {
   @@nagios_service { "check_http_${hostname}":
        check_command => "check_http!-u /test.html",
        use => "generic-service",
        host_name => "$fqdn",
        notification_period => "24x7",
        service_description => "${hostname}_check_http"
   }
   file_line { "command_check_httpd":
        line => "command[check_httpd]=/usr/lib64/nagios/plugins/check_procs -C httpd -c 1:",
        path => "/etc/nagios/nrpe.cfg",
        ensure => present,
        notify  => Service["nrpe"],
   }
   @@nagios_service { "check_httpd_${hostname}":
        check_command => "check_nrpe!check_httpd",
        use => "generic-service",
        host_name => "$fqdn",
        service_description => "${hostname}_check_httpd"
   }
}

Finally we add the corresponding classes to our nodes using the manifest file

node 'core.example.local' {
    include nagios::monitor
    include nagios::nrpe-command
}
node 'web01.example.local' {
    include nagios::target
    include nagios::nrpe
    include httpd
    include nagios::target::httpd
}

Once the puppet agents run on the managed node and nagios server, the monitoring of Apache will be ready:
services_httpd

Advertisements

Automate Nagios configuration with Puppet – Part 2

In this second post about Automate Nagios configuration with Puppet, we will include monitoring through NRPE.
As in the first post, the idea of the lab is use the simplest resources on puppet. In this case we are introducing a new feature: puppet modules download from Puppet forge.

Add module stdlib, used in the lab for editing the nrpe configuration file in each monitored server

puppet module install puppetlabs/stdlib

Create new clasess for NRPE:
A class for nrpe installation and configuration edit, and new service definitions for the default checks included in the CentOS nrpe package.

vi /etc/puppet/modules/nagios/manifests/nrpe.pp
class nagios::nrpe {
    package { [ nrpe, nagios-plugins, nagios-plugins-all ]:
        ensure => installed, }
    service { nrpe:
        ensure => running,
        enable => true,
        require => Package[nrpe],
    }
   file_line { "allowed_hosts":
        line => "allowed_hosts = 127.0.0.1,192.168.112.14",
        path => "/etc/nagios/nrpe.cfg",
        match => "allowed_hosts",
        ensure => present,
        notify  => Service["nrpe"],
   }
   @@nagios_service { "check_load_${hostname}":
        check_command => "check_nrpe!check_load",
        use => "generic-service",
        host_name => "$fqdn",
        service_description => "${hostname}_check_load"
   }
   @@nagios_service { "check_total_procs_${hostname}":
        check_command => "check_nrpe!check_total_procs",
        use => "generic-service",
        host_name => "$fqdn",
        service_description => "${hostname}_check_total_procs"
   }
   @@nagios_service { "check_zombie_procs_${hostname}":
        check_command => "check_nrpe!check_zombie_procs",
        use => "generic-service",
        host_name => "$fqdn",
        service_description => "${hostname}_check_zombie_procs"
   }
   @@nagios_service { "check_users_${hostname}":
        check_command => "check_nrpe!check_users",
        use => "generic-service",
        host_name => "$fqdn",
        service_description => "${hostname}_check_users"
   }
}

And a class to add the nrpe plugin and a command definition for it on the Nagios server

vi /etc/puppet/modules/nagios/manifests/nrpe-command.pp
class nagios::nrpe-command {
  package { "nagios-plugins-nrpe" :
        ensure => installed,
  }
  nagios_command { 'resource title':
    command_name => 'check_nrpe',
    ensure       => 'present',
    command_line => '/usr/lib64/nagios/plugins/check_nrpe -H $HOSTADDRESS$ -c $ARG1$',
  }
  file_line { "nagios_command.cfg":
        line => "cfg_file=/etc/nagios/nagios_command.cfg",
        path => "/etc/nagios/nagios.cfg",
        ensure => present,
        notify  => Service["nagios"],
   }
   file { "nagios_command":
        mode => 644,
   }
}

Include the new classes on the nodes manifests

vi /etc/puppet/manifests/nodes.pp
node 'core.example.local' {
      include nagios::monitor
      include nagios::nrpe-command
}
node 'web01.example.local' {
      include nagios::target
      include nagios::nrpe
}

Finally run the agent on the monitored server and then on puppet master

puppet agent --test --server core.example.local

Check the result on the Nagios console:
services_nrpe

Automate Nagios Configuration with Puppet

This is a full lab to show the automation of nagios configurations with Puppet, using CentOS 6.5. It’s based on the examples of the Exported Resources documentation on the PuppetLabs website: http://docs.puppetlabs.com/guides/exported_resources.html
It’s also published in Github: https://github.com/gfolga/puppet-nagios-lab

Nagios / Puppet Server core
Domain name example.local
Monitored server web01
Puppet version 3.4
Additional Puppet packages PuppetDB, puppet-dashboard
Nagios version 3.5
Additional Nagios packages nagios-plugins, nagios-plugins-all

Puppet/Nagios server configuration

Do a minimal install of CentOS
Configure Networking
Update packages

yum update -y

Add puppet & epel repositories

rpm -ivh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
rpm -ivh https://yum.puppetlabs.com/el/6/products/x86_64/puppetlabs-release-6-7.noarch.rpm

Install puppet packages

yum install -y puppet puppet-server puppetdb puppet-dashboard puppetdb-terminus

Configure PuppetDB and Puppet Master:

cat <<END > /etc/puppet/puppetdb.conf
[main]
   server = core.example.local
   port = 8081
END
cat <<END >> /etc/puppet/puppet.conf
[master]
  storeconfigs = true
  storeconfigs_backend = puppetdb
  reports = store,puppetdb
END
cat <<END > /etc/puppet/routes.yaml
---
master:
  facts:
    terminus: puppetdb
    cache: yaml
END

Run the SSL Configuration Script

 /usr/sbin/puppetdb ssl-setup 

Enable and start services

puppet resource service puppetdb ensure=running enable=true
puppet resource service puppetmaster ensure=running enable=true

Test puppet agent and puppetdb

puppet agent --server core.example.local -t

Module for install and initalization of nagios

cd /etc/puppet
mkdir modules/nagios
mkdir modules/nagios/manifest
chmod 755 modules/nagios/
chmod 755 modules/nagios/manifests/
vi modules/nagios/manifests/monitor.pp
class nagios::monitor {
    package { [ nagios, nagios-plugins, nagios-plugins-all ]: ensure => installed, }
    service { nagios:
        ensure => running,
        enable => true,
        #subscribe => File[$nagios_cfgdir],
        require => Package[nagios],
    }
    # collect resources and populate /etc/nagios/nagios_*.cfg
    Nagios_host <<||>>
    Nagios_service <<||>>
}
chmod 644 modules/nagios/manifests/monitor.pp

Monitored server configuration

Do a minimal install of CentOS
Configure Networking
Update packages

yum update -y

Add puppet & epel repositories

rpm -ivh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
rpm -ivh https://yum.puppetlabs.com/el/6/products/x86_64/puppetlabs-release-6-7.noarch.rpm

Install puppet packages

yum install -y puppet

Add a host entry on the puppet server for the monitored host

puppet resource host web01.example.local ip="192.168.112.15"

Add a host entry on the monitored host for the puppet server

puppet resource host core.example.local ip="192.168.112.14"

Initialize agent

puppet agent --test --server core.example.local

Sign cert on the puppet server

puppet cert sign web01.example.local

Create puppet module file and include it in the node definition

vi modules/nagios/manifests/target.pp
class nagios::target {
   @@nagios_host { $fqdn:
        ensure => present,
        alias => $hostname,
        address => $ipaddress,
        use => "linux-server",
   }
   @@nagios_service { "check_ping_${hostname}":
        check_command => "check_ping!100.0,20%!500.0,60%",
        use => "generic-service",
        host_name => "$fqdn",
        notification_period => "24x7",
        service_description => "${hostname}_check_ping"
   }
}
vi /etc/puppet/manifests/site.pp
import "nodes"
vi /etc/puppet/manifests/nodes.pp
 node 'core.example.local' {
      include nagios::monitor
    }
 node 'web01.example.local' {
      include nagios::target
    }

Run the agent on the monitored server and then on puppet master

puppet agent --test --server core.example.local

Add nagios_host.cfg and nagios_service.cfg to the main configuration file of nagios

cat <<END >> nagios.cfg
cfg_file=/etc/nagios/nagios_host.cfg
cfg_file=/etc/nagios/nagios_service.cfg
END
chmod 644 /etc/nagios/nagios_host.cfg
chmod 644 /etc/nagios/nagios_service.cfg

Restart Nagios & Apache, set nagiosadmin password

service nagios restart
service httpd restart
htpasswd -c /etc/nagios/passwd nagiosadmin

Access to Nagios and verify that the monitored host and the service defined appear on the nagios console
hosts services

That’s all. Just adding the class nagios::target to new servers, and puppet will take care of the nagios definitions.
In a next post I will extend the lab with advanced monitoring of the servers using NRPE.