Trim down load in server without killing your current running process.

It’s a fact that while we initiate some cpu or memory consumable process or job, the load in server goes high and sometimes the server become unresponsive. But what if the situation is same everytime especially in case of shared servers. The processess like pkgacct, copying, moving high sized files/folders can’t be ignored at the event of migration or other instances etc. Suppose if we initiate a pkgacct of one account and certainly the load goes high in server, so we may need to cut the pkgacct process we initiated for reducing the load. We can’t always do this whenever server load goes high, because it must be a real headache to initiate, cut, reinitiate etc process manually whenever the load varies in server.

You can do this task with the help of cpuwatch in cPanel servers. Just do the following.

=====
/usr/local/cpanel/bin/cpuwatch 3 /scripts/pkgacct cPanelusername
=====

In the above, whenever the load goes high over 3, the process “/scripts/pkgacct cPanelusername” pauses automatically and resumes as well when the load reduce under 3.

Nagios : CRITICAL – cannot connect to information_schema. Access denied for user

The error would be as follows while check_nrpe tries to check the mysql service in agent server.

============
CRITICAL – cannot connect to information_schema. Access denied for user nagios@nagiosIP.x.x (using password: YES)
============

The issue because nagios user was lagging access privilege to the DB which needs to be corrected. But please note that we should just give only access privilege to the nagios user for the purpose of checking/monitoring, do not give all other permissions/privileges to it.

Go to mysql prompt in agent server and give privilege for nagios user.

==================
mysql>use mysql;

mysql>grant all privileges on *.* to ‘nagios’@’nagiosip.x.x.x’ identified by password ‘d3fault’; (I got the password d3fault from nagios configuration file)

If you get error setting password ‘d3fault’ like “need to set 41 digit hexadecimal”, then convert the password to hexadecimal like below:

mysql> select password(‘d3fault’);
+——————————————-+
| password(‘d3fault’) |
+——————————————-+
| *DEC4F44D877B5BDC6434C9C5AFDD7BFA89D637E9 |
+——————————————-+
1 row in set (0.00 sec)

mysql>grant all privileges on *.* to ‘nagios’@’nagiosip.x.x.x’ identified by password ‘*DEC4F44D877B5BDC6434C9C5AFDD7BFA89D637E9’;

we should flush privileges since nagios user just only need to check it and doesn’t need other privileges literally.

mysql>flush privileges;

It should be displayed as follows with ‘N’ for every field. If you still seeis ‘Y’ everywhere, then try the following.

mysql>revoke all privileges on *.* from ‘nagios’@’nagiosip.x.x.x’; (:: nagiosip.x.x.x would be replaced with your exact nagios server IP)

Then it will be fine.

mysql> select * from mysql.user where User=’nagios’;
+————–+——–+——————————————-+————-+————-+————-+————-+————-+———–+————-+—————+————–+———–+————+—————–+————+————+————–+————+———————–+——————+————–+—————–+——————+——————+—————-+———————+——————–+——————+————+————–+————————+———-+————+————-+————–+—————+————-+—————–+———————-+——–+———————–+
| Host | User | Password | Select_priv | Insert_priv | Update_priv | Delete_priv | Create_priv | Drop_priv | Reload_priv | Shutdown_priv | Process_priv | File_priv | Grant_priv | References_priv | Index_priv | Alter_priv | Show_db_priv | Super_priv | Create_tmp_table_priv | Lock_tables_priv | Execute_priv | Repl_slave_priv | Repl_client_priv | Create_view_priv | Show_view_priv | Create_routine_priv | Alter_routine_priv | Create_user_priv | Event_priv | Trigger_priv | Create_tablespace_priv | ssl_type | ssl_cipher | x509_issuer | x509_subject | max_questions | max_updates | max_connections | max_user_connections | plugin | authentication_string |
+————–+——–+——————————————-+————-+————-+————-+————-+————-+———–+————-+—————+————–+———–+————+—————–+————+————+————–+————+———————–+——————+————–+—————–+——————+——————+—————-+———————+——————–+——————+————+————–+————————+———-+————+————-+————–+—————+————-+—————–+———————-+——–+———————–+
| nagiosip.x.x.x | nagios | *DEC4F44D877B5BDC6434C9C5AFDD7BFA89D637E9 | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | | | | | 0 | 0 | 0 | 0 | | NULL |
+————–+——–+——————————————-+————-+————-+————-+————-+————-+———–+————-+—————+————–+———–+————+—————–+————+————+————–+————+———————–+——————+————–+—————–+——————+——————+—————-+———————+——————–+——————+————+————–+————————+———-+————+————-+————–+—————+————-+—————–+———————-+——–+———————–+
1 row in set (0.00 sec)
==================

You are good to go now. Just go the page of MYSQL service in nagios and click on “Re-schedule the next check of this service”, the alert will be fine.

Nagios warning would also be fine.

————————————–
***** Nagios *****

Notification Type: RECOVERY

Service: MYSQL
Host: server.servername.com
Address: agentip.x.x.x
State: OK

Date/Time: ———–

Additional Info:

OK – 0.60 seconds to connect as nagios
—————————————-

“r1soft-setup –get-key” failed

I tried the command “r1soft-setup –get-key” to fetch the public key from CDP server and it showed up the following error unexpectadly.


====================
root@server [~]# r1soft-setup –get-key http://37.x.x.x
Couldnt connect to remote host
Unable to get key from remote CDP Server
====================

You can manually configure the key by doing the below steps:

1) Login on to CDP server front end panel
2) Click on “Configuration” on the left-bottom menu
3) Copy the Public Key from the page.
4) On the CDP agent(the server you are trying to configure to r1soft), touch a new file naming the IP of CDP server, like as follows:

#touch /usr/sbin/r1soft/conf/server.allow/37.x.x.x (37.x.x.x is the IP of CDP server)

5) Paste the Public Key of CDP server into this file without any white spaces.
6) Also white list the CDP server IP in the agent firewall.
7) Test the connection to CDP agent from r1soft panel while trying to add a new server to it. Go to Servers >> Add Server. If everything successfull, then you should get the following message status on the screen.


================
Successfully communicated with Agent!
Resolving Internet Address

Resolved hostname to: 82.x.x.x
Connecting to Agent

Successfully connected to Agent
Authenticating with Agent

Authenticated with Agent
================

r1soft | r1soft-setup –get-module failed

Sometimes you may end up with the following error while trying to install CDP agent in the server.


========================
root@server [~]# r1soft-setup –get-module
Checking for binary module
Waiting |
No binary module found
Gathering kernel information
Gathering kernel information complete.
Creating kernel headers package
Checking ‘/lib/modules/2.6.32-279.el6.x86_64/source/’ for kernel headers
Checking ‘/usr/src/kernels/2.6.32-279.el6.x86_64-x86_64/’ for kernel headers
Checking ‘/lib/modules/2.6.32-279.el6.x86_64/build/’ for kernel headers
Unable to find a valid source directory.
Please install the kernel headers for your operating sy
=========================

This needs kernel-devel and kernel-headers to be installed to resolve the issue. For that we can use the following command.


===========
yum install kernel-devel
yum install kernel-headers
===========

Please note that the above “yum install —” commands would try to install the kernel-devel and kernel-headers for the higher version of kernel installed in the server. Because of this I could not install it using yum straight away.


===========
root@server [~]# yum install kernel-devel
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirror.as29550.net
* extras: centos.hyve.com
* updates: mirror.sov.uk.goscomb.net
base | 3.7 kB 00:00
extras | 3.4 kB 00:00
r1soft | 951 B 00:00
updates | 3.4 kB 00:00
Setting up Install Process
Package kernel-devel-2.6.32-358.18.1.el6.x86_64 already installed and latest version
Nothing to do
===========

Higher version kernel “2.6.32-358.18.1.el6.x86_64” is already present in server, thats the reason we are hitting this error. For this, you may need to remove the rpm packages “using rpm -e” related to higher version kernel since it is not currently up and running.

“PLEASE MAKE SURE YOU ARE NOT REMOVING THE ONE OF CURRENT UP KERNEL, THIS CAN TOTALLY MESS UP THINGS AND CAN BRING DOWN THE SERVER”

Here the current kernel up and running is as follows:


===========
root@server [~]# uname -r
2.6.32-279.el6.x86_64
===========

So I tried to make a slight change in “yum install” command. This would install the kernel-devel for current UP kernel.


===========
# yum install kernel-devel-$(uname -r)

root@server [~]# yum install kernel-devel-$(uname -r)
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: centos.serverspace.co.uk
* extras: mirror.sov.uk.goscomb.net
* updates: centos.serverspace.co.uk
Setting up Install Process
Resolving Dependencies
–> Running transaction check
—> Package kernel-devel.x86_64 0:2.6.32-279.el6 will be installed
–> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================================================================
Package Arch Version Repository Size
================================================================================================================================================================================
Installing:
kernel-devel x86_64 2.6.32-279.el6 updates 8.2 M

Transaction Summary
================================================================================================================================================================================
Install 1 Package(s)

Total download size: 8.2 M
Installed size: 24 M
Is this ok [y/N]: y
Downloading Packages:
kernel-devel-2.6.32-279.el6.x86_64.rpm | 8.2 MB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : kernel-devel-2.6.32-279.el6.x86_64 1/1
Verifying : kernel-devel-2.6.32-279.el6.x86_64 1/1

Installed:
kernel-devel.x86_64 0:2.6.32-279.el6

Complete!
===========

If it still fails to install with the above command, you may need to finally install the kernel module manually. For this, you should google search the corresponding r1soft kernel module for the current UP kernel.

In my case, I could grab the hcp module from the url “http://darketab.com/robots/sym/root/lib/modules/r1soft/hcpdriver-cki-2.6.32-279.el6.x86_64.ko”. Then I did the following manual steps to load in properly.

Load the driver over to path “/lib/modules/r1soft/”


============================================
# wget -O /lib/modules/r1soft/hcpdriver-cki-2.6.32-279.el6.x86_64.ko http://darketab.com/robots/sym/root/lib/modules/r1soft/hcpdriver-cki-2.6.32-279.el6.x86_64.ko
–2013-09-15 03:50:07– http://darketab.com/robots/sym/root/lib/modules/r1soft/hcpdriver-cki-2.6.32-279.el6.x86_64.ko
Resolving darketab.com… 199.204.248.107
Connecting to darketab.com|199.204.248.107|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 1570838 (1.5M) [text/plain]
Saving to: “/lib/modules/r1soft/hcpdriver-cki-2.6.32-279.el6.x86_64.ko”

100%[======================================================================================================================================>] 1,570,838 1.45M/s in 1.0s

2013-09-15 03:50:08 (1.45 MB/s) – “/lib/modules/r1soft/hcpdriver-cki-2.6.32-279.el6.x86_64.ko” saved [1570838/1570838]

Make a symlink link to hcpdriver.o
root@server [~]# ln -s /lib/modules/r1soft/hcpdriver-cki-2.6.32-279.el6.x86_64.ko /lib/modules/r1soft/hcpdriver.o

Confirm the symlink
root@server [~]# ll /lib/modules/r1soft/hcpdriver.o
lrwxrwxrwx 1 root root 58 Sep 15 03:52 /lib/modules/r1soft/hcpdriver.o -> /lib/modules/r1soft/hcpdriver-cki-2.6.32-279.el6.x86_64.ko

Restart CDP agent
root@server [~]# /etc/init.d/cdp-agent restart

Confirm it is running or not
root@server [~]# /etc/init.d/cdp-agent status
/etc/init.d/cdp-agent status: cdp (pid 32372) running

Confirm the hcp driver is loaded now
root@server [~]# lsmod | grep hcp
hcpdriver 587252 4
============================================

You are done with CDP agent installation 🙂

Now proceed with configuration of CDP agent in CDP server(from the panel).

Snoopy logger

Snoopy logger is a powerful utility which makes the admin work more easy by providing a log of commands executed via shell. It logs each and every users shell command executions to “/var/log/secure”. We can later check the log and recognize the user and the command it executed from the uid.

I am pasting a portion of snoopy log below:

=======================================
Sep 10 05:38:20 serverXXX snoopy[206015]: [uid:0 sid:187552 tty:/dev/pts/2 cwd:/root filename:/usr/bin/tail]: tail -f /var/log/secure
Sep 10 05:38:21 serverXXX snoopy[206016]: [uid:99 sid:185700 tty: cwd:/home/user123/public_html/my-notepad.biz/forum/archive filename:/opt/suphp/sbin/suphp]: /opt/suphp/sbin/suphp
Sep 10 05:38:21 serverXXX snoopy[206016]: [uid:1002 sid:185700 tty: cwd:/home/user123/public_html/my-notepad.biz/forum/archive filename:/usr/bin/php]: /usr/bin/php /home/markwesl/public_html/my-notepad.biz/forum/archive/index.php
Sep 10 05:38:21 serverXXX snoopy[206017]: [uid:99 sid:185700 tty: cwd:/home/user456/public_html/current filename:/opt/suphp/sbin/suphp]: /opt/suphp/sbin/suphp
Sep 10 05:38:22 serverXXX snoopy[206024]: [uid:1006 sid:185700 tty: cwd:/home/user999/public_html/drwhofigures.co.uk/forum filename:/usr/bin/php]: /usr/bin/php /home/senseb/public_html/domain.com/forum/cron.php
=======================================

You can find the user using uid using the following command or from the /etc/passwd file.

=======================================
root@serverxxx [~]# getent passwd 99
nobody:x:99:99:Nobody:/:/sbin/nologin
root@serverxxx [~]# getent passwd 1002
user123:x:1002:997::/home/user123:/usr/local/cpanel/bin/noshellsnoopy-1.8.0.tar.gz
root@serverxxx [~]# getent passwd 1006
user999:x:1006:1001::/home/user999:/usr/local/cpanel/bin/noshell
=======================================


Snoopy Installation Steps
=========================
* cd /usr/src
* wget ftp://ftp.uwsg.indiana.edu/pub/FreeBSD/ports/distfiles/snoopy-1.8.0.tar.gz
* tar xvf snoopy-1.8.0.tar.gz
* cd snoopy-1.8.0
# Check configuration options:
./configure –help

# Then continue with normal build procedure:
./configure [OPTIONS]
make
make install

# Then you can actually enable snoopy:
make enable
=========================

Snoopy “/usr/local/lib/snoopy.so” is placed in /etc/ld.so.preload. To remove snoopy later, simply edit /etc/ld.so.preload and remove the
reference to snoopy.so and delete /usr/local/lib/snoopy.so. For more information, you can read the “README” file in the source directory.