April 4

Unknown linux rootkit?

Recently noticed a bunch of servers (CentOS 5.2 and CentOS 4.8) with SSH fingerprint mismatches. After poking around, it appears that they had been compromised. chkrootkit and RKhunter found nothing but the suspicious entry in /dev (see below).

WARNING: DSA key found for host ourserver.domain.com
in /Users/username/.ssh/known_hosts:578
DSA key fingerprint f4:27:d0:32:6b:4c:9f:6e:52:6f:49:dd:19:54:c2:f1.
+–[ DSA 1024]—-+
| ..o+|
| . o..|
| * . .E|
| + B . . .o|
| S * o …|
| . o = . |
| . o + |
| o . |
| |
+—————–+

The authenticity of host ‘ourserver.domain.com (10.0.0.2)’ can’t be established
but keys of different type are already known for this host.
RSA key fingerprint is 4c:5f:95:3e:52:08:6c:cd:0f:f8:37:38:3c:dd:bf:56.
Are you sure you want to continue connecting (yes/no)? yes

Affected files:

/dev/sax
/bin/suidshell
/usr/local/bin/ssh-agent
/usr/local/bin/sftp-server
/usr/local/bin/ssh-agent2
/usr/local/bin/ssh
/usr/local/bin/ssh-add2
/usr/local/bin/ssh-signer
/usr/local/bin/ssh-keygen2
/usr/local/bin/ssh-probe2
/usr/local/bin/ssh-probe
/usr/local/bin/scp
/usr/local/bin/sftp
/usr/local/bin/ssh-chrootmgr
/usr/local/bin/ssh-signer2
/usr/local/bin/sftp2
/usr/local/bin/scp2
/usr/local/bin/ssh-pubkeymgr
/usr/local/bin/ssh-dummy-shell
/usr/local/bin/ssh-add
/usr/local/bin/sftp-server2
/usr/local/bin/ssh-askpass
/usr/local/bin/ssh2
/usr/local/bin/ssh-keygen
/usr/local/sbin/sshd2
/usr/local/sbin/sshd-check-conf
/usr/local/sbin/sshd
    /usr/local/sbin/sshd2 — captures login credentials for both root and non-root logins over SSH/SCP/sftp etc and logs to /dev/sax

    /bin/suidshell — is exactly that, a shell with the suid bit set that instantly gives root access to any unprivileged user!

    /dev/sax — stores plaintext username and password for root and non-root accounts

Category: Hosting, Linux, Security | LEAVE A COMMENT
March 24

pure-ftpd Can’t change directory to /var/ftp/

The problem arises when a user attempts to make an anonymous FTP connection to Cpanel user’s account who has already enabled anonymous FTP connections in their control panel. However, pure-ftpd drops you with the error “421 Can’t change directory to /var/ftp/”.

workstation:~ user$ ftp testing.com
Connected to testing.com.
220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
220-You are user number 3 of 50 allowed.
220-Local time is now 11:02. Server port: 21.
220-IPv6 connections are also welcome on this server.
220 You will be disconnected after 15 minutes of inactivity.
Name (testing.com:user): anonymous
421 Can't change directory to /var/ftp/ [/]
ftp: Login failed.

The solution(s):
1. Use anonymous@domain.com and any password instead of just anonymous
…OR
2. Assign the Cpanel user a dedicated IP address where FTP logins with just “anonymous” will work.

Category: Cpanel, Hosting, Linux | 2 Comments
February 25

BackupPC – rebuilding your /etc/BackupPC/hosts file

Web interface allows administrators to view log files, configuration, current status and allows users to initiate and cancel backups and browse and restore files from backups.
Data deduplication: Identical files across multiple backups of the same or different PCs are stored only once resulting in substantial savings in disk storage and disk I/O.
Compression: Since only new files (not already pooled) need to be compressed, there is only a modest impact on CPU time.
Open-source: BackupPC is hosted on Github, and is distributed under a GPL license.
No client-side software is needed. The web user interface describes the way people interact with a site and the way users can access its functions such as many apps that work to improve other platforms with Social Boosting for social media.

BackupPC is terrible at removing old hosts where backups are no longer needed. Over time, it becomes necessary to get old servers out of your BackupPC “Host” list drop-down. You might find yourself in a situation where it becomes necessary or even easier to remove the config .pl files from your pc/ directory, and then just recreate your hosts file. Use the following steps to accomplish just that:

[root@backupserver pc]# for i in `ls /etc/BackupPC/pc | sed ‘s/.pl//g’`; do grep $i /etc/BackupPC/hosts; done > /etc/BackupPC/hosts-NEW
[root@backupserver pc]# cd /etc/BackupPC
[root@backupserver BackupPC]# mv hosts hosts.BAK
[root@backupserver BackupPC]# mv hosts-NEW hosts
[root@backupserver BackupPC]# chown apache:apache hosts
[root@backupserver BackupPC]# /etc/rc.d/init.d/backuppc restart
Shutting down BackupPC: [ OK ]
Starting BackupPC: [ OK ]
[root@backupserver BackupPC]#

Category: Linux | LEAVE A COMMENT
February 23

Disabling Drupal syslog messages on shared servers

We have had issues with Drupal sending log information to syslog, which conicidently gets copied to the console for ALL users to see. Needless to say, this is not acceptable in a shared webhosting environment.

To disable the syslog module using the MySQL command line, run the following SELECT to look at the state of your data before the change. This will also help you to find the full name and enabled/disabled status of the module too:

SELECT name,status FROM system WHERE type='module';

Then to disable your syslog module, set the status to 0 for the module name that you want to disable:

UPDATE system SET status='0' WHERE name='syslog';

Check your handiwork using the SELECT statement again. Hope this helps someone out there.

Category: Uncategorized | 1 Comment
February 22

apache_conf_distiller User data set has no ‘main_domain’ key.

After a server was hacked recently, the attackers replaced all files named index / default / main with their typical 0wned-by message and javascript. Of the “main” files that were affected, the /var/cpanel/userdata/USER/main files were also overwritten. These files contain important cpanel domain information which are required to build a new httpd.conf using the apache_conf_distiller. The following steps should regenerate a working apache userdata in order to fix subdomains. Thanks to Josh for finding userdata_update!

/etc/init.d/httpd stop
mv /usr/local/apache/conf/httpd.conf /usr/local/apache/conf/httpd.conf-notworking
cp -a OLDHTTPD.CONF /usr/local/apache/conf/
mv /var/cpanel/userdata /var/cpanel/userdata-BAK
/usr/local/cpanel/bin/userdata_update
cp -a /var/cpanel/userdata /usr/local/apache/conf
/etc/init.d/httpd start

Run the /usr/local/cpanel/bin/apache_conf_distiller –update to ensure the main_domain key errors are gone.

Category: Cpanel, Hosting, Linux | LEAVE A COMMENT
February 8

Finding problem CRON jobs

Find cronjobs that were modified recently:

[root@SERVER cron]# find /var/spool/cron -type f -mtime -3 | xargs ls -al

After commenting out suspect lines in the listed user’s crontabs, you can dump the process list to a file every 5 seconds or so with:

[root@SERVER ~]# touch /root/ps-list.txt
[root@SERVER ~]# watch -n 5 "ps aux >> /root/ps-list.txt"

If the server crashes, you can then review the last few lines of /root/ps-list.txt to see which processes appear to be overwhelming the server.

Category: Linux | LEAVE A COMMENT
February 1

Rebuilding raid with mdadm

Look at mdstat to see if a partition has been dropped from the array:

root@SERVER [~]# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdb1[1] sda1[0]
      521984 blocks [2/2] [UU]
      
md1 : active raid1 sda3[0]
      483668864 blocks [2/1] [U_]

The [U_] shows that sdb3 is out of the array md1. To add /dev/sdb3 back into the array, we do the following:

root@reseller10 [~]# mdadm /dev/md1 -a /dev/sdb3 
mdadm: re-added /dev/sdb3
root@SERVER [~]# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdb1[1] sda1[0]
      521984 blocks [2/2] [UU]

      
md1 : active raid1 sdb3[1] sda3[0]
      483668864 blocks [2/1] [U_]
      [>....................]  recovery =  0.0% (2432/483668864) finish=6448.8min speed=1216K/sec

Running: echo 100000 > /proc/sys/dev/raid/speed_limit_min will speed up the software raid rebuild process:

root@SERVER [~]# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdb1[1] sda1[0]
      521984 blocks [2/2] [UU]
      
md1 : active raid1 sdb3[2] sda3[0]
      483668864 blocks [2/1] [U_]
      [==>..................]  recovery = 11.0% (53583104/483668864) finish=1030.4min speed=6954K/sec
      
unused devices:
Category: Linux | LEAVE A COMMENT