My Server may be Compromised-Hacked-Rooted
From SuperbHosting Support Wiki
Contents
|
Determine the Level of Compromise
Quite often files are modified and malicious scripts are running but the server is not completely compromised beyond repair. If malicious files or processes are owned by root however you can be certain rooted and should not be trusted. If this is the case you must not make *any* connections to any services requiring authentication from the compromised server. Doing so would put other systems at risk. If at any time during your investigation a command exits with a segmentation fault you must be very suspicious and assume that the server has been rooted from that point on.
ALL of the following steps require that you are logged in as root
Examine log files
- Use commands like "last" to review who logged in recently and if the /var/log/wtmp file has been truncated in any way. Keep in mind that some successful logins may be Superb Internet Corporation staff.
- Although it can be time consuming you should check all the log files in /var/log/ for strange errors or entries from unknown hosts. Do not failed mistake attempts as the server being compromised since this is a normal part of being connected to the internet.
- Keep in mind that intruders quite often edit or 'clean' log files to hide their activity. Look for signs that the logs have been modified or truncated.
Look for SUID or SGID files
- To check for files that are SUID root:
# find / -user root -perm -4000 -print
- To avoid going into other file systems mounted somewhere beneath / you can include the -xdev option. This will save some time and avoid printing files in /dev/ or /proc/ that may have strange permissions but that are normal. You may want to check /dev/shm/ manually however after this.
# find / -xdev -user root -perm -4000 -print
# find /dev/shm -xdev -user root -perm -4000 -print
Check system binaries for modifications
An effective way to do this is to use 'rkhunter' and/or 'chkrootkit'.
Rkhunter
- This works for both Linux and FreeBSD. You may install the port for FreeBSD from the ports collection. For linux it must be installed manually.
- Check to see if it is already installed on your server and update it if it is.
rkhunter --update
- Download from [SourceForge]
- Unpack the tar.gz
# tar zxvf rkhunter-1.x.x.tar.gz
- Move into the installation directory
# cd rkhunter-x.x.x
- Install it
# ./installer.sh --layout /usr --install
- Update it
# rkhunter --update
- Execute it
# rkhunter --checkall -sk
Chkrootkit
- Download from [Chkrootkit.org]
- Unpack the tar.gz
# tar zxvf chkrootkit.tar.gz
- Move into the installation directory
# cd chkrootkit-x.xx
- Execute it
# ./chkrootkit
Check for processes and tcp/udp connections
It is highly likely that if the server has been rooted the binary that shows you the process list was modified. It is unlikely, but possible, that a miscreant could have modified the dates/times of the 'ps' binary. Use your judgment with this check. You can check this with# stat `which ps`If the modification date/time on the file seems OK you can trust it to give you a reliable list of processes running on the server.
# ps auxwww
If the 'lsof' binary is present and not modified you can use it to see the entire tcp/udp table. It will show you the tcp/udp ports being used by your server and show you what resources on the server are related to those services. Simply execute it as follows:
# lsof -i
Carefully check the /etc/passwd and /etc/shadow (or equivalent) files
View the contents of the /etc/passwd, /etc/shadow or /etc/master.passwd files and ensure you know who each user is. Look for accounts with a uid of "0" other than root and toor. Since the actual encrypted passwords are not stored in /etc/passwd any more you can use a command like this to find them quickly# grep "*:0" /etc/passwd or # grep "x:0" /etc/passwd
Look for unusual or hidden files and directories
To find all the files on the system modified within the last 30 twenty-four hour periods you can execute
# find / -xdev -ctime -30To find more recently modified files you may reduce the 30 to just a few days.
In addition to this check you should also check for common malicious directory names. Some examples are
# find / -name ".. " -type d -print -xdev
# find / -name ".*" -print -xdev
# find / -name ".,*" -print -xdev
Recover from the Intrusion
Root Account Compromised
YOU MUST ONLY DO THIS IF THE ROOT ACCOUNT IS COMPROMISED. If you are unsure, please ask staff to confirm that root has been compromised through myCP.
- Install a clean version of your operating system. This must be requested in a new ticket through myCP. While creating the ticket through myCP you must choose the correct category and subject for what it is you want done. Keep in mind that if a machine is compromised, anything on that system could have been modified, including the kernel, binaries, data files, running processes, and memory. In general, the only way to trust that a machine is free from back doors and intruder modifications is to reinstall the operating system from the distribution media and install all of the security patches before connecting back to the network. Merely determining and fixing the vulnerability that was used to initially compromise this machine may not be enough. We encourage you to restore your system using known clean binaries. In order to put the machine into a known state, you should re-install the operating system using the original distribution media. You may choose to either
- reinstall the operating system and delete all the data on the disk so you can restore from backups you have elsewhere or
- make the current system disk into a secondary disk and have a new disk installed with a fresh operating system so you can restore data from the disk containing the compromised system files.
- Caution use of data from backups When restoring data from a backup you must ultimately ensure that the backup itself is from an un-compromised machine if possible. You must also find the vulnerability allowing the compromise. Quite often the server is compromised because the operating system has not been updated. Keep in mind that you could re-introduce a vulnerability that would allow an intruder to gain unauthorized access. Also, if you are only restoring users' home directories and data files, keep in mind that any of those files could contain Trojan horse programs. You may want to pay close attention to .rhosts files in users' home directories.
- Change passwords After all security holes or configuration problems have been patched or corrected, we suggest that you change the passwords of ALL accounts on the affected system(s). Ensure that passwords for all accounts are not easy to guess. A good password generator to help you choose secure passwords can be found at:
http://www.pctools.com/guides/password/
In any and all cases the expected downtime for a re-installation of the operating system can be anywhere from 1 to 6 hours from the point the re-installation is started. This does not include the time it will take for you to recover configuration or content from your backups or old disk.
Please note that if you previously had a Fedora distribution you should move to the latest CentOS distribution because the life-span of CentOS is usually several years for each version versus several months for Fedora. The life-span of an operating system is the amount of time security and feature updates are available after its release.
If you have [cPanel] please see Restoring cPanel from disk
User Account Compromised
- Find all the malicious files related to the processes found and delete them.
- Kill the malicious processes
- Modify the password for the user that was compromised (or disable the account entirely)
- Investigate and correct the vulnerability allowing the compromise
- Perform system and application updates wherever applicable
Having the Recovery Done on Your Behalf
All work related to a compromised server that is performed on your behalf is billable via support credits. Complex or time consuming operations can be performed as custom work at a rate of $90.00/hour.
