Linux intrusion troubleshooting for emergency response


In case of hacker intrusion, system crash or other security events affecting the normal operation of business, it is urgent to deal with them at the first time, so that the enterprise's network information system can resume normal operation in the shortest time, further find the source of intrusion, restore the process of intrusion accident, and give solutions and preventive measures to recover or reduce economic losses for the enterprise.

Aiming at common attack events, combined with the analysis and solutions of emergency response events at work, this paper summarizes some ideas of Linux server intrusion troubleshooting.

1. Intrusion troubleshooting ideas

1.1 account security

Basic usage:

1,User information file/etc/passwd
 User name: Password: user ID: group ID: User description: Home Directory: after logging in shell
 Note: only local login is allowed without password, and remote login is not allowed

2,Shadow file/etc/shadow
User name: encryption password: date of last password modification: time interval between two password modifications: password validity: warning days after password modification expires: Grace days after password Expiration: account expiration time: reserved
who     View current login user( tty Local login  pts (remote login)
w       Check the system information and want to know the user's behavior at a certain time
uptime  Check the login time, users and load

Intrusion detection:

1,Query privileged user(uid Is 0)
[root@localhost ~]# awk -F: '$3==0{print $1}' /etc/passwd
2,Query the account information of remote login
[root@localhost ~]# awk '/\$1|\$6/{print $1}' /etc/shadow
3,except root Is there any other account besides the account sudo jurisdiction. If it is not necessary for management, the ordinary account should be deleted sudo jurisdiction
[root@localhost ~]# more /etc/sudoers | grep -v "^#\|^$" | grep "ALL=(ALL)"
4,Disable or delete redundant and suspicious accounts
    usermod -L user    Disable account, account cannot log in,/etc/shadow The second column is!start
	userdel user       delete user user
	userdel -r user    Will delete user User, and will/home Directory user Delete the directory together

1.2 historical orders

Basic usage:

adopt.bash_history View the system commands executed by the account
1,root Historical command of
2,open/home Under each account directory.bash_history,View the history command of ordinary account

Add login for historical commands IP Address, execution time and other information:
1)Save 10000 commands
sed -i 's/^HISTSIZE=1000/HISTSIZE=10000/g' /etc/profile
2)stay/etc/profile Add the following line number configuration information at the end of the file:
######jiagu history xianshi#########
USER_IP=`who -u am i 2>/dev/null | awk '{print $NF}' | sed -e 's/[()]//g'`
if [ "$USER_IP" = "" ]
export HISTTIMEFORMAT="%F %T $USER_IP `whoami` "
shopt -s histappend
export PROMPT_COMMAND="history -a"
######### jiagu history xianshi ##########
3)source /etc/profile Make configuration effective

Generation effect: 1 2018-07-10 19:45:39 root source /etc/profile

3,Clear history operation command: history -c
 However, this command does not clear the records saved in the file, so it needs to be deleted manually.bash_profile Records in the file.

Intrusion detection:

Enter the user directory
cat .bash_history >> history.txt

1.3 check the abnormal port

Use the netstat network connection command to analyze the suspicious port, IP and PID

netstat -antlp|more

Check next pid Corresponding process file path,
function ls -l /proc/$PID/exe or file /proc/$PID/exe($PID Is the corresponding pid No.)

1.4 check the abnormal process

Use the ps command to analyze the process

ps aux | grep pid 

1.5 check startup items

Basic usage:

Schematic diagram of system operation level:

Run level meaning
0 Shut down
1 Single user mode, which can be imagined as the security mode of windows, is mainly used for system repair
2 Incomplete command line mode without NFS service
3 The complete command line mode is the standard character interface
4 System retention
5 Graphic mode
6 Restart

View run level commands

System default allowable level

vi  /etc/inittab
id=3: initdefault  Which operation level does the system enter directly after startup

Boot profile


Example: when we need to start our own script, we just need to leave the executable script in / etc / init D directory, and then in / etc / RC d/rc*. D to establish a soft link

root@localhost ~]# ln -s /etc/init.d/sshd /etc/rc.d/rc3.d/S100ssh

Here, sshd is the script file of the specific service, S100ssh is its soft link, and the beginning of S represents self startup during loading; If it is a script file starting with K, it represents the script file that needs to be closed when the run level is loaded.

Intrusion detection:

Startup item file:
more /etc/rc.local
ls -l /etc/rc.d/rc3.d/

1.6 check scheduled tasks

Basic use

1. Creating scheduled tasks with crontab

  • Basic command

crontab -l lists the details of a user's cron service

Tips: the crontab file written by default will be saved in (/ var/spool/cron / user name), for example: / var/spool/cron/root

crontab -r delete cront tasks for each user (caution: delete all scheduled tasks)

crontab -e use the editor to edit the current crontab file

For example: * / 1 * * echo "hello world" > > / TMP / test Txt write files every minute

2. Asynchronous scheduled task scheduling using anacron

  • Use case

Run / home / backup.exe every day SH script:
vi /etc/anacrontab
@daily 10 example.daily /bin/bash /home/

When the machine is in backup SH is expected to be turned off when it is running. anacron will run it ten minutes after the machine is turned on, instead of waiting for another seven days.

Intrusion detection

Focus on whether there are malicious scripts in the following directories



 more /etc/cron.daily/*  View all files in the directory

1.7 Inspection Service

Service self start

The first modification method:

chkconfig [--level Run level] [Independent service name] [on|off]
chkconfig –level  2345 httpd on  Turn on self start
chkconfig httpd on (default level Yes (2345)

The second modification method:

modify/etc/re.d/rc.local file  
join /etc/init.d/httpd start

The third modification method:

Using ntsysv command to manage self startup, you can manage independent services and xinetd services.

Intrusion detection

1. Query installed services:

RPM package installed services

chkconfig  --list  View the service self start status, and you can see all the RPM Package installed services
ps aux | grep crond View current service

System startup items at levels 3 and 5 
Chinese environment
chkconfig --list | grep "3:Enable\|5:Enable"
English environment
chkconfig --list | grep "3:on\|5:on"

Services installed by source package

Check the service installation location, usually in/user/local/
service httpd start
 search/etc/rc.d/init.d/  Check to see if it exists

1.8 check abnormal documents

1. View sensitive directories, such as files in / tmp directory, and pay attention to hidden folders. Folders named "..." have hidden properties

2. Get the creation time of WEBSHELL and remote control Trojan horse. How to find the files created within the same time range?

You can use the find command to find, such as find /opt -iname "*" - atime 1 -type f to find the file accessed by / opt the next day

3. For suspicious files, you can use stat to create and modify the time.

1.9 check the system log

Default log storage location: / var/log/

Check the log configuration: more / etc / rsyslog conf

log file explain
/var/log/cron Logs related to system scheduled tasks are recorded
/var/log/cups Log of printing information
/var/log/dmesg It records the information of the kernel self-test when the system is powered on. You can also use the dmesg command to directly view the kernel self-test information
/var/log/mailog Record mail information
/var/log/message A log recording important information of the system. This log file will record most important information of the Linux system. If there is a problem with the system, the first thing to check should be this log file
/var/log/btmp Log the error login log. This file is a binary file and cannot be viewed directly by vi. instead, use the lastb command to view it
/var/log/lastlog Record the log of the last login time of all users in the system. This file is a binary file and cannot be viewed directly by vi. instead, use the lastlog command to view it
/var/log/wtmp Permanently record the login and logout information of all users, and record the startup, restart and shutdown events of the system. Similarly, this file is also a binary file, which cannot be viewed directly by vi, but needs to be viewed by using the last command
/var/log/utmp Record the information of the currently logged in user. This file will change with the login and logout of the user, and only record the information of the currently logged in user. Similarly, this file can't be queried directly by vi, but by using w,who,users and other commands
/var/log/secure Record the authentication and authorization information. Any program involving account and password will be recorded, such as SSH login, su switching users, sudo authorization, and even adding users and modifying user passwords will be recorded in this log file

Log analysis skills:

1,How many positioning IP In the blasting host root Account number:    
grep "Failed password for root" /var/log/secure | awk '{print $11}' | sort | uniq -c | sort -nr | more

What are the positioning IP During blasting:
grep "Failed password" /var/log/secure|grep -E -o "(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)"|uniq -c

What is a user name dictionary?
 grep "Failed password" /var/log/secure|perl -e 'while($_=<>){ /for(.*?) from/; print "$1\n";}'|uniq -c|sort -nr
2,Successful login IP What are: 	
grep "Accepted " /var/log/secure | awk '{print $11}' | sort | uniq -c | sort -nr | more

Date of successful login, user name IP: 
grep "Accepted " /var/log/secure | awk '{print $1,$2,$3,$9,$11}' 

3,Add a user kali journal:
Jul 10 00:12:15 localhost useradd[2382]: new group: name=kali, GID=1001
Jul 10 00:12:15 localhost useradd[2382]: new user: name=kali, UID=1001, GID=1001, home=/home/kali
, shell=/bin/bash
Jul 10 00:12:58 localhost passwd: pam_unix(passwd:chauthtok): password changed for kali
#grep "useradd" /var/log/secure 

4,delete user kali journal:
Jul 10 00:14:17 localhost userdel[2393]: delete user 'kali'
Jul 10 00:14:17 localhost userdel[2393]: removed group 'kali' owned by 'kali'
Jul 10 00:14:17 localhost userdel[2393]: removed shadow group 'kali' owned by 'kali'
# grep "userdel" /var/log/secure

5,su Switch users:
Jul 10 00:38:13 localhost su: pam_unix(su-l:session): session opened for user good by root(uid=0)

sudo Authorized execution:
sudo -l
Jul 10 00:43:09 localhost sudo:    good : TTY=pts/4 ; PWD=/home/good ; USER=root ; COMMAND=/sbin/shutdown -r now

2. Tools

2.1 Rootkit killing

  • chkrootkit


    usage method:
    tar zxvf chkrootkit.tar.gz
    cd chkrootkit-0.52
    make sense
    #Check if there is no error after compilation
  • rkhunter


    usage method:
    tar -zxvf rkhunter-1.4.4.tar.gz
    cd rkhunter-1.4.4
    ./ --install
    rkhunter -c

2.2 virus killing

  • Clamav

    The official download address of ClamAV is:

    Installation method I:

    1,install zlib: 
    tar -zxvf  zlib-1.2.7.tar.gz
    cd zlib-1.2.7
    #Install gcc compilation environment: yum install gcc
    CFLAGS="-O3 -fPIC" ./configure --prefix= /usr/local/zlib/
    make && make install
    2,Add user group clamav And group members clamav: 
    groupadd clamav
    useradd -g clamav -s /bin/false -c "Clam AntiVirus" clamav
    3,install Clamav
    tar –zxvf clamav-0.97.6.tar.gz
    cd clamav-0.97.6
    ./configure --prefix=/opt/clamav --disable-clamav -with-zlib=/usr/local/zlib
    make install
    4,to configure Clamav
    mkdir /opt/clamav/logs
    mkdir /opt/clamav/updata
    touch /opt/clamav/logs/freshclam.log
    touch /opt/clamav/logs/clamd.log
    cd /opt/clamav/logs
    chown clamav:clamav clamd.log
    chown clamav:clamav freshclam.log
    5,ClamAV use:
     /opt/clamav/bin/freshclam Upgrade virus library
    ./clamscan –h View the corresponding help information
    ./clamscan -r /home  Scan the home directory of all users and use it
    ./clamscan -r --bell -i /bin  scanning bin Directory and display the scanning results of the problematic files

    Installation mode 2:

    yum install -y clamav
    #Update virus library
    #Scanning method
    clamscan -r /etc --max-dir-recursion=5 -l /root/etcclamav.log
    clamscan -r /bin --max-dir-recursion=5 -l /root/binclamav.log
    clamscan -r /usr --max-dir-recursion=5 -l /root/usrclamav.log
    #Scan and kill virus
    clamscan -r  --remove  /usr/bin/bsd-port
    clamscan -r  --remove  /usr/bin/
    clamscan -r --remove  /usr/local/zabbix/sbin
    #View log discovery
    cat /root/usrclamav.log |grep FOUND

2.3 webshell killing

linux version:

Hippo webshell Killing:
 Deeply convinced Webshell Website backdoor detection tool:

2.4 RPM check

The system integrity can be verified through the - Va provided by rpm. Check all RPM software packages to see which commands have been replaced:

./rpm -Va > rpm.log

If everything is verified to be normal, no output will be generated. If there is any inconsistency, it will be displayed. The output format is an 8-bit long string. Each character is used to represent the comparison result between the file and an attribute in the RPM database. If yes (DOT) indicates that the test has passed.

The specific contents of the 8 information in the verification content are as follows:
		S         Does the file size change
		M         File type or file permissions( rwx)Is it changed
		5         file MD5 Check whether it is changed (it can be regarded as whether the file content is changed)
		D         Is the slave code changed in the device
		L         Whether the file path has changed
		U         Is the owner (owner) of the file changed
		G         Is the group of the file changed
		T         Is the modification time of the file changed

If the command is replaced and restored:

File extraction restore case:
rpm  -qf /bin/ls  query ls Which package does the command belong to
mv  /bin/ls /tmp  First ls Transfer to tmp Directory, causing ls The illusion of command loss
rpm2cpio /mnt/cdrom/Packages/coreutils-8.4-19.el6.i686.rpm | cpio -idv ./bin/ls extract rpm In the bag ls Command to the current directory/bin/ls lower
cp /root/bin/ls  /bin/ hold ls Copy command to/bin/Directory repair file missing

2.5 linux security check script

Github project address:

Quote a big man:
It's better to believe in books than to have no books. Tools are only auxiliary. Don't rely too much on them. The key lies in your ideas on how to solve problems.

Disclaimer: I firmly oppose the use of teaching methods to commit crimes. All criminal acts will be severely punished. The green network needs our joint maintenance. I recommend you to understand the principles behind them and better protect them. It is prohibited for anyone to reprint to other sites and use for any illegal purpose. If anyone does anything illegal with this, it has nothing to do with the author. It is hereby declared.

Tags: Cyber Security

Posted by feddie1984 on Fri, 06 May 2022 01:45:36 +0300