preface

In case of hacker intrusion, system crash or other security events affecting the normal operation of business, it is urgent to deal with them at the first time, so that the enterprise's network information system can resume normal operation in the shortest time, further find the source of intrusion, restore the process of intrusion accident, and give solutions and preventive measures to recover or reduce economic losses for the enterprise.

Aiming at common attack events, combined with the analysis and solutions of emergency response events at work, this paper summarizes some ideas of Linux server intrusion troubleshooting.

1. Intrusion troubleshooting ideas

1.1 account security

Basic usage:

1,User information file/etc/passwd
root:x:0:0:root:/root:/bin/bash
account:password:UID:GID:GECOS:directory:shell
 User name: Password: user ID: group ID: User description: Home Directory: after logging in shell
 Note: only local login is allowed without password, and remote login is not allowed

2,Shadow file/etc/shadow
root:$6$oGs1PqhL2p3ZetrE$X7o7bzoouHQVSEmSgsYN5UD4.kMHx6qgbTqwNVC5oOAouXvcjQSt.Ft7ql1WpkopY0UV9ajBwUt1DpYxTCVvI/:16809:0:99999:7:::
User name: encryption password: date of last password modification: time interval between two password modifications: password validity: warning days after password modification expires: Grace days after password Expiration: account expiration time: reserved

who     View current login user( tty Local login  pts (remote login)
w       Check the system information and want to know the user's behavior at a certain time
uptime  Check the login time, users and load

Intrusion detection:

1,Query privileged user(uid Is 0)
[root@localhost ~]# awk -F: '$3==0{print $1}' /etc/passwd
2,Query the account information of remote login
[root@localhost ~]# awk '/\$1|\$6/{print $1}' /etc/shadow
3,except root Is there any other account besides the account sudo jurisdiction. If it is not necessary for management, the ordinary account should be deleted sudo jurisdiction
[root@localhost ~]# more /etc/sudoers | grep -v "^#\|^$" | grep "ALL=(ALL)"
4,Disable or delete redundant and suspicious accounts
    usermod -L user    Disable account, account cannot log in,/etc/shadow The second column is!start
	userdel user       delete user user
	userdel -r user    Will delete user User, and will/home Directory user Delete the directory together

1.2 historical orders

Basic usage:

adopt.bash_history View the system commands executed by the account
1,root Historical command of
histroy
2,open/home Under each account directory.bash_history，View the history command of ordinary account

Add login for historical commands IP Address, execution time and other information:
1)Save 10000 commands
sed -i 's/^HISTSIZE=1000/HISTSIZE=10000/g' /etc/profile
2)stay/etc/profile Add the following line number configuration information at the end of the file:
######jiagu history xianshi#########
USER_IP=`who -u am i 2>/dev/null | awk '{print $NF}' | sed -e 's/[()]//g'`
if [ "$USER_IP" = "" ]
then
USER_IP=`hostname`
fi
export HISTTIMEFORMAT="%F %T $USER_IP `whoami` "
shopt -s histappend
export PROMPT_COMMAND="history -a"
######### jiagu history xianshi ##########
3)source /etc/profile Make configuration effective

Generation effect: 1 2018-07-10 19:45:39 192.168.204.1 root source /etc/profile

3,Clear history operation command: history -c
 However, this command does not clear the records saved in the file, so it needs to be deleted manually.bash_profile Records in the file.

Intrusion detection:

Enter the user directory
cat .bash_history >> history.txt

1.3 check the abnormal port

Use the netstat network connection command to analyze the suspicious port, IP and PID

netstat -antlp|more

Check next pid Corresponding process file path,
function ls -l /proc/$PID/exe or file /proc/$PID/exe($PID Is the corresponding pid No.)

1.4 check the abnormal process

Use the ps command to analyze the process

ps aux | grep pid 

1.5 check startup items

Basic usage:

Schematic diagram of system operation level:

View run level commands
runlevel

System default allowable level

vi  /etc/inittab
id=3: initdefault  Which operation level does the system enter directly after startup

Boot profile

/etc/rc.local
/etc/rc.d/rc[0~6].d

Example: when we need to start our own script, we just need to leave the executable script in / etc / init D directory, and then in / etc / RC d/rc*. D to establish a soft link

root@localhost ~]# ln -s /etc/init.d/sshd /etc/rc.d/rc3.d/S100ssh

Here, sshd is the script file of the specific service, S100ssh is its soft link, and the beginning of S represents self startup during loading; If it is a script file starting with K, it represents the script file that needs to be closed when the run level is loaded.

Intrusion detection:

Startup item file:
more /etc/rc.local
/etc/rc.d/rc[0~6].d
ls -l /etc/rc.d/rc3.d/

1.6 check scheduled tasks

Basic use

1. Creating scheduled tasks with crontab

• Basic command

crontab -l lists the details of a user's cron service

Tips: the crontab file written by default will be saved in (/ var/spool/cron / user name), for example: / var/spool/cron/root

crontab -r delete cront tasks for each user (caution: delete all scheduled tasks)

crontab -e use the editor to edit the current crontab file

For example: * / 1 * * echo "hello world" > > / TMP / test Txt write files every minute

2. Asynchronous scheduled task scheduling using anacron

• Use case

Run / home / backup.exe every day SH script:
vi /etc/anacrontab
@daily 10 example.daily /bin/bash /home/backup.sh

When the machine is in backup SH is expected to be turned off when it is running. anacron will run it ten minutes after the machine is turned on, instead of waiting for another seven days.

Intrusion detection

Focus on whether there are malicious scripts in the following directories

/var/spool/cron/* 
/etc/crontab
/etc/cron.d/*
/etc/cron.daily/* 
/etc/cron.hourly/* 
/etc/cron.monthly/*
/etc/cron.weekly/
/etc/anacrontab
/var/spool/anacron/*

Tips:

 more /etc/cron.daily/*  View all files in the directory

1.7 Inspection Service

Service self start

The first modification method:

chkconfig [--level Run level] [Independent service name] [on|off]
chkconfig –level  2345 httpd on  Turn on self start
chkconfig httpd on (default level Yes (2345)

The second modification method:

modify/etc/re.d/rc.local file  
join /etc/init.d/httpd start

The third modification method:

Using ntsysv command to manage self startup, you can manage independent services and xinetd services.

Intrusion detection

1. Query installed services:

RPM package installed services

chkconfig  --list  View the service self start status, and you can see all the RPM Package installed services
ps aux | grep crond View current service

System startup items at levels 3 and 5 
Chinese environment
chkconfig --list | grep "3:Enable\|5:Enable"
English environment
chkconfig --list | grep "3:on\|5:on"

Services installed by source package

Check the service installation location, usually in/user/local/
service httpd start
 search/etc/rc.d/init.d/  Check to see if it exists

1.8 check abnormal documents

1. View sensitive directories, such as files in / tmp directory, and pay attention to hidden folders. Folders named "..." have hidden properties

2. Get the creation time of WEBSHELL and remote control Trojan horse. How to find the files created within the same time range?

You can use the find command to find, such as find /opt -iname "*" - atime 1 -type f to find the file accessed by / opt the next day

3. For suspicious files, you can use stat to create and modify the time.

1.9 check the system log

Default log storage location: / var/log/

Check the log configuration: more / etc / rsyslog conf

Log analysis skills:

1,How many positioning IP In the blasting host root Account number:    
grep "Failed password for root" /var/log/secure | awk '{print $11}' | sort | uniq -c | sort -nr | more

What are the positioning IP During blasting:
grep "Failed password" /var/log/secure|grep -E -o "(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)"|uniq -c

What is a user name dictionary?
 grep "Failed password" /var/log/secure|perl -e 'while($_=<>){ /for(.*?) from/; print "$1\n";}'|uniq -c|sort -nr
 
2,Successful login IP What are: 	
grep "Accepted " /var/log/secure | awk '{print $11}' | sort | uniq -c | sort -nr | more

Date of successful login, user name IP: 
grep "Accepted " /var/log/secure | awk '{print $1,$2,$3,$9,$11}' 

3,Add a user kali journal:
Jul 10 00:12:15 localhost useradd[2382]: new group: name=kali, GID=1001
Jul 10 00:12:15 localhost useradd[2382]: new user: name=kali, UID=1001, GID=1001, home=/home/kali
, shell=/bin/bash
Jul 10 00:12:58 localhost passwd: pam_unix(passwd:chauthtok): password changed for kali
#grep "useradd" /var/log/secure 

4,delete user kali journal:
Jul 10 00:14:17 localhost userdel[2393]: delete user 'kali'
Jul 10 00:14:17 localhost userdel[2393]: removed group 'kali' owned by 'kali'
Jul 10 00:14:17 localhost userdel[2393]: removed shadow group 'kali' owned by 'kali'
# grep "userdel" /var/log/secure

5,su Switch users:
Jul 10 00:38:13 localhost su: pam_unix(su-l:session): session opened for user good by root(uid=0)

sudo Authorized execution:
sudo -l
Jul 10 00:43:09 localhost sudo:    good : TTY=pts/4 ; PWD=/home/good ; USER=root ; COMMAND=/sbin/shutdown -r now

2. Tools

2.1 Rootkit killing

• chkrootkit

  website: http://www.chkrootkit.org

  usage method:
  wget ftp://ftp.pangeia.com.br/pub/seg/pac/chkrootkit.tar.gz
  tar zxvf chkrootkit.tar.gz
  cd chkrootkit-0.52
  make sense
  #Check if there is no error after compilation
  ./chkrootkit
  

• rkhunter

  website: http://rkhunter.sourceforge.net

  usage method:
  Wget https://nchc.dl.sourceforge.net/project/rkhunter/rkhunter/1.4.4/rkhunter-1.4.4.tar.gz
  tar -zxvf rkhunter-1.4.4.tar.gz
  cd rkhunter-1.4.4
  ./installer.sh --install
  rkhunter -c
  

2.2 virus killing

• Clamav

  The official download address of ClamAV is: http://www.clamav.net/download.html

  Installation method I:

  1,install zlib: 
  wget http://nchc.dl.sourceforge.net/project/libpng/zlib/1.2.7/zlib-1.2.7.tar.gz 
  tar -zxvf  zlib-1.2.7.tar.gz
  cd zlib-1.2.7
  #Install gcc compilation environment: yum install gcc
  CFLAGS="-O3 -fPIC" ./configure --prefix= /usr/local/zlib/
  make && make install
  
  2,Add user group clamav And group members clamav: 
  groupadd clamav
  useradd -g clamav -s /bin/false -c "Clam AntiVirus" clamav
  
  3,install Clamav
  tar –zxvf clamav-0.97.6.tar.gz
  cd clamav-0.97.6
  ./configure --prefix=/opt/clamav --disable-clamav -with-zlib=/usr/local/zlib
  make
  make install
  
  4,to configure Clamav
  mkdir /opt/clamav/logs
  mkdir /opt/clamav/updata
  touch /opt/clamav/logs/freshclam.log
  touch /opt/clamav/logs/clamd.log
  cd /opt/clamav/logs
  chown clamav:clamav clamd.log
  chown clamav:clamav freshclam.log
  
  5,ClamAV use:
   /opt/clamav/bin/freshclam Upgrade virus library
  ./clamscan –h View the corresponding help information
  ./clamscan -r /home  Scan the home directory of all users and use it
  ./clamscan -r --bell -i /bin  scanning bin Directory and display the scanning results of the problematic files
  

  Installation mode 2:

  #install
  yum install -y clamav
  #Update virus library
  freshclam
  #Scanning method
  clamscan -r /etc --max-dir-recursion=5 -l /root/etcclamav.log
  clamscan -r /bin --max-dir-recursion=5 -l /root/binclamav.log
  clamscan -r /usr --max-dir-recursion=5 -l /root/usrclamav.log
  #Scan and kill virus
  clamscan -r  --remove  /usr/bin/bsd-port
  clamscan -r  --remove  /usr/bin/
  clamscan -r --remove  /usr/local/zabbix/sbin
  #View log discovery
  cat /root/usrclamav.log |grep FOUND
  

2.3 webshell killing

linux version:

Hippo webshell Killing: http://www.shellpub.com
 Deeply convinced Webshell Website backdoor detection tool: http://edr.sangfor.com.cn/backdoor_detection.html

2.4 RPM check

The system integrity can be verified through the - Va provided by rpm. Check all RPM software packages to see which commands have been replaced:

./rpm -Va > rpm.log

If everything is verified to be normal, no output will be generated. If there is any inconsistency, it will be displayed. The output format is an 8-bit long string. Each character is used to represent the comparison result between the file and an attribute in the RPM database. If yes (DOT) indicates that the test has passed.

The specific contents of the 8 information in the verification content are as follows:
		S         Does the file size change
		M         File type or file permissions( rwx)Is it changed
		5         file MD5 Check whether it is changed (it can be regarded as whether the file content is changed)
		D         Is the slave code changed in the device
		L         Whether the file path has changed
		U         Is the owner (owner) of the file changed
		G         Is the group of the file changed
		T         Is the modification time of the file changed

If the command is replaced and restored:

File extraction restore case:
rpm  -qf /bin/ls  query ls Which package does the command belong to
mv  /bin/ls /tmp  First ls Transfer to tmp Directory, causing ls The illusion of command loss
rpm2cpio /mnt/cdrom/Packages/coreutils-8.4-19.el6.i686.rpm | cpio -idv ./bin/ls extract rpm In the bag ls Command to the current directory/bin/ls lower
cp /root/bin/ls  /bin/ hold ls Copy command to/bin/Directory repair file missing

2.5 linux security check script

Github project address:

https://github.com/grayddq/GScan

https://github.com/ppabc/security_check

https://github.com/T0xst/linux

Quote a big man:
It's better to believe in books than to have no books. Tools are only auxiliary. Don't rely too much on them. The key lies in your ideas on how to solve problems.

Disclaimer: I firmly oppose the use of teaching methods to commit crimes. All criminal acts will be severely punished. The green network needs our joint maintenance. I recommend you to understand the principles behind them and better protect them. It is prohibited for anyone to reprint to other sites and use for any illegal purpose. If anyone does anything illegal with this, it has nothing to do with the author. It is hereby declared.