Linux ~ ~ ~ log management - December 3, 2020

Log management

Classification:
rsyslog system log management: concerns: what kind of program - "what logs are generated" -- where are they placed
logrotate log rotation: divide and manage a large number of logs and delete old logs.

rsyslog system log management

Process for processing logs

first kind

rsyslogd: system full-time log program. (included with the operating system)
Handle most of the logging,
Information related to system operation, such as login information, program startup and shutdown information and error information

Class II

httpd/nginx/mysql: various applications can log in their own way The corresponding procedures will be introduced step by step

Observe the rsyslog program

If the log is not recorded, you can enter / usr/sbin/rsyslogd -n to restart it

[root@localhost ~]# ps aux | grep rsyslog
root        962  0.0  0.2 216420  4472 ?        Ssl  19:42   0:00 /usr/sbin/rsyslogd -n

2, Common log files (system, process, application)

(generated by rsyslog and specified rules)

Tail - 10 / var / log / messages -- > system main log file

Tail - F / var / log / messages -- > dynamically view the tail of the log file
Example: use systemctl restart firewalld to verify the changes inside the log

Tail / var / log / secure -- > authentication and security

tail /var/log/yum.log --> yum

Tail / var / log / maillog -- > related to mail postfix (systemctl restart postfix)

Tail / var / log / cron -- > logs generated by crond and at processes

Tail / var / log / dmesg -- > is related to system startup

example:
Terminal I:

[root@localhost ~]# tailf /var/log/messages

Terminal II

[root@localhost ~]# systemctl restart httpd

Then there will be dynamic changes in terminal 1,

Understand:
tail /var/log/audit/audit. Log - > system audit log

tail /var/log/mysqld.log -> MySQL

Tail / var / log / xferlog - > related to accessing FTP server

Tail / var / log / wtmp - > currently logged in user (command: w)

Tail / var / log / BTMP - > recently logged in user (command last)

Tail / var / log / lastlog - > login status of all users (command lastlog)

Network log management

install
[root@localhost ~]#yum install -y httpd
[root@localhost ~]#systemctl  start  httpd
[root@localhost ~]#systemctl stop  firewalld
Adjust network

[root@localhost ~]#systemctl  restart  network
[root@localhost ~]#ip a

Access IP

Open the browser and enter the IP of the server to browse

Observation log:

[root@localhost ~]#tailf  /var/log/httpd/access_log

rsyslog configuration

Related procedures

Installed by default (if there is a problem, reinstall systemctl restart rsyslog.service)

[root@localhost ~]# yum install rsyslog   logrotate

Start program

[root@localhost ~]# systemctl  start rsyslog 

Relevant documents

Observe the configuration file of the logger
-q: Query the installation of software package
-c: Configuration file

[root@localhost ~]# rpm -qc rsyslog 
/etc/logrotate.d/syslog  -->rsyslogd Master profile for (critical)
/etc/rsyslog.conf   -->rsyslogd Relevant documents, definition level (learn about it)
/etc/sysconfig/rsyslog -->Related to log rotation (cutting)

Master profile

Tell the rsyslogd process what logs and where they should be stored.

RULES

RULES: RULES are a set of policies for generating logs and storing logs.
RULES are RULES, which are composed of three parts (equipment + level + storage location (PATH))
It consists of FACILITY+LEVEL+FILE.

[root@localhost ~]# vim /etc/rsyslog.conf

mail: device *: any level (everything is recorded) (. Is the separator)

authpriv.* 		/var/log/secure(SSH Information)

mail.* 			-/var/log/maillog((email)
Here's one-Symbol, Indicates that it is recorded asynchronously (it will be recorded later, there is a time difference), Because the log is usually large (synchronous, and the generated information is stored immediately)

cron.* 			/var/log/cron((create task)

*.info;mail.none;authpriv.none;cron.none                /var/log/messages   
System logs exclude mail, authentication, and plan logs.

facility equipment

It is the system's definition of certain types of APP events. For example, AUTHPRIV is a security event and CRON is a planned task event. Used to collect similar program logs.

Type:
LOG_ Syslog -- > log generated by syslogd itself

LOG_ Authpriv -- > Security Certification

LOG_ Cron -- > scheduler (cron and at)

LOG_ Mail -- > mail subsystem

LOG_ User (default) - > user related

LOG_ Daemon -- > daemon

LOG_ FTP -- > File Server ftp daemon

LOG_ Kernel -- > kernel messages

LOG_ LPR -- > printer equipment: printer subsystem

LOG_ LOCAL0 through LOG_ Local7 -- > user defined device

level (from high to low)

LOG_ Emerg -- > emergency, fatal, the service cannot continue to run, for example, the configuration file is lost
LOG_ Alert -- > alarm, which needs to be handled immediately. For example, 95% of the disk is empty
LOG_ Crit -- > fatal behavior
LOG_ Err -- > error behavior
LOG_ Warning -- > warning message
LOG_ Note -- > General and important standard information
LOG_ Info -- > standard information
LOG_ Debug -- > debugging information, required for troubleshooting, generally not recommended
From bottom to top, from low level to high level, less and less information is recorded (the lower the level, the more trivia is recorded)

info, semicolon, mail, semicolon and authentication of all devices are not put in

Schematic diagram of rules:
Objective: to understand the working mechanism of log by drawing.
Black box: Server

Example of program type

As for the connection between the program and the device, the program itself will decide what kind of device to give the log to. For example, the SSH program will select safe devices. This is defined by the developer.

Modify the device type of ssh program

Change the storage location to LOCAL5

[root@localhost ~]# vim /etc/ssh/sshd_config 

Rules for modifying rsyslog program

[root@localhost ~]# vim /etc/rsyslog.conf 
local5.*     /var/log/server  Find the corresponding position and write down the custom server

Restart rsyslog program and ssh program

[root@localhost ~]# systemctl restart  rsyslog.service sshd

Log in to the server with another terminal and observe the new log file.

[root@localhost ~]# tailf /var/log/messages 

logrotate log rotation

brief introduction

Log: records all kinds of information when the program is running.
Through logs, you can analyze user behavior, record running tracks, and find program problems.
Log rotation is like a black box in an aircraft. No matter how important the recorded information is, it can only record what happened in the last period of time.
In order to save space and facilitate sorting, log files often need to press! Time or! Size and other dimensions are divided into multiple copies to delete log files with a long time.

Working principle: rotate according to the configuration

Type of configuration file

Main configuration file: / etc / logrotate Conf -- > (determines how each log file rotates)
Sub configuration folder: / etc / logrotate D / * -- > custom configuration for easy management

Observe the main and sub files

[root@qianfeng ~]# ls /etc/logrotate.conf /etc/logrotate.d/
/etc/logrotate.conf
/etc/logrotate.d/:
acpid cups iscsiuiolog ppp rpm subscription-manager up2date wpa_supplicant
conman httpd mgetty psacct setroubleshoot syslog vsftpd.log yum

Introduction to master profile

[root@localhost ~]# vim /etc/logrotate.conf 
=========Global settings==========
weekly 		//Cycle of rotation
rotate 4 		//Keep 4 copies
create 		//Create a new file after rotation
dateext 		//Use date as suffix
#compress 	 // Compression will occupy CPU resources, so it is not opened here
include /etc/logrotate.d	//Contains the sub configuration files under this directory
/var/log/wtmp { 	//Method of setting rotation for a log file
monthly 				//Rotate once a month (less log content)
minsize 1M 		//Rotate at least 1M, monthly and minsize (maxsize 1M: rotate at 1M, even if the time is not yet up) create 0664 root utmp 	 // Create a new file after rotation and set permissions
rotate 1 		//Keep one copy (when rotating to the second copy, delete the first copy and leave only one copy)
}  --->boundary {}The content of is set for this log
/var/log/btmp {
missingok 		//No prompt for loss
monthly 			//Rotate once a month
create 0600 root utmp 	//Create a new file after rotation and set permissions (only the owner can read and write)
rotate 1 		//Keep one copy
}

The messages in the red box with the date are the old file, the log contents from 20201124 to 1129, and the latest log without the date. During rotation, the old file will be renamed (plus the time) - > so as to create a new file

yum log rotation instance

Target file of rotation / var / log / yum.com log

Configure rotation rules

[root@localhost log]# vim /etc/logrotate.d/yum 
/var/log/yum.log {
missingok		//Missing do not execute
 notifempty		//Empty file regardless of transfer
maxsize 30k		//Up to 30k rotation, daily or size
 yearly		//Or once a year
daily		//Reduce cycle to 1 day
rotate 3		//Rotation of old documents (three times reserved)
create 0777 root root
}

Test:
Error demonstration:

Manual rotation:

[root@localhost log]# /usr/sbin/logrotate /etc/logrotate.conf 
[root@localhost log]# ll yum*
-rw-------. 1 root root 963 12 March 20:26 yum.log    -->There is only one file because the date has not changed
[root@localhost log]# ll yum*
-rwxrwxrwx  1 root root   0 12 April 00:01 yum.log
-rw-------. 1 root root 963 12 March 20:26 yum.log-20201204

Even if the day is rotated many times, there will only be one document of the day (because the date has not changed – > there is a time stamp)

Correct demonstration:
Modify the time and manually trigger the rotation

date month day hour (8 digits)
Move time to the future

[root@localhost logrotate.d]# hwclock
2020 Thursday, December 3, 2011 11:20:51  -0.380237 second
[root@localhost log]# date 12040000
2020 Friday, December 4, 2000 00:00:00 CST
[root@localhost log]# date
2020 Friday, December 4, 2000 00:00:01 CST
[root@localhost log]# /usr/sbin/logrotate /etc/logrotate.conf 

After modifying the time, synchronize the time to return to normal

[root@localhost logrotate.d]# ntpdate ntp.aliyun.com

More than one log file has appeared

[root@localhost log]# ll yum*
-rwxrwxrwx  1 root root   0 12 May 00:03 yum.log
-rw-------. 1 root root 963 12 March 20:26 yum.log-20201204
-rwxrwxrwx  1 root root   0 12 April 00:01 yum.log-20201205

About time

[root@localhost log]#grep 'yum' /var/lib/logrotate/logrotate.status / / records the latest rotation time of all log files
[root@localhost log]#"/ var/log/yum.log" 2019-3-31-10:0:23 / / if there is no rotation, there are only records for the first time

Test httpd
Enter sub configuration file:

[root@localhost logrotate.d]# vim /etc/logrotate.d/httpd

Amend the content as follows

/var/log/httpd/*log {
    missingok
    daily
   # Notiempty empty files do not rotate
    #sharedscripts shared scripts
    #delaycompress
    #The postrotate script starts
     #   /bin/systemctl reload httpd. Service > / dev / null 2 > / dev / null | true -- > will automatically restart httpd 
    #End of endscript script
}
   

As soon as the terminal is opened, observe the dynamic situation in real time

[root@localhost ~]# watch -n1 'ls -l  /var/log/httpd/*'

At this time, when the website logs in to the local IP and refreshes, the generated data exists at the place indicated by the arrow

Here is the index number of httpd process

reason:
The inode of httpd is associated with the system. If the file is not updated without rotation, the inode will remain unchanged and the information will be written to the old file

Restarting httpd will generate a new log file (new index number), and the information will be written to the latest file (but it needs to be restarted manually every day)

[root@localhost logrotate.d]# systemctl restart httpd


After modification

[root@localhost ~]# vim /etc/logrotate.d/httpd
[root@localhost ~]# systemctl restart httpd


It can be restored to normal, and the latest information will be written to the latest file (the file will restart automatically)

Tags: Linux CentOS

Posted by stfuji on Wed, 04 May 2022 09:11:00 +0300