Quick start rsync to achieve remote synchronization

 1 grammar
 2 rsync [options] original position target position
 3 
 4 Common options
 5 -a:          Archive mode, recursively and preserves object properties, equivalent to-rlptgoD
 6 -v:          Display the details of the synchronization process ( verbose)information
 7 -z:          Compression when transferring files ( compress)
 8 -H:          keep hard link files
 9 -A:          reserve ACL attribute information
10 -p:          Preserve permission flags for files
11 -t:          Keep time stamps for files
12 -g:          keep the group flag of the file
13 -o:          Keep the owner tag of the file
14 -delete:     Delete files that are in the target location but not in the original location
15 -checksum:   Decide whether to skip files based on the object's checksum

2.2. Two representation methods of configuration source

1 Format 1
2 username@host address::shared module name
3 
4 Format 2
5 rsync://username@host address/shared module name

2.3. Interaction-free processing of rsync sources

Use --password-file= password file

Three, rsync remote synchronization deployment

3.1. Environmental description

3.2. Configure rsync source server A

①Check if rsync is installed

1 [root@rsync ~]# rpm -q rsync
2 rsync-3.0.9-18.el7.x86_64

② Modify the configuration file

 1 [root@rsync ~]# vi /etc/rsyncd.conf 
 2 uid = nobody
 3 gid = nobody
 4 use chroot = yes                   #Imprisoned in the host directory
 5 address = 20.0.0.10              #listen address
 6 port 873                                    #The port number
 7 pid file = /var/run/rsyncd.pid       #Process file location
 8 log file = /var/log/rsyncd.log         #log file location
 9 hosts allow = 20.0.0.0/24               #address range allowed
10 dont compress   = *.gz *.tgz *.zip *.z *.Z *.rpm *.deb *.bz2         #No longer compress files in these formats
11 [wwwroot]
12 path = /var/www/html                   #Synchronized directory
13 comment = www.xuhao.com
14 read only = yes                        #read only
15 auth users = backuper              
16 secrets file = /etc/rsyncd_users.db      #User password storage location

③Create a password file for the backup user

1 [root@rsync ~]# vi /etc/rsyncd_users.db
2 backuper:123456

④Set the execute permission to the password file

1 [root@rsync ~]# chmod 600 /etc/rsyncd_users.db

⑤Start rsync

1 [root@rsync ~]# rsync --daemon
2 [root@rsync ~]# netstat -antp |grep rsync
3 tcp        0      0 20.0.0.10:873           0.0.0.0:*               LISTEN      14461/rsync

⑥ Install httpd, there is a shared directory of /var/www/html, or you can create it yourself

1 [root@rsync ~]# yum -y install httpd
2 [root@rsync ~]# echo "this is test web" > /var/www/html/index.html

3.3. Client server B test

①Format 1

 1 [root@client ~]# rsync -avz backuper@20.0.0.10::wwwroot /opt
 2 Password: 
 3 receiving incremental file list
 4 ./
 5 index.html
 6 
 7 sent 83 bytes  received 172 bytes  56.67 bytes/sec
 8 total size is 17  speedup is 0.07
 9 [root@client ~]# cat /opt/index.html          #Check if it is successful
10 this is test web

②Format 2

 1 Delete the synced files and test again
 2 [root@client ~]# rm -rf /opt/index.html 
 3 [root@client ~]# cat /opt/index.html 
 4 cat: /opt/index.html: does not have that file or directory
 5 
 6 Synchronize
 7 [root@client ~]# rsync -avz rsync://backuper@20.0.0.10/wwwroot /opt
 8 Password: 
 9 receiving incremental file list
10 ./
11 index.html
12 
13 sent 83 bytes  received 172 bytes  56.67 bytes/sec
14 total size is 17  speedup is 0.07
15 [root@client ~]# cat /opt/index.html 
16 this is test web

③ Password-free login

 1 Create a password-free login file
 2 [root@client ~]# vi /etc/server.pass
 2 123456
 3 [root@client ~]# chmod 600 /etc/server.pass 
 4 [root@client ~]# rm -rf /opt/index.html 
 5 [root@client ~]# rsync -avz --delete --password-file=/etc/server.pass backuper@20.0.0.10::wwwroot /opt
 6 receiving incremental file list
 7 deleting rh/
 8 ./
 9 index.html
10 
11 sent 83 bytes  received 172 bytes  72.86 bytes/sec
12 total size is 17  speedup is 0.07
13 [root@client ~]# cat /opt/index.html 
14 this is test web

Fourth, rsync+inotify real-time synchronization

4.1, rsync real-time synchronization

4.1.1. Insufficiency of regular synchronization

①The time for performing backup is fixed, the delay is obvious, and the real-time performance is poor

② When the synchronization source does not change for a long time, intensive periodic tasks are unnecessary

4.1.2. Advantages of real-time synchronization

① Once the synchronization source changes, start the backup immediately

②As long as there is no change in the synchronization source, no backup will be performed

4.2. About inotify

The inotify mechanism of the Linux kernel

① Available from version 2.6.13

②You can monitor the changes of the file system and make a notification response

③Auxiliary software: inotify-tools

4.3. Environmental description

4.4. Configure rsync source server A

①Turn off read-only mode

1 [root@rsync ~]# vi /etc/rsyncd.conf 
2 read only = no

②The client changes the kernel parameters, specifies the queue size, the maximum number of monitored instances, and the maximum number of monitored files per instance

 1 [root@rsync ~]# vim /etc/sysctl.conf
 2 fs.inotify.max_queued_events = 16384     #Maximum number of queues to monitor events
 3 fs.inotify.max_user_instances = 1024       #Maximum number of instances to monitor
 4 fs.inotify.max_user_watches = 1048576     #Maximum number of files monitored per instance
 5 
 6 load
 7 [root@rsync ~]# sysctl -p
 8 fs.inotify.max_queued_events = 16384
 9 fs.inotify.max_user_instances = 1024
10 fs.inotify.max_user_watches = 1048576

③Restart the rsync service, note: the difference between kill pid number and kill -9 pid number: the latter will not delete the pid file, causing the service to fail

 1 [root@rsync ~]# cd /var/run
 2 [root@rsync run]# cat rsyncd.pid 
 3 14461
 4 [root@rsync run]# kill 14461            #delete pid file
 5 [root@rsync run]# ll | grep rsync.pid
 6 [root@rsync run]# rsync --daemon
 7 [root@rsync run]# ll | grep rsyncd.pid 
 8 -rw-r--r--.  1 root           root              6 11 Jan 12 13:19 rsyncd.pid
 9 
10 [root@rsync run]# cat rsyncd.pid 
11 54028
12 [root@rsync run]# kill -9 54028        #If the pid file is not deleted, it will not start
13 [root@rsync run]# ll | grep rsyncd.pid 
14 -rw-r--r--.  1 root           root              6 11 Jan 12 13:19 rsyncd.pid
15 [root@rsync run]# rsync --daemon
16 [root@rsync run]# failed to create pid file /var/run/rsyncd.pid: File exists
17 
18 [root@rsync run]# rm -rf rsyncd.pid    #Delete the pid file and restart
19 [root@rsync run]# rsync --daemon
20 [root@rsync run]# netstat -anpt | grep rsync
21 tcp        0      0 20.0.0.10:873           0.0.0.0:*               LISTEN      5406

4.5, client server B configuration

① Unzip inotify and install

inotifywait: for continuous monitoring, real-time output results

inotifywatch: used for short-term monitoring, and the results will be output after the task is completed

1 [root@client ~]# tar zxf inotify-tools-3.14.tar.gz 
2 [root@client ~]# cd inotify-tools-3.14/
3 [root@client inotify-tools-3.14]# ./configure 
4 [root@client inotify-tools-3.14]# make && make install

②Test whether the inotifywait monitoring command is used normally

1 [root@client inotify-tools-3.14]# mkdir -p /var/www/html 
2 [root@client inotify-tools-3.14]# inotifywait -mrq -e modify,create,move,delete /var/www/html

③ After executing the last command above, it is in a monitoring state and cannot perform input operations. After opening a new terminal device

 1 Another terminal file test
 2 [root@client ~]# cd /var/www/html/
 3 [root@client html]# touch a.txt
 4 [root@client html]# vi 1.tat
 5 [root@client html]# rm -rf 1.txt 
 6 [root@client html]# mv a.txt /opt
 7 
 8 Monitor end view
 9 [root@client inotify-tools-3.14]# inotifywait -mrq -e modify,create,move,delete /var/www/html
10 /var/www/html/ CREATE a.txt
11 /var/www/html/ CREATE 1.txt
12 /var/www/html/ DELETE 1.txt
13 /var/www/html/ MOVED_FROM a.txt

④ Write a script on the client to combine inotify monitoring and rsync remote synchronization

 1 [root@client inotify-tools-3.14]# vi /opt/inotify.sh
 2 #!/bin/bash
 3 INOTIFY="inotifywait -mrq -e modify,create,attrib,move,delete /var/www/html"
 4 RSYNC="rsync -azH --delete --password-file=/etc/server.pass /var/www/html/  backuper@20.0.0.10::wwwroot/"
 5 $INOTIFY | while read DIRECTORY EVENT FILE   #Read monitoring records one by one
 6 do
 7         if [ $(pgrep rsync | wc -l) -le 0 ];then
 8             $RSYNC
 9         fi
10 done
11 
12 [root@client inotify-tools-3.14]# chmod +x /opt/inotify.sh 
13 [root@client inotify-tools-3.14]# cd /opt/
14 
15 Note: Sync directory permissions on both sides are set to 777
16 [root@rsync run]# chmod 777 /var/www/html/
17 [root@client html]# chmod 777 /var/www/html/
18 
19 run script
20 [root@client opt]# ./inotify.sh

⑤Create a file in the client /var/www/html directory and check whether the source /var/www/html directory is synchronized to

 1 client/var/www/html Create a directory under the directory
 2 [root@client html]# touch a.txt
 3 [root@client html]# touch b.txt
 4 [root@client html]# touch c.txt
 5 [root@client html]# touch d.txt
 6 
 7 View source/var/www/html directory, the newly created and existing files are successfully synchronized, and the owner and group are both nobody user
 8 [root@rsync ~]# cd /var/www/html/
 9 [root@rsync html]# ll
10 Total usage 0
11 -rw-r--r--. 1 nobody nobody 0 11 Jan 12 13:45 a.txt
12 -rw-r--r--. 1 nobody nobody 0 11 Jan 12 13:45 b.txt
13 -rw-r--r--. 1 nobody nobody 0 11 Jan 12 13:47 c.txt
14 -rw-r--r--. 1 nobody nobody 0 11 Jan 12 13:48 d.txt

Five, matters needing attention

5.1, the difference between kill pid number and kill -9 pid number

 1 [root@rsync ~]# cd /var/run
 2 [root@rsync run]# cat rsyncd.pid
 3 14461
 4 [root@rsync run]# kill 14461            #delete pid file
 5 [root@rsync run]# ll | grep rsync.pid
 6 [root@rsync run]# rsync --daemon
 7 [root@rsync run]# ll | grep rsyncd.pid
 8 -rw-r--r--.  1 root           root              6 11 Jan 12 13:19 rsyncd.pid
 9 
10 [root@rsync run]# cat rsyncd.pid
11 54028
12 [root@rsync run]# kill -9 54028        #If the pid file is not deleted, it will not start
13 [root@rsync run]# ll | grep rsyncd.pid
14 -rw-r--r--.  1 root           root              6 11 Jan 12 13:19 rsyncd.pid
15 [root@rsync run]# rsync --daemon
16 [root@rsync run]# failed to create pid file /var/run/rsyncd.pid: File exists
17 
18 [root@rsync run]# rm -rf rsyncd.pid    #Delete the pid file and restart
19 [root@rsync run]# rsync --daemon
20 [root@rsync run]# netstat -anpt | grep rsync
21 tcp        0      0 20.0.0.10:873           0.0.0.0:*               LISTEN      5406

5.2. Notes on synchronization

When using the following command for uplink synchronization (upload), the local directory should be added with / at the end, otherwise the directory will be synchronized to the target folder of the other party, not just the files in the local directory

①Change the above script source directory /var/www/html/ to /var/www/html

 1 [root@client opt]# vi /opt/inotify.sh
 2 #!/bin/bash
 3 INOTIFY="inotifywait -mrq -e modify,create,attrib,move,delete /var/www/html"
 4 RSYNC="rsync -azH --delete --password-file=/etc/server.pass /var/www/html  backuper@20.0.0.10::wwwroot"
 5 $INOTIFY | while read DIRECTORY EVENT FILE   #Read monitoring records one by one
 6 do
 7         if [ $(pgrep rsync | wc -l) -le 0 ];then
 8             $RSYNC
 9         fi
10 done
11 
12 run script
13 [root@client opt]# ./inotify.sh

②Create a file in the /var/www/html directory of the client, and check the /var/www/html directory of the source to find the time directory for direct synchronization

 1 Client writes data
 2 [root@client html]# touch e.txt
 3 
 4 rsync View on
 5 [root@rsync html]# ll
 6 Total usage 0
 7 -rw-r--r--. 1 nobody nobody  0 11 Jan 12 13:45 a.txt
 8 -rw-r--r--. 1 nobody nobody  0 11 Jan 12 13:45 b.txt
 9 -rw-r--r--. 1 nobody nobody  0 11 Jan 12 13:47 c.txt
10 -rw-r--r--. 1 nobody nobody  0 11 Jan 12 13:48 d.txt
11 drwxrwxrwx. 2 nobody nobody 71 11 Jan 12 14:01 html              #The entire directory is synchronized to the target host          
12 [root@rsync html]# cd html/
13 [root@rsync html]# ll
14 Total usage 0
15 -rw-r--r--. 1 nobody nobody 0 11 Jan 12 13:45 a.txt
16 -rw-r--r--. 1 nobody nobody 0 11 Jan 12 13:45 b.txt
17 -rw-r--r--. 1 nobody nobody 0 11 Jan 12 13:47 c.txt
18 -rw-r--r--. 1 nobody nobody 0 11 Jan 12 13:48 d.txt
19 -rw-r--r--. 1 nobody nobody 0 11 Jan 12 14:01 e.txt

Posted by 1veedo on Sat, 07 May 2022 12:22:51 +0300

Algorithm Zone

Quick start rsync to achieve remote synchronization

1. Configure the rsync source server

1.1, about rsync

1.2, rsync synchronization source

1.3, the basic idea

1.4. Application example

1.5, the configuration file rsync.conf

1.6, rsync account file

1.7, enable rsync service

2. Use rsync backup tool

2.1, the usage of rsync command