RSYNC+SERSYNC real-time synchronization (linuxR7)

Rsync(remote sync)

A data mirror backup software, which can perform differential backup according to the changes of data.
It can copy local data or remote data, or use SSH secure tunnel for encrypted data transmission.

Principle: the Rsync server defines the source data. The Rsync client will actually copy the data from the server to the local only after the source data is changed. If the source data is deleted on the server, the client data will also be deleted to ensure that the data between hosts is the same.

Rsync uses TCP protocol by default and listens to port 873.

Rsync mainly performs two actions
Push: push the local data to the backup server.
Pull pull: pull the data of the backup server locally.

rsync has three transmission modes

1, Local transmission
2, Remote transport ssh protocol:
3, Daemon rsync protocol:

1, Local transmission:
Data copy between single host directories, similar to cp
yum install rsync -y
rsync -avz /root/file.txt /tmp/

2, Remote transmission:
1. Transmit with ssh protocol (which means he needs to know the user name and password of the opposite end)
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
    
2. Set local / root / file Txt file is pushed to the / opt directory of 172.16.1.31 server, and the root user of 31 server is used.
rsync -avz /root/file.txt root@172.16.1.31:/opt
Pull the file in the remote server 172.16.1.31 /opt directory Txt file to local / mnt directory
rsync -avz root@172.16.1.31:/opt/file.txt /mnt/
  
3. Differences between push directories
rsync -avz /etc/ root@172.16.1.31:/opt / # add / indicates all contents in the push directory
rsync -avz /etc root@172.16.1.31 : / opt # do not add / means to push the directory itself and its contents

Remote transmission features:
Relying on the user identity and authority of the opposite host:
1. Use the system root user (unsafe)
2. Ordinary users using the system (there will be insufficient permissions)

3, Daemons:
Instead of using system users, you can use a virtual user (non-existent user) to push.
Service: run in our background and provide continuous service support
Push:
1. Use 172.16.1.31 to push the client Txt file to backup module of backup server 41 --------- > > >:
rsync -avz client.txt rsync_backup@172.16.1.41::backup

Pull:
2. Pull the data under the backup module of 172.16.1.41 backup server
rsync -avz rsync_backup@172.16.1.41::backup /opt

Configuration details
cat /etc/rsyncd.conf

uid = rsync — Represents user information for managing transmission data
gid = rsync — Represents user group information for managing transmission data
port = 873 — Specify service port information
#fake super = yes - pretend to be a super administrator
 The daemon needs to modify data permissions to back up data, Ability to modify permissions
use chroot = no — Realize the security configuration and realize the pseudo root security configuration
max connections = 200 — Set the maximum number of service connections (Number of concurrent connections)
timeout = 300 — There is a timeout set
pid file = /var/run/rsyncd.pid — Create pid Process number file
1. You can quickly stop the process according to the file number information
2. Determine the service running status according to whether the file exists or not
lock file = /var/run/rsync.lock — Limit according to the number of connections, Lock the service network connection
log file = /var/log/rsyncd.log — Backup service log file
ignore errors = yes — Ignoring the error message is not a particularly serious network error, Will ignore first, Ensure data transmission
read only = false — Confirm whether the backup directory is read-only
list = false   Can it be listed
hosts allow = 172.16.1.0/24 — Set whitelist function
hosts deny = 0.0.0.0/32 — Set blacklist function
auth users = rsync_backup — Define virtual authenticated users
secrets file = /etc/rsync.password — Define authenticated user password file

[backup] — A module corresponds to a backup directory
comment = "backup dir by oldboy" — Module function description
path = /backup — Define a backup data directory extension configuration parameter: reverse lookup = no — Close the service reverse parsing process

According to the interpretation of the configuration file, we found that some initialization operations need to be done:
Create a user for the rsync process to run (operating on the rsync server)

useradd -M -s /sbin/nologin rsync(Create process (user)
id rsync
uid=1008(rsync) gid=1008(rsync) group=1008(rsync)

Define virtual user name Rsync_ The password of bakcup needs to be established on both server and client, and can be skipped after ssh key authentication

echo "rsync_backup:123456" > /etc/rsync.passwd
chmod 600 /etc/rsync.passwd

–password-file=/etc/rsync.password # free

Create a directory of stored data / backup

mkdir /backup
chown -R rsync.rsync /backup

Sersync

A tool developed based on inotify and similar to inotify tools can record the name of a specific file or directory that changes (including adding, deleting and modifying) in the monitored directory.

RSYNC+SERSYNC real-time synchronization

Principle steps:

  1. Start the sersync service on the synchronization server (Master), and the sersync load monitors the changes of file system events in the configuration path.
  2. Call the rsync command to synchronize the updated files to the target server (S1 and S2).
  3. You need to configure sersync on the primary server and rsync server on the synchronization target server.

Environmental requirements:
For all hosts that receive data, start rsync service, open port 873, and wait for the host to send data at any time. The host realizes ssh login and non secret login to other nodes.

Implementation process:
1. Install rsync on all nodes
yum install rsync

2. Configure rsyncd Conf and synchronize to other nodes.
3. Start rsync service on other nodes and open ports.
4. On the publishing node (host), install the Sersync tool. (tools without yum source)
5. Edit the configuration file of sersync
Example:

<?xml version="1.0" encoding="ISO-8859-1"?>
<head version="2.5">
    <host hostip="localhost" port="8008"></host>
    <debug start="false"/>
    <fileSystem xfs="true"/>
    <filter start="false">
	<exclude expression="(.*)\.svn"></exclude>
	<exclude expression="(.*)\.gz"></exclude>
	<exclude expression="^info/*"></exclude>
	<exclude expression="^static/*"></exclude>
    </filter>
    <inotify>
	<delete start="true"/>
	<createFolder start="true"/>
	<createFile start="true"/>
	<closeWrite start="false"/>
	<moveFrom start="true"/>
	<moveTo start="true"/>
	<attrib start="false"/>
	<modify start="true"/>
    </inotify>

    <sersync>
	<localpath watch="/data/web-data/news.com">
	    <remote ip="192.168.10.21" name="news.com"/>  #Name is the rsync module name
	    <remote ip="192.168.10.22" name="news.com"/>
	    <remote ip="192.168.10.131" name="news.com"/>
	</localpath>
	<rsync>
	    <commonParams params="-av"/>

6. Check the help manual of the application and modify the kernel related parameters according to the suggestions
sersync2 -h

7. Perform full mirror synchronization manually
/usr/local/sersync/sersync2 -r -o /usr/local/sersync/confxml.xml #-r: before monitoring, push the monitoring directory and remote host with rsync command, - O: specify the configuration file, and conf XML is used by default XML file

8. Start real-time monitoring and enter real-time synchronization mode
/usr/local/sersync/sersync2 -d -n 10 -o /usr/local/sersync/confxml.xml #-n: Specifies the number of daemon threads to start. The default is 10

ps -aux view the monitoring process. Use kill to stop monitoring

Posted by LuaMadman on Thu, 05 May 2022 11:13:52 +0300