ZooKeeper configuration under Hadoop cluster

Install zookeeper environment

zookeeper installation package: https://pan.baidu.com/s/1fpdBs8kbjPj5rlrwusv1iw
Extraction code: h1wv
jdk environment to be prepared: Reference: https://blog.csdn.net/weixin_44147632/article/details/107796624
Decompression: tar -zxf zookeeper-3.4.5-cdh5 14.2. tar. gz -C /opt/bigdata/hadoop/
Renamed: MV zookeeper-3.4.5 zookeeper 345
Create a new directory under hadoop Directory: mkdir zookeeper
Establish soft connection: ln -s zookeeper345/ zookeeper/
Create a new directory mkdir zkdata under the zookeeper directory
Modify the zookeeper configuration file: CD / opt / bigdata / Hadoop / zookeeper 345 / conf
Rename: mv zoo_sample.cfg zoo.cfg file, then: VI zoo CFG modify configuration
Configuration information:

tickTime=2000	Milliseconds, heartbeat interval between server and client, and between servers. The minimum timeout is 2 times
initLimit=10	follower After startup and leade Synchronize data between and determine that the maximum time limit for external service status is 10*tickTime
syncLimit=5		follower and leader If between syncLimit*tickTime It cannot be confirmed within the heartbeat time leader Determine the follower Death, remove from service list
dataDir=/opt/bigdata/hadoop/zookeeper/zkdata
clientPort=2181
server.1=vwmaster:2888:3888
server.2=vwslave01:2888:3888
server.3=vwslave02:2888:3888
server.4=vwslave03:2888:3888

Create a myid file in the zkdata directory and save the corresponding server number 1 / 2 / 3 / 4 under the current host
Configure environment variables

export ZK_HOME=/opt/bigdata/hadoop/zookeeper345
export PATH=$ZK_HOME/bin:$$ZK_HOME/sbin:$PATH

Activate environment variable:

source /etc/profile

After copying the above process to the other three machines, on the four machines respectively, pay attention to modify the myid in the zkdata directory to the serial number of the corresponding node 1 / 2 / 3 / 4
Start zookeeper service: zkserver sh start
Close: zkserver sh stop
View status: zkserver sh status

"One click" to start the zookeeper script

starthd.sh

#!/bin/bash
ES_HNS='vwmaster vwslave01 vwslave02 vwslave03'
THIS_HN=$HOSTNAME

exitOnErr(){
 if [ 0 -ne $? ]
 then
  echo 'Exception:'$1
  exit 0
 fi
}

for HN in $ES_HNS
do
 if [ $THIS_HN != $HN ]
  then
  cd /root/hdstart/
  ./remotehd.sh $HN
  exitOnErr 'file ./remotehd.sh non-existent'
 else
  cd ~
 zkServer.sh start
 exitOnErr $HN'Start failed'
 cd ~
fi
done

remotehd.sh

#!/bin/bash
CMD='ssh root@'$1
$CMD > /dev/null 2>&1 << eeooff
 cd ~
 zkServer.sh start
 exit
eeooff
echo done!

zookeeper's Master-Slave selection mechanism

The follower is the slave and the leader is the host. With the voting selection mechanism, more than half of the nodes survive
For example, there are five nodes 1, 2, 3, 4 and 5. After node 1 is started, only one of them is waiting for other nodes to start. At this time, node 2 starts. It is found that only 1 and itself start. Continue to wait until the third node starts. At this time, more than half of the surviving nodes start voting. Since each node will vote first, node 3 naturally becomes a leader node, Other 1 and 2, including 4 and 5 started after, are followers. At this time, the leader maintains communication with each follower to view the status of each node.
If node 3 suddenly hangs up and the communication between follower and leader is broken, a new round of voting will be started to determine the new leader.

Tags: Linux Big Data Hadoop Zookeeper

Posted by madmindz on Sun, 22 May 2022 16:24:13 +0300