Big data environment construction - Kafka and Zookeeper

foreword

https://kafka.apache.org/downloads

Stand-alone environment construction

Install zookeeper

kafka depends on zookeeper, which is built-in in the installation package. You can skip this step if you use the built-in one

You can also download it separately

https://zookeeper.apache.org/releases.html#download

install kafka

wget https://archive.apache.org/dist/kafka/2.8.1/kafka_2.12-2.8.1.tgz
tar -xzf kafka_2.12-2.8.1.tgz
cd kafka_2.12-2.8.1
copy

add environment variable

name

path

KAFKA_HOME

D:\Tools\bigdata\kafka_2.12-2.8.1

Path

%KAFKA_HOME%\bin

Start ZK

Start ZK

%KAFKA_HOME%/bin/windows/zookeeper-server-start.bat %KAFKA_HOME%/config/zookeeper.properties
copy

Enter

%KAFKA_HOME%/bin/windows/zookeeper-shell.bat localhost
copy

input the command

#View the kafka related nodes in the root directory of zk
ls / 
#View kafka nodes
ls /brokers
ls /brokers/topics
copy

Common commands

  1. Display the files in the root directory: ls / Use the ls command to view the content contained in the current ZooKeeper
  2. Display the files in the root directory: ls2 / View the current node data and see data such as the number of updates
  3. Create the file, and set the initial content: create /zk "test" Creates a new znode "zk" and the string associated with it
  4. Get the file content: get /zk to make sure the znode contains the string we created
  5. Modify the content of the file: set /zk "zkbak" Set the string associated with zk
  6. Delete file: delete /zk delete the znode just created
  7. Quit the client: quit
  8. Help command: help

Change setting

Modify the configuration file config/server.properties

#The broker.id property must be unique in the kafka cluster
broker.id=0
#The ip of the machine deployed by kafka and the port number of the service provided
listeners=PLAINTEXT://localhost:9092  
#kafka's message storage file
log.dir=/kafka/kafka-logs
#The address where kafka connects to zookeeper
zookeeper.connect=localhost:2181
copy

Start Kafka

Linux

Startup script syntax:

kafka-server-start.sh [-daemon] server.properties
copy

It can be seen that the configuration path of server.properties is a mandatory parameter, -daemon means running as a background process, otherwise the service will be stopped after the ssh client exits. (Note that the ip address associated with the linux hostname will be used when starting kafka, so you need to configure the hostname and linux ip mapping to the local host, use vim /etc/hosts)

View host name hostname View hostname hostname -i: View the IP corresponding to the machine Modify the hostname:

vi /etc/hostname
copy

Start kafka and run the log in the server.log file in the logs directory

bin/kafka-server-start.sh -daemon config/server.properties  #Starts in the background and does not print logs to the console
copy

or use

bin/kafka-server-start.sh config/server.properties &
copy

stop kafka

bin/kafka-server-stop.sh
copy

Win

Start kafka and run the log in the server.log file in the logs directory

%KAFKA_HOME%/bin/windows/kafka-server-start.bat %KAFKA_HOME%/config/server.properties
copy

stop kafka

%KAFKA_HOME%/bin/windows/kafka-server-stop.bat
copy

command test message

Linux

1. Create a theme

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
copy

2. List all topics

bin/kafka-topics.sh --list --zookeeper localhost:2181
copy

3. Send a message

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
copy

4. Consume news, the default is to consume the latest news

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
copy

5. Create multiple partition topics

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 2 --topic test1
copy

Expansion partition

bin/kafka-topics.sh -alter --partitions 3 --zookeeper localhost:2181 --topic test1
copy

6. View the topic situation

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test1
copy

7. The consumption offset of the consumer is maintained by the consumer himself, check the consumption offset of the theme

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group test
copy

Win

1. Create a theme

%KAFKA_HOME%/bin/windows/kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
copy

2. List all topics

%KAFKA_HOME%/bin/windows/kafka-topics.bat --list --zookeeper localhost:2181
copy

3. Send a message

%KAFKA_HOME%/bin/windows/kafka-console-producer.bat --broker-list localhost:9092 --topic test
copy

4. Consume news, the default is to consume the latest news

%KAFKA_HOME%/bin/windows/kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test
copy

5. Create multiple partition topics

%KAFKA_HOME%/bin/windows/kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 2 --topic test1
copy

Expansion partition

%KAFKA_HOME%/bin/windows/kafka-topics.bat -alter --partitions 3 --zookeeper localhost:2181 --topic test1
copy

6. View the topic situation

%KAFKA_HOME%/bin/windows/kafka-topics.bat --describe --zookeeper localhost:2181 --topic test
copy

7. The consumption offset of the consumer is maintained by the consumer himself, check the consumption offset of the theme

%KAFKA_HOME%/bin/windows/kafka-consumer-groups.bat --bootstrap-server localhost:9092 --describe --group test
copy

Cluster building

cluster server

zookeeper 1: 192.168.10.10 kafka 2: 192.168.10.11 and 192.168.10.12

Modify the configuration file

On the 192.168.10.11 server

vi config/server.properties
copy

The content is as follows

#The broker.id property must be unique in the kafka cluster
broker.id=0
#The ip of the machine deployed by kafka and the port number of the service provided
listeners=PLAINTEXT://192.168.10.11:9093  
log.dir=/usr/local/data/kafka-logs-1
#The address where kafka connects to the zookeeper. To form a cluster of multiple kafka instances, the corresponding connected zookeepers must be the same
zookeeper.connect=192.168.10.10:2181
copy

On the 192.168.10.12 server

vi config/server.properties
copy

The content is as follows

#The broker.id property must be unique in the kafka cluster
broker.id=1
#The ip of the machine deployed by kafka and the port number of the service provided
listeners=PLAINTEXT://192.168.10.12:9093  
log.dir=/usr/local/data/kafka-logs-1
#The address where kafka connects to the zookeeper. To form a cluster of multiple kafka instances, the corresponding connected zookeepers must be the same
zookeeper.connect=192.168.10.10:2181
copy

Start two kafka s separately

bin/kafka-server-start.sh -daemon config/server.properties

bin/kafka-server-start.sh -daemon config/server.properties
copy

test

Create a new topic with the number of replicas set to 3 and the number of partitions set to 2

bin/kafka-topics.sh --create --zookeeper 192.168.10.10:2181 --replication-factor 2 --partitions 2 --topic my-topic
copy

View topic information

bin/kafka-topics.sh --describe --zookeeper 192.168.10.10:2181 --topic my-topic
copy

Send a message to the topic my-topic

bin/kafka-console-producer.sh --broker-list 192.168.10.11:9092,192.168.10.12:9092 --topic my-topic
copy

Consuming messages

bin/kafka-console-consumer.sh --bootstrap-server 192.168.10.11:9092,192.168.10.12:9092 --from-beginning --topic my-topic
copy

Posted by iversonm on Wed, 25 May 2022 08:33:10 +0300