nacos cluster building (docker compose solution)

nacos cluster construction (docker compose scheme)

Stay up to date with the latest version: the current version is nacos:1.4.0

Prepare

Note: This article requires knowledge of linux, docker, docker-compose and nginx

  • At least three linux machines

    (Install docker, docker-compose, installation tutorial: pay attention to the public number: Jeesoul open source)

  • Both machines can access each other

start

  • Prepare three machines
    • 10.0.0.5
    • 10.0.0.6
    • 10.0.0.7

This mirror version: nacos/nacos-server:1.4.0(nacos:nacos-server/latest)

The current version is the latest version (2020-12-04)

  • Using an external mysql database
  • nacos 1.4.0 sql address
    • https://github.com/alibaba/nacos/blob/1.4.0/distribution/conf/nacos-mysql.sql
/*
 * Copyright 1999-2018 Alibaba Group Holding Ltd.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

/******************************************/
/*   Database full name = nacos_config   */
/*   table name = config_info   */
/******************************************/
CREATE TABLE `config_info` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'id',
  `data_id` varchar(255) NOT NULL COMMENT 'data_id',
  `group_id` varchar(255) DEFAULT NULL,
  `content` longtext NOT NULL COMMENT 'content',
  `md5` varchar(32) DEFAULT NULL COMMENT 'md5',
  `gmt_create` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'creation time',
  `gmt_modified` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Change the time',
  `src_user` text COMMENT 'source user',
  `src_ip` varchar(20) DEFAULT NULL COMMENT 'source ip',
  `app_name` varchar(128) DEFAULT NULL,
  `tenant_id` varchar(128) DEFAULT '' COMMENT 'Tenant field',
  `c_desc` varchar(256) DEFAULT NULL,
  `c_use` varchar(64) DEFAULT NULL,
  `effect` varchar(64) DEFAULT NULL,
  `type` varchar(64) DEFAULT NULL,
  `c_schema` text,
  PRIMARY KEY (`id`),
  UNIQUE KEY `uk_configinfo_datagrouptenant` (`data_id`,`group_id`,`tenant_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='config_info';

/******************************************/
/*   Database full name = nacos_config   */
/*   table name = config_info_aggr   */
/******************************************/
CREATE TABLE `config_info_aggr` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'id',
  `data_id` varchar(255) NOT NULL COMMENT 'data_id',
  `group_id` varchar(255) NOT NULL COMMENT 'group_id',
  `datum_id` varchar(255) NOT NULL COMMENT 'datum_id',
  `content` longtext NOT NULL COMMENT 'content',
  `gmt_modified` datetime NOT NULL COMMENT 'Change the time',
  `app_name` varchar(128) DEFAULT NULL,
  `tenant_id` varchar(128) DEFAULT '' COMMENT 'Tenant field',
  PRIMARY KEY (`id`),
  UNIQUE KEY `uk_configinfoaggr_datagrouptenantdatum` (`data_id`,`group_id`,`tenant_id`,`datum_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='Add tenant field';


/******************************************/
/*   Database full name = nacos_config   */
/*   table name = config_info_beta   */
/******************************************/
CREATE TABLE `config_info_beta` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'id',
  `data_id` varchar(255) NOT NULL COMMENT 'data_id',
  `group_id` varchar(128) NOT NULL COMMENT 'group_id',
  `app_name` varchar(128) DEFAULT NULL COMMENT 'app_name',
  `content` longtext NOT NULL COMMENT 'content',
  `beta_ips` varchar(1024) DEFAULT NULL COMMENT 'betaIps',
  `md5` varchar(32) DEFAULT NULL COMMENT 'md5',
  `gmt_create` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'creation time',
  `gmt_modified` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Change the time',
  `src_user` text COMMENT 'source user',
  `src_ip` varchar(20) DEFAULT NULL COMMENT 'source ip',
  `tenant_id` varchar(128) DEFAULT '' COMMENT 'Tenant field',
  PRIMARY KEY (`id`),
  UNIQUE KEY `uk_configinfobeta_datagrouptenant` (`data_id`,`group_id`,`tenant_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='config_info_beta';

/******************************************/
/*   Database full name = nacos_config   */
/*   table name = config_info_tag   */
/******************************************/
CREATE TABLE `config_info_tag` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'id',
  `data_id` varchar(255) NOT NULL COMMENT 'data_id',
  `group_id` varchar(128) NOT NULL COMMENT 'group_id',
  `tenant_id` varchar(128) DEFAULT '' COMMENT 'tenant_id',
  `tag_id` varchar(128) NOT NULL COMMENT 'tag_id',
  `app_name` varchar(128) DEFAULT NULL COMMENT 'app_name',
  `content` longtext NOT NULL COMMENT 'content',
  `md5` varchar(32) DEFAULT NULL COMMENT 'md5',
  `gmt_create` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'creation time',
  `gmt_modified` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Change the time',
  `src_user` text COMMENT 'source user',
  `src_ip` varchar(20) DEFAULT NULL COMMENT 'source ip',
  PRIMARY KEY (`id`),
  UNIQUE KEY `uk_configinfotag_datagrouptenanttag` (`data_id`,`group_id`,`tenant_id`,`tag_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='config_info_tag';

/******************************************/
/*   Database full name = nacos_config   */
/*   table name = config_tags_relation   */
/******************************************/
CREATE TABLE `config_tags_relation` (
  `id` bigint(20) NOT NULL COMMENT 'id',
  `tag_name` varchar(128) NOT NULL COMMENT 'tag_name',
  `tag_type` varchar(64) DEFAULT NULL COMMENT 'tag_type',
  `data_id` varchar(255) NOT NULL COMMENT 'data_id',
  `group_id` varchar(128) NOT NULL COMMENT 'group_id',
  `tenant_id` varchar(128) DEFAULT '' COMMENT 'tenant_id',
  `nid` bigint(20) NOT NULL AUTO_INCREMENT,
  PRIMARY KEY (`nid`),
  UNIQUE KEY `uk_configtagrelation_configidtag` (`id`,`tag_name`,`tag_type`),
  KEY `idx_tenant_id` (`tenant_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='config_tag_relation';

/******************************************/
/*   Database full name = nacos_config   */
/*   table name = group_capacity   */
/******************************************/
CREATE TABLE `group_capacity` (
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'primary key ID',
  `group_id` varchar(128) NOT NULL DEFAULT '' COMMENT 'Group ID´╝îNull character means the entire cluster',
  `quota` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'Quota, 0 means use default',
  `usage` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'Usage amount',
  `max_size` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'The upper limit of the size of a single configuration, in bytes, 0 means use the default value',
  `max_aggr_count` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'The maximum number of aggregate sub-configurations, 0 means use the default value',
  `max_aggr_size` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'The upper limit of the sub-configuration size of a single aggregated data, in bytes, 0 means use the default value',
  `max_history_count` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'Maximum number of change history',
  `gmt_create` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'creation time',
  `gmt_modified` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Change the time',
  PRIMARY KEY (`id`),
  UNIQUE KEY `uk_group_id` (`group_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='cluster, each Group Capacity Information Sheet';

/******************************************/
/*   Database full name = nacos_config   */
/*   table name = his_config_info   */
/******************************************/
CREATE TABLE `his_config_info` (
  `id` bigint(64) unsigned NOT NULL,
  `nid` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `data_id` varchar(255) NOT NULL,
  `group_id` varchar(128) NOT NULL,
  `app_name` varchar(128) DEFAULT NULL COMMENT 'app_name',
  `content` longtext NOT NULL,
  `md5` varchar(32) DEFAULT NULL,
  `gmt_create` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `gmt_modified` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `src_user` text,
  `src_ip` varchar(20) DEFAULT NULL,
  `op_type` char(10) DEFAULT NULL,
  `tenant_id` varchar(128) DEFAULT '' COMMENT 'Tenant field',
  PRIMARY KEY (`nid`),
  KEY `idx_gmt_create` (`gmt_create`),
  KEY `idx_gmt_modified` (`gmt_modified`),
  KEY `idx_did` (`data_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='Multi-tenancy retrofit';


/******************************************/
/*   Database full name = nacos_config   */
/*   table name = tenant_capacity   */
/******************************************/
CREATE TABLE `tenant_capacity` (
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'primary key ID',
  `tenant_id` varchar(128) NOT NULL DEFAULT '' COMMENT 'Tenant ID',
  `quota` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'Quota, 0 means use default',
  `usage` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'Usage amount',
  `max_size` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'The upper limit of the size of a single configuration, in bytes, 0 means use the default value',
  `max_aggr_count` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'Maximum number of aggregate sub-configurations',
  `max_aggr_size` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'The upper limit of the sub-configuration size of a single aggregated data, in bytes, 0 means use the default value',
  `max_history_count` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'Maximum number of change history',
  `gmt_create` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'creation time',
  `gmt_modified` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Change the time',
  PRIMARY KEY (`id`),
  UNIQUE KEY `uk_tenant_id` (`tenant_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='Tenant Capacity Information Sheet';


CREATE TABLE `tenant_info` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'id',
  `kp` varchar(128) NOT NULL COMMENT 'kp',
  `tenant_id` varchar(128) default '' COMMENT 'tenant_id',
  `tenant_name` varchar(128) default '' COMMENT 'tenant_name',
  `tenant_desc` varchar(256) DEFAULT NULL COMMENT 'tenant_desc',
  `create_source` varchar(32) DEFAULT NULL COMMENT 'create_source',
  `gmt_create` bigint(20) NOT NULL COMMENT 'creation time',
  `gmt_modified` bigint(20) NOT NULL COMMENT 'Change the time',
  PRIMARY KEY (`id`),
  UNIQUE KEY `uk_tenant_info_kptenantid` (`kp`,`tenant_id`),
  KEY `idx_tenant_id` (`tenant_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='tenant_info';

CREATE TABLE `users` (
	`username` varchar(50) NOT NULL PRIMARY KEY,
	`password` varchar(500) NOT NULL,
	`enabled` boolean NOT NULL
);

CREATE TABLE `roles` (
	`username` varchar(50) NOT NULL,
	`role` varchar(50) NOT NULL,
	UNIQUE INDEX `idx_user_role` (`username` ASC, `role` ASC) USING BTREE
);

CREATE TABLE `permissions` (
    `role` varchar(50) NOT NULL,
    `resource` varchar(255) NOT NULL,
    `action` varchar(8) NOT NULL,
    UNIQUE INDEX `uk_role_permission` (`role`,`resource`,`action`) USING BTREE
);

INSERT INTO users (username, password, enabled) VALUES ('nacos', '$2a$10$EuWPZHzz32dJN7jexM34MOeYirDdFAZm2kuWj7VEOJhhZkDrxfvUu', TRUE);

INSERT INTO roles (username, role) VALUES ('nacos', 'ROLE_ADMIN');

Orchestrate the cluster-ip.yaml file

Create a new cluster-ip.yaml file in each of the three machines

# Create a new nacos directory Create nacos1, nacos2, and naocs3 respectively
mkdir /usr/local/docker/nacos1
# Enter the newly created naocs1 directory
cd /usr/local/docker/nacos1
# Create a new file cluster-ip.yaml
touch cluster-ip.yaml
# Create a new directory init.d
mkdir init.d
# Enter the init.d directory 
cd init.d
# New custom.properties file
touch custom.properties
# Enter the newly created naocs1 directory
cd /usr/local/docker/nacos1
# Edit cluster-ip.yaml
vi cluster-ip.yaml
version: "3"
services:
  nacos:
    container_name: nacos1
    image: nacos/nacos-server:1.4.0
    networks:
      - nacos_net
    environment:
      - MODE=cluster # cluster mode
      - NACOS_SERVERS=10.0.0.5:8848 10.0.0.6:8848 10.0.0.7:8848 # The server external network corresponding to the three machines: the nacos service port corresponding to the three machines
      - NACOS_SERVER_IP=10.0.0.6 # The current local server external network
      - SPRING_DATASOURCE_PLATFORM=mysql
      - MYSQL_SERVICE_HOST=10.0.0.3 # ip corresponding to mysql database
      - MYSQL_SERVICE_PORT=3306    # The port corresponding to the mysql database
      - MYSQL_SERVICE_DB_NAME=nacos_1.4.0 # database name
      - MYSQL_SERVICE_USER=root # database user
      - MYSQL_SERVICE_PASSWORD=123456 # database password
      - MYSQL_DATABASE_NUM=1 # 1 data source
      - JVM_XMS=256m
      - JVM_XMX=256m
      - JVM_XMN=256m
    volumes:
      - ./cluster-logs/nacos:/home/nacos/logs
      - ./init.d/custom.properties:/home/nacos/init.d/custom.properties
    ports:
      - "8848:8848"
      - "9555:9555"
    restart: on-failure
networks:
  nacos_net:
    driver: bridge

Note: Some properties are not the same as before 1.3.0. The property configuration of the corresponding version is posted below

Property configuration list

property name describe Options
MODE System startup mode: cluster/single machine cluster/standalone default cluster
NACOS_SERVERS cluster address p1:port1 space ip2:port2 space ip3:port3
PREFER_HOST_MODE Support IP or domain name mode hostname/ip default ip
NACOS_SERVER_PORT Nacos running port Default 8848
NACOS_SERVER_IP IP can be specified in multi-NIC mode
SPRING_DATASOURCE_PLATFORM Support MYSQL database in stand-alone mode mysql / empty default: empty
MYSQL_SERVICE_HOST database connection address
MYSQL_SERVICE_PORT database port Default : 3306
MYSQL_SERVICE_DB_NAME database name
MYSQL_SERVICE_USER database username
MYSQL_SERVICE_PASSWORD database user password
MYSQL_SERVICE_DB_PARAM Database connection parameters default : characterEncoding=utf8&connectTimeout=1000&socketTimeout=3000&autoReconnect=true
MYSQL_DATABASE_NUM It indicates the number of database Default: 1
JVM_XMS -Xms Default: 2g
JVM_XMX -Xmx Default: 2g
JVM_XMN -Xmn Default: 1g
JVM_MS -XX:MetaspaceSize Default: 128m
JVM_MMS -XX:MaxMetaspaceSize Default: 320m
NACOS_DEBUG Whether to enable remote DEBUG y/n default :n
TOMCAT_ACCESSLOG_ENABLED server.tomcat.accesslog.enabled Default : false
NACOS_AUTH_SYSTEM_TYPE Permission system type selection, currently only nacos type is supported Default :nacos
NACOS_AUTH_ENABLE Whether to enable the permission system Default : false
NACOS_AUTH_TOKEN_EXPIRE_SECONDS token expiration time Default: 18000
NACOS_AUTH_TOKEN token Default: SecretKey012345678901234567890123456789012345678901234567890123456789
NACOS_AUTH_CACHE_ENABLE Permission cache switch, the update of permission cache will have a default delay of 15 seconds after it is turned on Default : false
MEMBER_LIST Set the cluster address by means of environment variables Example: 192.168.16.101:8847?raft_port=8807,192.168.16.101?raft_port=8808,192.168.16.101:8849?raft_port=8809
EMBEDDED_STORAGE Whether to enable the cluster embedded storage mode embedded Default : none

Advanced configuration

If the above property list does not meet your needs, you can mount custom.properties to the /home/nacos/init.d/ directory, and then configure properties in it like using Spring Boot's application.properties file, and this file configures The properties take precedence over application.properties

start the container

Start nacos of three machines separately

# Enter the newly created naocs1 directory
cd /usr/local/docker/nacos1
# start command
docker-compose -f cluster-ip.yaml up -d

Wait for startup to complete. .

cluster mode

Three modes are officially recommended:

http://ip1 :port/openAPI is directly connected to ip mode. If the machine is hung up, you need to modify the ip before it can be used.

http://VIP :port/openAPI mount the VIP mode, you can directly connect to the vip, the real ip of the server is hung below, and the readability is not good.

http://nacos.com :port/openAPI domain name + VIP mode, good readability, and easy to change ip, recommended mode

Implementation architecture diagram:

Description: The VIP in the above picture is virtual IP, which means virtual IP

Recommended Keepalived+ dual-system hot backup

Configure a real cluster with Nginx

install nginx

Baidu on your own (I will provide installation documentation later)

nginx configuration file

    # Cluster point configuration Three machines ip:port configuration
    upstream cluster{
        server 10.0.0.5:8848 weight=1 max_fails=2 fail_timeout=30s;
		server 10.0.0.6:8848 weight=1 max_fails=2 fail_timeout=30s;
		server 10.0.0.7:8848 weight=1 max_fails=2 fail_timeout=30s;
    }
 
    server{
        listen 80;
        server_name localhost;
		
		large_client_header_buffers 4 16k;     # Maximum number and size of buffers to read large client request headers
        client_max_body_size 300m;     #Sets the maximum request body size that nginx can handle.
        client_body_buffer_size 128k;  #The buffer size of the request body. 
		
		fastcgi_connect_timeout 300;
		fastcgi_send_timeout 300;
        fastcgi_read_timeout 300;
		fastcgi_buffer_size 64k;
        fastcgi_buffers 8 32k;
        fastcgi_busy_buffers_size 128k;
		fastcgi_temp_file_write_size 64k;

		location / {
	    	proxy_pass http://cluster;
		}
    }

start nginx

ok

If you have any questions, please get in touch with me~ Remember to give a like


Jeesoul shares various language technologies with the purpose of open source. There is no ability to learn, only unwillingness to learn. Welcome to exchange and learn together!

Tags: Docker Spring Cloud Nacos docker compose

Posted by brokenme on Wed, 04 May 2022 04:30:47 +0300