mongodb4.0 for docker 0 synchronize data to ES7 Simple configuration of 3.2 (mongo has no security verification)

Currently, mongo connector, reviewer and logstash are the most available information about mongo synchronization data to es on the Internet. They are all outdated and do not support higher versions of ES. At present, only monstache support can be found. Version reference official website:

 https://rwynn.github.io/monstache-site/start/

  es7.3.2 construction. Read my last article, which has detailed elk7 3.2 construction of. This article directly uses the ES built last time.

First, set up mongodb. I use version 4.0. First, you need to configure to open the replica set and modify the configuration file / etc / mongod conf.orig:

storage:
  dbPath: /data/db
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 27017
#Turn on remote access
  bindIp: 0.0.0.0


# how the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo

#security:

#operationProfiling:
#Replica set configuration content.
replication:
  replSetName: es
  oplogSizeMB: 10240

 

At this time, the docker compose of mongo is:

version: '3.7'

services:
  mongo:
    image: mongo:4.0
    restart: always
    container_name: mongodb
    environment:
      TZ: Asia/Shanghai       
    volumes:
      - /data/javaProject/mongo-data:/data/db
      - /data/javaProject/mongo-config/mongod.conf.orig:/etc/mongod.conf.orig
    command: --config /etc/mongod.conf.orig
    ports:
      - 12000:27017
  mongo-express:
    image: mongo-express
    container_name: mongo-express
    restart: always
    depends_on:
      - mongo
    ports:
      - 8081:8081
      TZ: Asia/Shanghai

 

At this point, start the container. The log in the mongo container reads:

No logs available

 

This is normal. After docker exec enters the mongo container, execute the command. The mongo command enters the client and executes the command:

> show dbs
2020-09-25T15:32:36.098+0800 E QUERY    [js] Error: listDatabases failed:{
    "operationTime" : Timestamp(0, 0),
    "ok" : 0,
    "errmsg" : "not master and slaveOk=false",
    "code" : 13435,
    "codeName" : "NotMasterNoSlaveOk",
    "$clusterTime" : {
        "clusterTime" : Timestamp(0, 0),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
} :
_getErrorWithCode@src/mongo/shell/utils.js:25:13
Mongo.prototype.getDBs@src/mongo/shell/mongo.js:139:1
shellHelper.show@src/mongo/shell/utils.js:882:13
shellHelper@src/mongo/shell/utils.js:766:15
@(shellhelp2):1:1

According to the data, this error is due to the failure to initialize the replica set:

rs.initiate({_id:"es",members:[{_id:0,host:'127.0.0.1:27017'}]})
among es To correspond to in the configuration file replSetName Value, and this command needs to be knocked by hand. If it is copied and executed, an analysis error will be reported
> use admin
switched to db admin
> rs.initiate({_id:"es",members:[{_id:0,host:'127.0.0.1:27017'}]})
2020-09-25T15:50:35.176+0800 E QUERY    [js] SyntaxError: illegal character @(shell):1:17
Hand tapping execution:
> rs.initiate({_id:"es",members:[{_id:0,host:'127.0.0.1:27017'}]})
{
    "ok" : 1,
    "operationTime" : Timestamp(1601020319, 1),
    "$clusterTime" : {
        "clusterTime" : Timestamp(1601020319, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
}

At this point, create a database, create a collection, and create a document. Then find the data in the Mongo express visualization page:

In its own local Database oplog.rs You can find that the data you just added has entered the replica set

 

After pulling monstache, mount the configuration file monstache config. Toml into the container. The contents of the configuration file are as follows:

 

# connectionn settings

# connect to MongoDB using the following URL
mongo-url = "mongodb://192.168.10.20:12000"
# connect to the Elasticsearch REST API at the following node URLs
elasticsearch-urls = ["http://192.168.10.20:9200"]

# frequently required settings

# if you need to seed an index from a collection and not just listen and sync changes events
# you can copy entire collections or views from MongoDB to Elasticsearch
direct-read-namespaces = ["testdemo.*"]

# if you want to use MongoDB change streams instead of legacy oplog tailing use change-stream-namespaces
# change streams require at least MongoDB API 3.6+
# if you have MongoDB 4+ you can listen for changes to an entire database or entire deployment
# in this case you usually don't need regexes in your config to filter collections unless you target the deployment.
# to listen to an entire db use only the database name.  For a deployment use an empty string.
#change-stream-namespaces = [""]

# additional settings

# if you don't want to listen for changes to all collections in MongoDB but only a few
# e.g. only listen for inserts, updates, deletes, and drops from mydb.mycollection
# this setting does not initiate a copy, it is only a filter on the change event listener
#namespace-regex = ''
# compress requests to Elasticsearch
#gzip = true
# generate indexing statistics
#stats = true
# index statistics into Elasticsearch
#index-stats = true
# use the following PEM file for connections to MongoDB
#mongo-pem-file = ""
# disable PEM validation
#mongo-validate-pem-file = true
# use the following user name for Elasticsearch basic auth
elasticsearch-user = "elastic"
# use the following password for Elasticsearch basic auth
elasticsearch-password = "pwd"
# use 4 go routines concurrently pushing documents to Elasticsearch
elasticsearch-max-conns = 4 
# use the following PEM file to connections to Elasticsearch
#elasticsearch-pem-file = ""
# validate connections to Elasticsearch
#elastic-validate-pem-file = false
# propogate dropped collections in MongoDB as index deletes in Elasticsearch
dropped-collections = true
# propogate dropped databases in MongoDB as index deletes in Elasticsearch
dropped-databases = true
# do not start processing at the beginning of the MongoDB oplog
# if you set the replay to true you may see version conflict messages
# in the log if you had synced previously. This just means that you are replaying old docs which are already
# in Elasticsearch with a newer version. Elasticsearch is preventing the old docs from overwriting new ones.
#replay = false
# resume processing from a timestamp saved in a previous run
resume = true
# do not validate that progress timestamps have been saved
#resume-write-unsafe = false
# override the name under which resume state is saved
#resume-name = "default"
# use a custom resume strategy (tokens) instead of the default strategy (timestamps)
# tokens work with MongoDB API 3.6+ while timestamps work only with MongoDB API 4.0+
resume-strategy = 0
# exclude documents whose namespace matches the following pattern
#namespace-exclude-regex = '^mydb\.ignorecollection$'
# turn on indexing of GridFS file content
#index-files = true
# turn on search result highlighting of GridFS content
#file-highlighting = true
# index GridFS files inserted into the following collections
#file-namespaces = ["users.fs.files"]
# print detailed information including request traces
verbose = true
# enable clustering mode
cluster-name = 'docker-cluster'
# do not exit after full-sync, rather continue tailing the oplog
#exit-after-direct-reads = false

 

The configuration file mounted by monstache config. Toml needs to check the official website or Alibaba cloud case. Alibaba cloud has a detailed configuration file and the meaning of each key. Refer to: https://help.aliyun.com/knowledge_detail/171650.html

At this point, the complete docker compose The YML file is:

version: '3.7'

services:

  mongo:
    image: mongo:4.0
    restart: always
    container_name: mongodb
    environment:
      TZ: Asia/Shanghai       
    volumes:
      - /data/javaProject/mongo-data:/data/db
      - /data/javaProject/mongo-config/mongod.conf.orig:/etc/mongod.conf.orig
    command: --config /etc/mongod.conf.orig
    ports:
      - 12000:27017
  mongo-express:
    image: mongo-express
    container_name: mongo-express
    restart: always
    depends_on:
      - mongo
    ports:
      - 8081:8081
    environment:
      TZ: Asia/Shanghai
  monstache: 
    image: rwynn/monstache:rel6
    restart: always
    container_name: monstache
    volumes: 
      - /data/javaProject/monstache-conf/monstache.config.toml:/app/monstache.config.toml
    command: -f /app/monstache.config.toml

 

At this time, the build starts, and the log error of monstache shows that the connection is not available:

current topology: { Type: ReplicaSetNoPrimary,

It is found that the mongo replica set needs to be initialized after it is opened, but I have already initialized it. After the mongo container is deleted, all are rebuilt. After the container is started, enter the mongo container to reinitialize the replica set. At this time, check the startup log of monstache and the connection has been successful.

Initialization reference: https://blog.csdn.net/jack_brandy/article/details/88887795 . After the start of this build, the monstache connection is successful:

ERROR 2020/09/28 03:44:32 Unable to connect to MongoDB using URL mongodb://192.168.10.20:12000: server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: 192.168.10.20:12000, Type: RSGhost, State: Connected, Average RTT: 631124 }, ] }


INFO 2020/09/28 03:44:47 Started monstache version 6.7.0


INFO 2020/09/28 03:44:47 Go version go1.14


INFO 2020/09/28 03:44:47 MongoDB go driver v1.3.5


INFO 2020/09/28 03:44:47 Elasticsearch go driver 7.0.18


INFO 2020/09/28 03:44:47 Successfully connected to MongoDB version 4.0.20


INFO 2020/09/28 03:44:47 Successfully connected to Elasticsearch version 7.3.2


INFO 2020/09/28 03:44:47 Sending systemd READY=1


WARN 2020/09/28 03:44:47 Systemd notification not supported (i.e. NOTIFY_SOCKET is unset)


INFO 2020/09/28 03:44:47 Joined cluster docker-cluster


INFO 2020/09/28 03:44:47 Starting work for cluster docker-cluster


INFO 2020/09/28 03:44:47 Watching changes on the deployment


INFO 2020/09/28 03:44:47 Listening for events


INFO 2020/09/28 03:44:47 Resuming from timestamp {T:1601264687 I:5}


INFO 2020/09/28 03:44:47 Direct reads completed

At this time, add a new database, collection and document in mongo. Log in kibana to view the new data. Synchronization complete.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

  

 

Tags: Java Docker MongoDB

Posted by pt4siek on Sat, 14 May 2022 15:00:34 +0300