Best practices for deploying various Beat s of the Elastic Stack under Tencent Cloud

Overview

Using the various Beats modules of the Elastic Stack can completely end the twisted practice of manually collecting logs and checking metrics on the server. With the Elasticsearch service provided by Tencent Cloud, the operation and maintenance of a large-scale cloud environment can be easily done. This article helps you sort out the necessary basic operations at one time to ensure that you can use the Elastic Stack to continuously monitor your production environment in a safe, stable, and scalable manner.

Create an ES cluster

Log in to the Tencent Cloud Service console, query and enter the Elasticsearch service, and click the New button to create an Elasticsearch cluster. As shown below.

 

Cluster configuration instructions:

  • Beijing District
  • 7.5.1 - Platinum Edition
  • Single Availability Zone
  • Hot and cold mode

 

Other parameters in this instance are kept by default and can be modified according to actual business requirements.

After clicking Next, set a super username and password for the Elasticsearch cluster.

The cluster was successfully created after a few minutes. Check out these basic configurations below.

This way we have a secure, scalable and performant ES backend service.

Create Beats write roles and users

Log in to Kibana, click Roles and User Management, and create a 'write-only' user for the Beast profile.

  • Create the beats-writer role
  • Create a beats-writer user, which only assigns the beats-writer role and customizes a secure and complex password.

The Beats-write role settings are shown below:

 

This user will be used in all subsequent Beats configuration files, and the risk of data leakage is greatly reduced with minimal privilege users.

Beats initial configuration

Log in to a prepared Linux server and do Beats-related initialization on this machine; that is, execute a series of setup commands; the functions of these commands are:

  • Loads the index template in the ES backend, along with the ILM policy for the index.
  • Loads Kibana related objects and visualization dashboards.

Note that this is a one-time job, and only needs to be successfully executed once on a virtual machine.

To log in to the prepared Linux server with SSH, you first need to install the rpm installation package of the relevant beats. The installation commands are ignored here, otherwise these commands cannot be executed. After installing the rpm packages of filebeat and metricbeat, execute the following reference commands.


filebeat setup -e \
  -E output.logstash.enabled=false \
  -E output.elasticsearch.hosts=['192.168.0.43:9200'] \
  -E output.elasticsearch.username=elastic \
  -E output.elasticsearch.password=YourPassWord \
  -E setup.kibana.host=es-ot7wei87.internal.kibana.tencentelasticsearch.com:5601

metricbeat setup -e   \
  -E output.elasticsearch.hosts=['192.168.0.43:9200']   \
  -E output.elasticsearch.username=elastic   \
  -E output.elasticsearch.password=YourPassWord  \
  -E setup.kibana.host=es-ot7wei87.internal.kibana.tencentelasticsearch.com:5601

When running the above command, Beats is installed by default. These command line parameters are necessary to check. With these parameters, beats will ignore the default configuration file.

The above commands are based on your needs. If you need to use other Beats, please use the relevant setup commands. For the es and kibana-related information requirements, please refer to the es cluster information created in the previous step.

After all the above commands are successful, log in to the Kibana interface, click the Dashboard menu, there should be a lot of dashboards loaded here. So far the Elastic Stack background has been initialized successfully.

Officially deploy Beats on nodes

Reference and modify the installation script, one-click installation and configuration of Beats

git clone https://github.com/martinliu/elastic-stack-lab.git
cd tencent
sh add-agent.sh

After successfully executing the above script, the related beats service should be running normally. After executing this command, use sudo systemctl status filebeat on the Linux server to check whether the service is running normally; use this command to see that the filebeat service is running normally.

Highlights from the config file used by this script:

  • Removed all configurations not related to data ingestion (such as es and kibana configuration and initialization, etc.)
  • Added set of necessary best practice parameters for minimization
  • It is recommended to add beats related modules as needed
  • Add the necessary Beat configuration parameters as required

Instance configuration files eg.

filebeat.yml

#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: log
  enabled: false
  paths:
    - /var/log/*.log

#============================= Filebeat modules ===============================
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 60s

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  hosts: ["${INT_ES_SRV}"]
  password: ${BEATS_WRITER_PW}
  username: ${BEATS_WRITER_USERNAME}

#================================ Processors =====================================
processors:
  - add_host_metadata: 
      netinfo.enabled: true
      cache.ttl: 5m
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  - add_fields:
      target: ''
      fields:
        service.name: 'Joint Lab'
        service.id: 'es-qq'

#==================== Best Practice Configuration ==========================
setup.ilm.check_exists: false
logging.level: error
queue.spool: ~
monitoring:
  enabled: true

metricbeat.yml

# =========================== Modules configuration ============================
metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  hosts: ["${INT_ES_SRV}"]
  password: ${BEATS_WRITER_PW}
  username: ${BEATS_WRITER_USERNAME}

#================================ Processors =====================================
processors:
  - add_host_metadata: 
      netinfo.enabled: true
      cache.ttl: 5m
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  - add_fields:
      target: ''
      fields:
        service.name: 'Joint Lab'
        service.id: 'es-qq'

#==================== Best Practice Configuration ==========================
setup.ilm.check_exists: false
logging.level: error
queue.spool: ~
monitoring:
  enabled: true

The above configuration files use these general best practice configuration parameters.

  • Use ECS extension fields to enrich contextual meaning
  • Enable beats endpoint monitoring
  • Hide all sensitive information with keystore
#Disable index ilm policy checks to avoid useless actions
setup.ilm.check_exists: false

#Turn Beats' own logging to the lowest level
logging.level: error

#Enable local default endpoint caching behavior
queue.spool: ~

#Enable monitoring of endpoints
monitoring:
  enabled: true

Troubleshooting

filebeat setup unsuccessful

The first time any beats do the setup command, it will probably end successfully at the minute level. If there is a failure or a freeze, you can wait and see for a longer time. If it is unsuccessful, you need to execute it repeatedly to check whether the es and kibana services can work normally. Knowing Chen Gong can proceed to the next step of installation work.

The service cannot be started due to a configuration file error

Since the above customized configuration file may also have errors, especially when deploying this configuration file for the first time, you can comment out the line of log level error and the two lines that start the service.

Then execute filebeat -e on the command line to view the startup process of the entire feilebeat. This command will read the customized configuration file, then start to connect to the background es service, and then enter the state of normal data transmission. If there is any configuration error during this process, you can also see the relevant information intuitively until it is adjusted to a normal state.

After the above process is adjusted, it must be managed through the git version, and then you can safely perform one-click deployment of beast on other nodes.

Summarize

The above are the basic best practices for deploying Beats, that is to say, the combination of ES backend and beats in the production environment, and the content involved in this article are all baseline configurations. It is recommended to do more tuning according to your own needs. Here we use shell scripts to deploy beats and related configurations. Shell scripts are suitable for demonstrating the principle. It is recommended to replace them with automated operation and maintenance tools you are familiar with, such as ansible and other tools. This ensures larger-scale automated deployment and maintenance.

The related configuration files and scripts are located at: https://github.com/martinliu/elastic-stack-lab.git

Tags: Operation & Maintenance ElasticSearch DevOps

Posted by brauchi on Wed, 25 May 2022 03:37:30 +0300