SkyWalking is an application performance monitoring system, especially designed for micro services, cloud native and container based (Docker, Kubernetes, Mesos) architecture. In addition to application index monitoring, it can also track distributed call links. Components with similar functions include Zipkin, Pinpoint, CAT, etc.

Take a few pictures to see the effect, and then build and use them step by step

1. Concept and structure

SkyWalking is an open source monitoring platform for collecting, analyzing, aggregating and visualizing data from services and cloud native infrastructure. SkyWalking provides an easy way to maintain a clear view of distributed systems, even across the cloud. It is a modern APM designed specifically for cloud native, container based distributed systems.  

SkyWalking monitors applications from three dimensions: service, service instance, and endpoint

Not much about services and instances. An endpoint is a path or URI in a service

SkyWalking allows users to understand the topology relationship between Services and Endpoints, to view the metrics of every Service/Service Instance/Endpoint and to set alarm rules.

SkyWalking allows users to understand the topological relationship between services and endpoints, view the metrics of each service / service instance / endpoint, and set alarm rules.

1.1. Structure

SkyWalking is logically divided into four parts: Probes, Platform backend, Storage and UI

This structure is very clear. The probe is the Agent, which is responsible for collecting data and reporting it to the server. The server processes and stores the data, and the UI is responsible for displaying it

2. Download and install

SkyWalking has two versions, ES version and non es version. If we decide to use ElasticSearch as storage, download the ES version.  

https://skywalking.apache.org/downloads/

https://archive.apache.org/dist/skywalking/

The agent directory will be copied to the machine where each service is located as a probe in the future

The bin directory is the service startup script

The config directory is a configuration file

The oap LIBS directory is the jar package required for the operation of the oap service

The webapp directory is the jar package required for the operation of web services

Next, you need to select storage. The supported storage includes:

• H2
• ElasticSearch 6, 7
• MySQL
• TiDB
• InfluxDB

As a monitoring system, H2 and MySQL are excluded first. Here we recommend InfluxDB, which is a time series database and is very suitable for this scenario

But I'm not very familiar with InfluxDB, so I'll use ElasticSearch7 first

https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-storage.md

2.1. Install ElasticSearch

https://www.elastic.co/guide/en/elasticsearch/reference/7.10/targz.html 

# start-up
./bin/elasticsearch -d -p pid
# stop it
pkill -F pid

ElasticSearch7.x requires Java 11 or above, but if you set the environment variable JAVA_HOME, it will use your own java version

Generally, the following three errors will be reported during startup:

[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[3]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured

resolvent:

In / etc / security / limits Add the following contents to the conf file:

* soft nofile 65536
* hard nofile 65536
* soft nproc  4096
* hard nproc  4096

You can view the modification results through the following four commands:

ulimit -Hn
ulimit -Sn
ulimit -Hu
ulimit -Su

Modify / etc / sysctl Conf file, add the following contents:

vm.max_map_count=262144

Modify es configuration file elasticsearch YML uncomment and keep one node

cluster.initial_master_nodes: ["node-1"]

In order to access ip:port mode, the network configuration needs to be modified

network.host: 0.0.0.0

After modification, it is as follows:

So far, ElasticSearch has been started successfully

Next, in config / application Configure the es address in YML

storage:
  selector: ${SW_STORAGE:elasticsearch7}
  elasticsearch7:
    clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:192.168.100.19:9200}

2.2. Install Agent

https://github.com/apache/skywalking/blob/v8.2.0/docs/en/setup/service-agent/java-agent/README.md

Copy the agent directory to the machine where each service is located

scp -r ./agent chengjs@192.168.100.12:~/

Here, I copy it to each service directory

Plugins are various plug-ins used by probes. SkyWalking plug-ins are plug and play. You can put the plug-ins in optional plugins into plugins

Modify agent / config / agent Config configuration file, which can also be specified through command line parameters

It mainly configures the service name and back-end service address

agent.service_name=${SW_AGENT_NAME:user-center}
collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:192.168.100.17:11800}

Of course, it can also be set through environment variables or system attributes, for example:

export SW_AGENT_COLLECTOR_BACKEND_SERVICES=127.0.0.1:11800

Finally, when the service starts, use the command line parameter - javaagent to specify the probe

java -javaagent:/path/to/skywalking-agent/skywalking-agent.jar -jar yourApp.jar

For example:

java -javaagent:./agent/skywalking-agent.jar -Dspring.profiles.active=dev -Xms512m -Xmx1024m -jar demo-0.0.1-SNAPSHOT.jar

3. Start service

Modify webapp / webapp YML file, change the port number and back-end service address

server:
  port: 8080

collector:
  path: /graphql
  ribbon:
    ReadTimeout: 10000
    # Point to all backend's restHost:restPort, split by ,
    listOfServers: 127.0.0.1:12800

Start service

bin/startup.sh

Or start them in turn

bin/oapService.sh
bin/webappService.sh

Check the log file in the logs directory to see if it starts successfully

Browser access http://127.0.0.1:8080

4. Alarm

Edit alarm settings YML set alarm rules and notifications

https://github.com/apache/skywalking/blob/v8.2.0/docs/en/setup/backend/backend-alarm.md

Focus on the alarm notification

In order to use the nailing robot notification, next, create a new project

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.4.0</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.wt.monitor</groupId>
    <artifactId>skywalking-alarm</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <name>skywalking-alarm</name>

    <properties>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>com.aliyun</groupId>
            <artifactId>alibaba-dingtalk-service-sdk</artifactId>
            <version>1.0.1</version>
        </dependency>

        <dependency>
            <groupId>commons-codec</groupId>
            <artifactId>commons-codec</artifactId>
            <version>1.15</version>
        </dependency>

        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.75</version>
        </dependency>

        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

Optional dependency (not recommended)

<dependency
    <groupId>org.apache.skywalking</groupId>
    <artifactId>server-core</artifactId>
    <version>8.2.0</version>
</dependency>

Define alarm message entity class

package com.wt.monitor.skywalking.alarm.domain;

import lombok.Data;

import java.io.Serializable;

/**
 * @author ChengJianSheng
 * @date 2020/12/1
 */
@Data
public class AlarmMessageDTO implements Serializable {

    private int scopeId;

    private String scope;

    /**
     * Target scope entity name
     */
    private String name;

    private String id0;

    private String id1;

    private String ruleName;

    /**
     * Alarm text message
     */
    private String alarmMessage;

    /**
     * Alarm time measured in milliseconds
     */
    private long startTime;

}

Send nail robot message

package com.wt.monitor.skywalking.alarm.service;

import com.dingtalk.api.DefaultDingTalkClient;
import com.dingtalk.api.DingTalkClient;
import com.dingtalk.api.request.OapiRobotSendRequest;
import com.taobao.api.ApiException;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.codec.binary.Base64;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;

/**
 * https://ding-doc.dingtalk.com/doc#/serverapi2/qf2nxq
 * @author ChengJianSheng
 * @data 2020/12/1
 */
@Slf4j
@Service
public class DingTalkAlarmService {

    @Value("${dingtalk.webhook}")
    private String webhook;
    @Value("${dingtalk.secret}")
    private String secret;

    public void sendMessage(String content) {
        try {
            Long timestamp = System.currentTimeMillis();
            String stringToSign = timestamp + "\n" + secret;
            Mac mac = Mac.getInstance("HmacSHA256");
            mac.init(new SecretKeySpec(secret.getBytes("UTF-8"), "HmacSHA256"));
            byte[] signData = mac.doFinal(stringToSign.getBytes("UTF-8"));
            String sign = URLEncoder.encode(new String(Base64.encodeBase64(signData)),"UTF-8");

            String serverUrl = webhook + "&timestamp=" + timestamp + "&sign=" + sign;
            DingTalkClient client = new DefaultDingTalkClient(serverUrl);
            OapiRobotSendRequest request = new OapiRobotSendRequest();
            request.setMsgtype("text");
            OapiRobotSendRequest.Text text = new OapiRobotSendRequest.Text();
            text.setContent(content);
            request.setText(text);

            client.execute(request);
        } catch (ApiException e) {
            e.printStackTrace();
            log.error(e.getMessage(), e);
        } catch (NoSuchAlgorithmException e) {
            e.printStackTrace();
            log.error(e.getMessage(), e);
        } catch (UnsupportedEncodingException e) {
            e.printStackTrace();
            log.error(e.getMessage(), e);
        } catch (InvalidKeyException e) {
            e.printStackTrace();
            log.error(e.getMessage(), e);
        }
    }
}

AlarmController.java

package com.wt.monitor.skywalking.alarm.controller;

import com.alibaba.fastjson.JSON;
import com.wt.monitor.skywalking.alarm.domain.AlarmMessageDTO;
import com.wt.monitor.skywalking.alarm.service.DingTalkAlarmService;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.text.MessageFormat;
import java.util.List;

/**
 * @author ChengJianSheng
 * @date 2020/12/1
 */
@Slf4j
@RestController
@RequestMapping("/skywalking")
public class AlarmController {

    @Autowired
    private DingTalkAlarmService dingTalkAlarmService;

    @PostMapping("/alarm")
    public void alarm(@RequestBody List<AlarmMessageDTO> alarmMessageDTOList) {
       log.info("Alarm information received: {}", JSON.toJSONString(alarmMessageDTOList));
       if (null != alarmMessageDTOList) {
           alarmMessageDTOList.forEach(e->dingTalkAlarmService.sendMessage(MessageFormat.format("-----come from SkyWalking Alarm of-----\n[[name]: {0}\n[[message]: {1}\n", e.getName(), e.getAlarmMessage())));
       }
    }
}

5. Documentation

https://skywalking.apache.org/

https://skywalking.apache.org/zh/ 

https://github.com/apache/skywalking/tree/v8.2.0/docs

https://archive.apache.org/dist/

https://www.elastic.co/guide/en/elasticsearch/reference/master/index.html