Sampler is a tool for shell command execution, visualization and alerting. Its configuration uses a simple YAML file.
1. Why do I need it?
You can sample any dynamic process directly from the terminal - watch for changes in the database, monitor MQ in-flight messages, trigger deployment scripts and get notified when it's done.
If there is a way to get metric s using shell commands, then it can be visualized instantly using Sampler.
2. Installation
macOS
brew cask install sampler
or
sudo curl -Lo /usr/local/bin/sampler https://github.com/sqshq/sampler/releases/download/v1.0.3/sampler-1.0.3-darwin-amd64 sudo chmod +x /usr/local/bin/sampler
Linux
sudo wget https://github.com/sqshq/sampler/releases/download/v1.0.3/sampler-1.0.3-linux-amd64 -O /usr/local/bin/sampler sudo chmod +x /usr/local/bin/sampler
Note: The libasound2-dev system library needs to be installed for Sampler to play trigger sounds. Usually the library is already installed in the corresponding location, but if not - you can install it using your usual package manager, e.g. apt install libasound2-dev
Windows (experimental)
Recommended for advanced console emulators such as Cmder
Download .exe
3. Use
Specify shell commands, and Sampler will execute these commands at the corresponding rate. The output is for visualization.
Using Sampler's basic three-step process:
Define shell commands in YAML configuration files
run sampler -c config.yml
Resize and position components on UI
There are already many monitoring systems on the market
Sampler is by no means a replacement for a monitoring system, but an easy-to-setup development tool.
If spinning up and configuring Prometheus with Grafana is a completely redundant task, then Sampler might be the right solution. No server, no database, no deployment required - you specify the shell command and it works.
Do I need to install it on every server I monitor?
No, you can run Sampler locally and still collect telemetry from multiple remote machines. Any visualization may have an init command where you can ssh to a remote server. See the SSH example
4. Components
Below is a list of configuration examples for each component type, with macOS-compatible sample scripts.
Runchart
runcharts:
- title: Search engine response time
rate-ms: 500 # sampling rate, default = 1000
scale: 2 # number of digits after sample decimal point, default = 1
legend:
enabled: true # enables item labels, default = true
details: false # enables item statistics: cur/min/max/dlt values, default = true
items:
- label: GOOGLE
sample: curl -o /dev/null -s -w '%{time_total}' https://www.google.com
color: 178 # 8-bit color number, default one is chosen from a pre-defined palette
- label: YAHOO
sample: curl -o /dev/null -s -w '%{time_total}' https://search.yahoo.com
- label: BING
sample: curl -o /dev/null -s -w '%{time_total}' https://www.bing.comSparkline
sparklines:
- title: CPU usage
rate-ms: 200
scale: 0
sample: ps -A -o %cpu | awk '{s+=$1} END {print s}'
- title: Free memory pages
rate-ms: 200
scale: 0
sample: memory_pressure | grep 'Pages free' | awk '{print $3}'Barchart
barcharts:
- title: Local network activity
rate-ms: 500 # sampling rate, default = 1000
scale: 0 # number of digits after sample decimal point, default = 1
items:
- label: UDP bytes in
sample: nettop -J bytes_in -l 1 -m udp | awk '{sum += $4} END {print sum}'
- label: UDP bytes out
sample: nettop -J bytes_out -l 1 -m udp | awk '{sum += $4} END {print sum}'
- label: TCP bytes in
sample: nettop -J bytes_in -l 1 -m tcp | awk '{sum += $4} END {print sum}'
- label: TCP bytes out
sample: nettop -J bytes_out -l 1 -m tcp | awk '{sum += $4} END {print sum}'Gauge
gauges: - title: Minute progress rate-ms: 500 # sampling rate, default = 1000 scale: 2 # number of digits after sample decimal point, default = 1 percent-only: false # toggle display of the current value, default = false color: 178 # 8-bit color number, default one is chosen from a pre-defined palette cur: sample: date +%S # sample script for current value max: sample: echo 60 # sample script for max value min: sample: echo 0 # sample script for min value - title: Year progress cur: sample: date +%j max: sample: echo 365 min: sample: echo 0
Textbox
textboxes:
- title: Local weather
rate-ms: 10000 # sampling rate, default = 1000
sample: curl wttr.in?0ATQF
border: false # border around the item, default = true
color: 178 # 8-bit color number, default is white
- title: Docker containers stats
rate-ms: 500
sample: docker stats --no-stream --format "table {{.Name}}t{{.CPUPerc}}t{{.MemUsage}}t{{.PIDs}}"Asciibox
asciiboxes: - title: UTC time rate-ms: 500 # sampling rate, default = 1000 font: 3d # font type, default = 2d border: false # border around the item, default = true color: 43 # 8-bit color number, default is white sample: env TZ=UTC date +%r
5. Additional features
Triggers
Triggers allow to perform conditional actions such as visual/audio alerts or arbitrary shell commands. The following example illustrates this concept.
Clock gauge, showing time progress and current time every minute from start
gauges: - title: MINUTE PROGRESS position: [[0, 18], [80, 0]] cur: sample: date +%S max: sample: echo 60 min: sample: echo 0 triggers: - title: CLOCK BELL EVERY MINUTE condition: '[ $label == "cur" ] && [ $cur -eq 0 ] && echo 1 || echo 0' # expects "1" as TRUE indicator actions: terminal-bell: true # standard terminal bell, default = false sound: true # NASA quindar tone, default = false visual: false # notification with current value on top of the component area, default = false script: say -v samantha `date +%I:%M%p` # an arbitrary script, which can use $cur, $prev and $label variables
Search engine latency graphs, alerting users when latency exceeds a threshold
runcharts:
- title: SEARCH ENGINE RESPONSE TIME (sec)
rate-ms: 200
items:
- label: GOOGLE
sample: curl -o /dev/null -s -w '%{time_total}' https://www.google.com
- label: YAHOO
sample: curl -o /dev/null -s -w '%{time_total}' https://search.yahoo.com
triggers:
- title: Latency threshold exceeded
condition: echo "$prev < 0.3 && $cur > 0.3" |bc -l # expects "1" as TRUE indicator
actions:
terminal-bell: true # standard terminal bell, default = false
sound: true # NASA quindar tone, default = false
visual: true # visual notification on top of the component area, default = false
script: 'say alert: ${label} latency exceeded ${cur} second' # an arbitrary script, which can use $cur, $prev and $label variablesInteractive shell support
In addition to the sample command, the init command (executed only once before sampling) and the transform command (post-processing the sample command output) can also be specified. This includes interactive shell use cases such as establishing a connection to a database only once, and then performing polling in an interactive shell session.
Basic mode
textboxes: - title: MongoDB polling rate-ms: 500 init: mongo --quiet --host=localhost test # executes only once to start the interactive session sample: Date.now(); # executes with a required rate, in scope of the interactive session transform: echo result = $sample # executes in scope of local session, $sample variable is available for transformation
PTY mode
In some cases an interactive shell won't work because its stdin is not a terminal. In this case we can use PTY mode:
textboxes: - title: Neo4j polling pty: true # enables pseudo-terminal mode, default = false init: cypher-shell -u neo4j -p pwd --format plain sample: RETURN rand(); transform: echo "$sample" | tail -n 1 - title: Top on a remote server pty: true # enables pseudo-terminal mode, default = false init: ssh -i ~/user.pem ec2-user@1.2.3.4 sample: top
init command step by step
Multiple init commands can also be executed one after the other before starting sampling.
textboxes: - title: Java application uptime multistep-init: - java -jar jmxterm-1.0.0-uber.jar - open host:port # or local PID - bean java.lang:type=Runtime sample: get Uptime
variable
If the configuration file contains repeated patterns, they can be extracted into the variables section. Additionally, variables can be specified at startup with the -v/–variable flag, and arbitrary system environment variables can also be used in scripts.
variables:
mongoconnection: mongo --quiet --host=localhost test
barcharts:
- title: MongoDB documents by status
items:
- label: IN_PROGRESS
init: $mongoconnection
sample: db.getCollection('events').find({status:'IN_PROGRESS'}).count()
- label: SUCCESS
init: $mongoconnection
sample: db.getCollection('events').find({status:'SUCCESS'}).count()
- label: FAIL
init: $mongoconnection
sample: db.getCollection('events').find({status:'FAIL'}).count()color themes
theme: light # default = dark
sparklines:
- title: CPU usage
sample: ps -A -o %cpu | awk '{s+=$1} END {print s}'6. Real scene
database
Below are examples of different database connections. It is recommended to use an interactive shell (init script) to establish the connection only once and reuse it during sampling.
MySQL
# prerequisite: installed mysql shell variables: mysql_connection: mysql -u root -s --database mysql --skip-column-names sparklines: - title: MySQL (random number example) pty: true init: $mysql_connection sample: select rand();
PostgreSQL
# prerequisite: installed psql shell variables: PGPASSWORD: pwd postgres_connection: psql -h localhost -U postgres --no-align --tuples-only sparklines: - title: PostgreSQL (random number example) init: $postgres_connection sample: select random();
MongoDB
# prerequisite: installed mongo shell variables: mongo_connection: mongo --quiet --host=localhost test sparklines: - title: MongoDB (random number example) init: $mongo_connection sample: Math.random();
Neo4j
# prerequisite: installed cypher shell variables: neo4j_connection: cypher-shell -u neo4j -p pwd --format plain sparklines: - title: Neo4j (random number example) pty: true init: $neo4j_connection sample: RETURN rand(); transform: echo "$sample" | tail -n 1
Kafka
Check the kafka lag value, calculate the sum of the lag value of each queue, alarm above the threshold, multiple consumer groups, and multiple topic s.
variables:
kafka_connection: $KAFKA_HOME/bin/kafka-consumer-groups --bootstrap-server localhost:9092
runcharts:
- title: Kafka lag per consumer group
rate-ms: 5000
scale: 0
items:
- label: A->B
sample: $kafka_connection --group group_a --describe | awk 'NR>1 {sum += $5} END {print sum}'
- label: B->C
sample: $kafka_connection --group group_b --describe | awk 'NR>1 {sum += $5} END {print sum}'
- label: C->D
sample: $kafka_connection --group group_c --describe | awk 'NR>1 {sum += $5} END {print sum}'Docker
Docker container statistics (CPU, MEM, O/I)
textboxes:
- title: Docker containers stats
sample: docker stats --no-stream --format "table {{.Name}}t{{.CPUPerc}}t{{.MemPerc}}t{{.MemUsage}}t{{.NetIO}}t{{.BlockIO}}t{{.PIDs}}"SSH
TOP command on remote server
variables: sshconnection: ssh -i ~/my-key-pair.pem ec2-user@1.2.3.4 textboxes: - title: SSH pty: true init: $sshconnection sample: top
JMX
Normal running example of a Java application
# prerequisite: download [jmxterm jar file](https://docs.cyclopsgroup.org/jmxterm)
textboxes:
- title: Java application uptime
multistep-init:
- java -jar jmxterm-1.0.0-uber.jar
- open host:port # or local PID
- bean java.lang:type=Runtime
sample: get Uptime
transform: echo $sample | tr -dc '0-9' | awk '{printf "%.1f min", $1/1000/60}'From: FreeBuf.COM, Author: secist
Link: https://www.freebuf.com/secto...
