3

prometheus发送告警信息

 7 months ago
source link: https://blog.51cto.com/yxh1990/9580784
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

配置邮件示例

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
           - 192.168.19.55:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
   - "rules/node_rules.yml"
  # - "second_rules.yml"

prometheus.yml

groups:
- name: node-rules
  rules:
  - alert: node-up
    expr: up == 0
    for: 15s
    labels:
      severity: 1
      team: node
    annotations:
      summary: "{{$labels.instance}}Instance has been down for more than 5 minutes"

node_rules.yml

global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.163.com:25'
  smtp_from: '[email protected]'
  smtp_auth_username: '[email protected]'
  smtp_auth_password: '1111111'
  smtp_require_tls: false
templates:
  - '/root/prom/alertmanager-0.26.0.linux-amd64/email.tmpl'
route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'email'
receivers:
  - name: 'email'
    email_configs:
      - to: '[email protected]'
        html: '{{ template "email.to.html" . }}'
        send_resolved: true
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname']

alertmanager.yml

{{ define "email.to.html" }}
{{ if gt (len .Alerts.Firing) 0 }}{{ range .Alerts }}
@告警: <br>
告警程序: prometheus_alert <br>
告警级别: {{ .Labels.severity }} 级<br>
告警类型: {{ .Labels.alertname }} <br>
故障主机: {{ .Labels.instance }} <br>
告警主题: {{ .Annotations.summary }} <br>
告警详情: {{ .Annotations.description }} <br>
触发时间: {{ .StartsAt }} <br>
{{ end }}
{{ end }}
{{ if gt (len .Alerts.Resolved) 0 }}{{ range .Alerts }}
@恢复: <br>
告警主机: {{ .Labels.instance }} <br>
告警主题: {{ .Annotations.summary }} <br>
恢复时间: {{ .EndsAt }} <br>
{{ end }}
{{ end }}
{{ end }}

email.tmpl   

prometheus发送告警信息_环境变量

     up指标用来监控主机是否宕机下线

     告警规则的rule的数据  就是这里的查询语句查询出来的结果

prometheus发送告警信息_git_02
prometheus发送告警信息_环境变量_03

内网代理发送邮件

#!/usr/bin/python3

import smtplib
import requests
from email.mime.text import MIMEText


def send_email():
    #构建邮件内容
    msg = MIMEText('主机要爆炸了 主机要爆炸了 主机要爆炸了 主机要爆炸了 主机要爆炸了 请速速撤离 ')
    msg['Subject'] = 'you are so bad'
    msg['From'] = '[email protected]'
    msg['To'] = '龙腾四海 运河就笑'

    server = None
    try:
        #创建SMTP对象并配置代理服务器地址及端口号
        server = smtplib.SMTP(host="smtp.163.com",port=25)

        #开始TLS加密传输
        server.starttls()

        #登录到SMTP服务器(若有)
        username = '[email protected]'
        password = '11111111'
        server.login(username, password)

        #发送邮件
        server.sendmail('[email protected]', ['[email protected]','[email protected]','[email protected]','[email protected]'], msg.as_string())

        print("邮件已成功发送")

    except Exception as e:
        print("邮件发送失败:", str(e))

    finally:
        # 关闭与SMTP服务器的连接
        if server != None:
                server.quit()

if __name__ == "__main__":
    send_email()

python无代理发送邮件

import socks
import socket
import smtplib
from email.mime.text import MIMEText

socks.set_default_proxy(socks.SOCKS5, "10.30.90.15", 18)
socket.socket = socks.socksocket

smtp_server = "smtp.163.com"
smtp_port = "25"
sender_email = "[email protected]"
sender_password = "222222"

server = smtplib.SMTP(smtp_server, smtp_port)
server.login(sender_email, sender_password)

msg = MIMEText('这是测试邮件', _charset='utf-8')
msg['Subject'] = '测试邮件主题'
msg['From'] = sender_email
msg['To'] = '[email protected]'

server.sendmail(sender_email, msg['To'], msg.as_string())
print("邮件已成功发送!")

python代理代码发送    

prometheus发送告警信息_html_04

Alertmanager集成python自研告警组件

 配置钉钉消息告警

global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.163.com:25'
  smtp_from: '[email protected]'
  smtp_auth_username: '[email protected]'
  smtp_auth_password: 'TCTJGF'
  smtp_require_tls: false
templates:
  - '/root/prom/alertmanager-0.26.0.linux-amd64/email.tmpl'
route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'dingding.webhook1'
  routes:
    - receiver: 'dingding.webhook1'
      continue: true  #发送完钉钉消息后继续发送邮件
    - receiver: 'email'

receivers:
  - name: 'email'
    email_configs:
      - to: '[email protected],[email protected],[email protected],[email protected]'
        html: '{{ template "email.to.html" . }}'
        send_resolved: true
  - name: 'dingding.webhook1'
    webhook_configs:
    - url: 'http://192.168.19.55:8060/dingtalk/webhook1/send'
      send_resolved: true

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname']

alertmanager.yml

## Request timeout
# timeout: 5s

## Uncomment following line in order to write template from scratch (be careful!)
#no_builtin_template: true

## Customizable templates path
templates:
  - contrib/templates/legacy/template.tmpl

## You can also override default template using `default_message`
## The following example to use the 'legacy' template from v0.3.0
#default_message:
#  title: '{{ template "legacy.title" . }}'
#  text: '{{ template "legacy.content" . }}'

## Targets, previously was known as "profiles"
targets:
  webhook1:
    url: https://oapi.dingtalk.com/robot/send?access_token=7f7fe25
    # secret for signature
    secret: 4ee16602f4108eb793475803cfa26a1be5210ccf1
 # webhook2:
 #   url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx
 # webhook_legacy:
 #   url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx
    # Customize template content
 #   message:
      # Use legacy template
 #     title: '{{ template "legacy.title" . }}'
 #     text: '{{ template "legacy.content" . }}'
 # webhook_mention_all:
 #   url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx
 #   mention:
 #     all: true
 # webhook_mention_users:
 #   url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx
 #   mention:
 #     mobiles: ['156xxxx8827', '189xxxx8325']

prometheus-webhook-dingtalk.yml   

prometheus发送告警信息_html_05

    需单独安装  prometheus-webhook-dingtalk-2.1.0.linux-amd64组件

内网代理钉钉消息

     内网中只有一台机器可以联网

     监控的主机无法连接外部的机器

     给prometheus-webhook-dingtalk进程设置代理地址环境变量   不能设置成系统变量,因为这样会影响系统上所有的https请求

prometheus发送告警信息_html_06

     添加进程环境变量

      //在程序启动的时候注入多个环境变量
      export https_proxy=http://ct:[email protected]:3128 && export http_proxy=http://cta:[email protected]:3128 && ./prometheus-webhook-dingtalk --config.file=config.yml

prometheus发送告警信息_环境变量_07
prometheus发送告警信息_html_08
prometheus发送告警信息_环境变量_09

      1.把代理地址做为环境变量注入到指定的进程中

         //在程序启动的时候注入一个环境变量
         export https_proxy=http://aaaaa:[email protected]:3128 && ./prometheus-webhook-dingtalk --config.file=config.yml

        //在程序启动的时候注入多个环境变量
        export https_proxy=http://ct:[email protected]:3128 && export http_proxy=http://cta:[email protected]:3128 && ./prometheus-webhook-dingtalk --config.file=config.yml

       2.查看进程的环境变量

          ps -ef | grep webhook

          cat /proc/5663/environ | tr '\0' '\n'

          cat /proc/5663/comm

prometheus发送告警信息_环境变量_10
prometheus发送告警信息_git_11

作者:不懂123


About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK