promcheck

A tool to identify faulty Prometheus rules

About

promcheck enables you to identify recording or alerting rules using missing metrics or wrong label matchers (e.g. because of exporter changes or human-errors).

promcheck validates Prometheus vector selectors and checks if they return a result value or not. As a basis for validation, promcheck uses Prometheus rule files, but it can also query rules directly from a running Prometheus instance. It scans the PromQL expression of each recording and alerting rule, takes the individual referenced selectors out of it and probes them against a remote Prometheus instance.

Overview

Installation

promcheck is available on Linux, OSX and Windows platforms. Binaries for Linux, Windows and Mac are available as tarballs in the release page.

You may also build promcheck from source (using Go 1.17+). In order to build promcheck from source you must:

Clone this repository
Run make build

Basic Usage

promcheck can be used in two different modes:

Validate rules from existing rule files (and export results in various formats)
Validate rules from a running Prometheus instance (and export results in various formats)

promcheck can also be executed as a Prometheus exporter to check a set of rules on a regular basis and export results as scrapeable Prometheus metrics via http.

Use-Cases

What can you do with this? Possible use-cases might be:

Run promcheck manually as a cli tool to check rules.
Add promcheck to your CI/CD automation pipeline to run integration tests on your rules.
Run promcheck as an exporter, scrape its metrics and alert in case selectors do not return results anymore.

Validate rules from a running Prometheus instance

promcheck --prometheus.url="http://0.0.0.0:9090"

Argument Reference:

--prometheus.url - The Prometheus instance to probe selectors against

Validate rules from existing rule files

promcheck --prometheus.url="http://0.0.0.0:9090" --check.file=rules.yaml

Argument Reference:

--prometheus.url - The Prometheus instance to probe selectors against
--check.file - The Prometheus rule file(s) to validate

Rule group files can be passed in various ways. Click to expand!

Prometheus Exporter

# example: run promcheck as a prometheus exporter.
# promcheck will validate all rules from the remote instance.
promcheck --prometheus.url="http://0.0.0.0:9090" \
          --exporter.enabled=true
          
# example: bind on port 9093, run promcheck every 5 min (300 sec.)
promcheck --prometheus.url="http://0.0.0.0:9090" \
          --exporter.enabled=true \ 
          --exporter.interval=300 \
          --exporter.addr=0.0.0.0:9093

# example: run promcheck as a prometheus exporter.
# promcheck will validate all rules from the rules.yaml file  
promcheck --prometheus.url="http://0.0.0.0:9090" \
          --exporter.enabled=true \ 
          --exporter.interval=300 \
          --exporter.addr=0.0.0.0:9093 \
          --check.file=rules.yaml

Argument Reference:

--prometheus.url - The Prometheus instance to probe selectors against
--check.file - The Prometheus rule file(s) to validate.
--exporter.enabled - Run promcheck as a Prometheus exporter
--exporter.addr - The exporter's http address
--exporter.interval - The interval in minutes to run promcheck and update metrics

Configuration

For a full list of flags, please also use promcheck --help.

Flags:
  -h, --help                                               Show context-sensitive help.
      --prometheus.url="http://0.0.0.0:9090"               The Prometheus base url
      --check.ignore-selector=CHECK.IGNORE-SELECTOR,...    Regexp of selectors to ignore
      --check.ignore-group=CHECK.IGNORE-GROUP,...          Regexp of rule groups to ignore
      --check.delay=0.1                                    Delay in seconds between probe requests
      --check.file=STRING                                  The rule files to check.
      --output.format="graph"                              The output format to use
      --output.no-color                                    Toggle colored output
      --exporter.enabled                                   Run promcheck as a prometheus exporter
      --exporter.addr="0.0.0:9093"                         The address the http server is running at
      --exporter.interval=300                              Delay in seconds between promcheck runs
      --metrics.profile                                    Enable pprof profiling
      --metrics.runtime                                    Enable runtime metrics
      --metrics.prefix=""                                  Set metrics prefix path
      --log.json                                           Tell promcheck to log json and not key value pairs
      --log.level="info"                                   The log level to use for filtering logs

promcheck uses 256 colors terminal mode. On 'nix OS system make sure the TERM environment variable is set.

export TERM=xterm-256color

Usage Information

Keep in mind that promcheck may also contain false positives, since there may be vector selectors in rules that intentionally do not return a result value.

promcheck does a single HTTP request per vector selector to be probed against the remote Prometheus instance. With many rules to validate, execution time can take longer and lead to many HTTP requests. The interval between probes can be changed with the --check.delay flag, which results in fewer requests but increases the runtime of the tool.

Output formats

Right now, the following output formats are supported:

--output.format=graph - Text format, colored or non-colored (--output.no-color) (Default)
--output.format=json - JSON format
--output.format=yaml - YAML format

There might be more formats in near future. Feel free to contribute!

Container Usage

promcheck can also be executed from within a container. The latest container image of promcheck is hosted on quay.io.

To run promcheck from within a container (assuming that there is a rule file named rules.yaml in the current directory), run:

docker run -v $(pwd):/tmp --rm quay.io/cbrgm/promcheck:latest --prometheus.url='http://0.0.0.0:9090' --check.file="/tmp/rules.yaml"

To run promcheck from within a container as a Prometheus exporter, run:

docker run --rm -p 9093:9093 quay.io/cbrgm/promcheck:latest --prometheus.url='http://0.0.0.0:9090' --exporter.enabled

Kubernetes Deployment

promcheck can be executed as a Prometheus exporter to validate a set of rules on a regular basis. Please refer to the kubernetes.yaml file for a basic deployment example.

Metrics

promcheck_validation_rule_groups_total - (Gauge) Total number of evaluated rule groups.
promcheck_validation_rules_total - (Gauge) Total number of evaluated rules.
promcheck_validation_selectors_total - (Gauge) Total number of evaluated selectors. Label selectors:
- file - The rules file
- group - The rule group name
- rule - The rule name
- status - The status failed or success

Here are some basic examples:

Example: PromQL queries Click to expand!

Example: Alert on selectors without a result Click to expand!

Examples

Please refer below for some basic usage examples demonstrating what promcheck can do for you!

Basic Example validating multiple rule groups

Input:

rules.yaml (Click to expand!)

Command:

➜ ./promcheck --check.file 'rules.yaml' --prometheus.url http://0.0.0.0:9090

Prometheus instance running locally on http://0.0.0.0:9090

Output:

.
└── [file] examples/rules_multiple_groups.yaml
    ├── [group] kubernetes-apps-demo-group
    │   ├── [0/1] KubePodCrashLooping
    │   │   └── [✖] kube_pod_container_status_waiting_reason{job="kube-state-metrics",reason="CrashLoopBackOff"}
    │   └── [2/2] KubePodNotReady
    │       ├── [✔] kube_pod_status_phase{job="kube-state-metrics",phase=~"Pending|Unknown"}
    │       └── [✔] kube_pod_owner{owner_kind!="Job"}
    ├── [group] kubernetes-system-scheduler-demo-group
    │   └── [1/1] KubeSchedulerDown
    │       └── [✔] up{job="kube-scheduler"}
    └── [group] kubernetes-system-controller-manager-demo-group
        └── [1/1] KubeControllerManagerDown
            └── [✔] up{job="kube-controller-manager"}

Rules validated total: 4
Selectors total: 5, Results found: 4, No Results found 1 (Failed/Total: 20.00%)

Command:

Ignore rule group kubernetes-system-controller-manager-demo-group

➜ ./promcheck --check.file 'rules.yaml' \
              --check.ignore-group 'kubernetes-system-controller-manager-demo-group' \
              --prometheus.url http://0.0.0.0:9090

Prometheus instance running locally on http://0.0.0.0:9090

Output:

.
└── [file] examples/rules_multiple_groups.yaml
    ├── [group] kubernetes-apps-demo-group
    │   ├── [2/2] KubePodNotReady
    │   │   ├── [✔] kube_pod_status_phase{job="kube-state-metrics",phase=~"Pending|Unknown"}
    │   │   └── [✔] kube_pod_owner{owner_kind!="Job"}
    │   └── [0/1] KubePodCrashLooping
    │       └── [✖] kube_pod_container_status_waiting_reason{job="kube-state-metrics",reason="CrashLoopBackOff"}
    └── [group] kubernetes-system-scheduler-demo-group
        └── [1/1] KubeSchedulerDown
            └── [✔] up{job="kube-scheduler"}

Rules validated total: 4
Selectors total: 4, Results found: 3, No Results found 1 (Failed/Total: 25.00%)

Command:

Output json:

➜ ./promcheck --check.file 'rules.yaml' \
              --check.ignore-group 'kubernetes-system-controller-manager-demo-group' \
              --prometheus.url http://0.0.0.0:9090
              --output.format json

Output:

{
  "promcheck": {
    "results": [
      {
        "file": "examples/rules_multiple_groups.yaml",
        "group": "kubernetes-apps-demo-group",
        "name": "KubePodCrashLooping",
        "expression": "max_over_time(kube_pod_container_status_waiting_reason{reason=\"CrashLoopBackOff\", job=\"kube-state-metrics\"}[5m]) \u003e= 1\n",
        "no_results": [
          "kube_pod_container_status_waiting_reason{job=\"kube-state-metrics\",reason=\"CrashLoopBackOff\"}"
        ],
        "results": []
      },
      {
        "file": "examples/rules_multiple_groups.yaml",
        "group": "kubernetes-apps-demo-group",
        "name": "KubePodNotReady",
        "expression": "sum by (namespace, pod) (\n  max by(namespace, pod) (\n    kube_pod_status_phase{job=\"kube-state-metrics\", phase=~\"Pending|Unknown\"}\n  ) * on(namespace, pod) group_left(owner_kind) topk by(namespace, pod) (\n    1, max by(namespace, pod, owner_kind) (kube_pod_owner{owner_kind!=\"Job\"})\n  )\n) \u003e 0\n",
        "no_results": [],
        "results": [
          "kube_pod_status_phase{job=\"kube-state-metrics\",phase=~\"Pending|Unknown\"}",
          "kube_pod_owner{owner_kind!=\"Job\"}"
        ]
      },
      {
        "file": "examples/rules_multiple_groups.yaml",
        "group": "kubernetes-system-scheduler-demo-group",
        "name": "KubeSchedulerDown",
        "expression": "absent(up{job=\"kube-scheduler\"} == 1)\n",
        "no_results": [],
        "results": [
          "up{job=\"kube-scheduler\"}"
        ]
      }
    ],
    "groups_total": 2,
    "rules_warnings": 3,
    "rules_total": 4,
    "selectors_success_total": 3,
    "selectors_failed_total": 1,
    "ratio_failed_total": 25.00
  }
}

Contributing & License

Feel free to submit changes! See the Contributing Guide. This project is open-source and is developed under the terms of the Apache 2.0 License.

GitHub - cbrgm/promcheck: A tool and exporter to identify faulty Prometheus rule...

promcheck

About

Overview

Installation

Basic Usage

Use-Cases

Validate rules from a running Prometheus instance

Validate rules from existing rule files

Prometheus Exporter

Configuration

Usage Information

Output formats

Container Usage

Kubernetes Deployment

Metrics

Examples

Basic Example validating multiple rule groups

Contributing & License

Recommend

Belt drive vs chain: Which is best for your ebike?

AMD 宣布采用 3D 缓存技术的 Epyc 处理器

万泰生物2021年扣非净利大增215.32% 二价HPV疫苗持续放量

飞利浦 Hue 推出 Play 电视渐变灯带：1600 万种色彩渐变氛围

Shields up! Why Russia’s war on Ukraine should matter to security pros

用python-dotenv解决代码与敏感信息的分离

Can Shiba Inu Outperform Other Top Meme Coins For Daily Gains This Weekend?

15 年后，苹果重新变回「电脑公司」

Python爬虫编程思想（138）：多线程和多进程爬虫--从Thread类继承

Okta’s Investigation of the January 2022 Compromise

About Joyk