Prometheus跨集群采集
source link: https://blog.51cto.com/saynaihe/5663048
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
恩不想搭建太多prometheus了，想用一个prometheus,当然了 前提是我A集群可以连通B集群网络，实现
Prometheus跨集群采集采集
关于A集群
A集群 以及prometheus搭建 参照： Kubernetes 1.20.5 安装Prometheus-Oprator
B集群操作参照：阳明大佬 Prometheus 监控外部 Kubernetes 集群
创建RBAC对象：
cat rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus
rules:
- apiGroups:
- ""
resources:
- nodes
- services
- endpoints
- pods
- nodes/proxy
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- configmaps
- nodes/metrics
verbs:
- get
- nonResourceURLs:
- /metrics
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: monitoring
kuberctl apply -f rbac.yaml
[root@sh-master-01 prometheus]# kubectl get sa -n monitoring
[root@sh-master-01 prometheus]# kubectl get secret -n monitoring
特别强调sa secret token
why怎么没有见secret？参照： https://itnext.io/big-change-in-k8s-1-24-about-serviceaccounts-and-their-secrets-4b909a4af4e0 恩 1.24发生了改变。我这里的版本是1.25.so :
[root@sh-master-01 manifests]# kubectl create token prometheus -n monitoring --duration=999999h
网上很多yq方式用的？
0 kubectl get secret prometheus -n monitoring -o yaml|yq r - data.token|base64 -D
可是yq安装上了还是不太会玩？怎么办？还是用奔方法吧！：
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: pod1
name: pod1
spec:
terminationGracePeriodSeconds: 0
serviceAccount: prometheus
containers:
- image: nginx
imagePullPolicy: IfNotPresent
name: pod1
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Always
status: {}
kubectl apply -f pod.yaml -n monitoring
kubectl exec -it pod1 -n monitoring -- cat /run/secrets/kubernetes.io/serviceaccount/token
注意没有获取两次的token还会不一样的这个稍后研究.恩不知道是不是可以kubectl create token prometheus --duration=999999h 这样？(还是参照 https://itnext.io/big-change-in-k8s-1-24-about-serviceaccounts-and-their-secrets-4b909a4af4e0)
prometheus集群中重新生成additional-configs
A集群 promethus配置文件夹中：修改prometheus-additional.yaml，复制token 替换bearer_token 中XXXXXXXXXXXXXXXXXXX
注：B集群apiserver 地址为10.0.2.28:6443，自己整要修改！
- job_name: 'kubernetes-apiservers-other-cluster'
kubernetes_sd_configs:
- role: endpoints
api_server: https://10.0.2.28:6443
tls_config:
insecure_skip_verify: true
bearer_token: 'XXXXXXXXXXXXXXXXXXX'
tls_config:
insecure_skip_verify: true
bearer_token: 'XXXXXXXXXXXXXXXXXXX'
scheme: https
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: default;kubernetes;https
- target_label: __address__
replacement: 10.0.2.28:6443
- job_name: 'kubernetes-nodes-other-cluster'
kubernetes_sd_configs:
- role: node
api_server: https://10.0.2.28:6443
tls_config:
insecure_skip_verify: true
bearer_token: 'XXXXXXXXXXXXXXXXXXX'
tls_config:
insecure_skip_verify: true
bearer_token: 'XXXXXXXXXXXXXXXXXXX'
scheme: https
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: 10.0.2.28:6443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics
- job_name: 'kubernetes-nodes-cadvisor-other-cluster'
kubernetes_sd_configs:
- role: node
api_server: https://10.0.2.28:6443
tls_config:
insecure_skip_verify: true
bearer_token: 'XXXXXXXXXXXXXXXXXXX'
tls_config:
insecure_skip_verify: true
bearer_token: 'XXXXXXXXXXXXXXXXXXX'
scheme: https
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: 10.0.2.28:6443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
- job_name: 'kubernetes-state-metrics-other-cluster'
kubernetes_sd_configs:
- role: endpoints
api_server: https://10.0.2.28:6443
tls_config:
insecure_skip_verify: true
bearer_token: 'XXXXXXXXXXXXXXXXXXX'
tls_config:
insecure_skip_verify: true
bearer_token: 'XXXXXXXXXXXXXXXXXXX'
scheme: https
relabel_configs:
- source_labels: [__meta_kubernetes_service_name]
action: keep
regex: '^(kube-state-metrics)$'
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__address__]
action: replace
target_label: instance
- target_label: __address__
replacement: 10.0.2.28:6443
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_name, __meta_kubernetes_pod_container_port_number]
regex: ([^;]+);([^;]+);([^;]+)
target_label: __metrics_path__
replacement: /api/v1/namespaces/${1}/pods/http:${2}:${3}/proxy/metrics
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: service_name
kubectl delete secret additional-configs -n monitoring
kubectl create secret generic additional-configs --from-file=prometheus-additional.yaml -n monitoring
grafana看一下，图还是有点问题忽略了哈哈哈。不想去研究是版本问题或者其他了，目前这样就算是实现这些吧！
总结一下：
现实环境中我应该不会那么玩，还是跟原来一样，每个k8s集群搞一个prometheus-oprator集群，然后可以连接一个grafana…
其实那么的搞了一圈玩一下就发现了K8s1.24 后 BIG change in K8s 1.24 about ServiceAccounts and their Secrets
Recommend
-
33
kubernetes集群全栈监控报警方案kube-prometheus http://www.servicemesher.com/blog/prometheus-operator-manual/ https://github.com/coreos/prometheus-operator https://github.com/coreos/kube-prometheus...
-
25
在 Kubernetes 集群上部署 Kafka 发表于 March 17, 2020 person standing on a mountain overlooking the ocean 最近在测试日志采集的时候，发现日志数据量...
-
13
【编者的话】对于Prometheus的组件能力是毋庸置疑的，但是使用久了会发现很多的性能问题，诸如内存问题、大规模拉取问题、大规模存储问题等等。如何基于云原生Prometheus进行Kubernetes集群基础监控大规模数据拉取，本文将会给出答案。
-
6
如何优雅地使用云原生 Prometheus 监控集群 作者陈凯烨，腾讯云前端开发工程师。负责 TKE 集群，弹性集群和云原生监控等模块控制台开发。 Prometheus 是一套开源的系统监控报警框架。2016 年，Prometheus 正式加入 Cloud Native Computing Foun...
-
12
Prometheus 监控外部 Kubernetes 集群 发表于 June 29, 2020 ...
-
21
本文预计阅读时间 12 分钟 prometheus监控es，同样采用exporter的方案。 项目地址： elasticsearch_exporter：https://github.com/justwatchcom/elasticsearch_exporter 1、安装部署 现有es三...
-
8
【中间件】Prometheus基于AOP实现埋点采集上报 ...
-
6
Prometheus监控Kubernetes系列3——业务指标采集 · Service Mesh|服务网格中文社区 由于容器化和微服务的大力发展，Kubernetes基本已经统一了容器管理方案，当我们使用Kubernetes来进行容器化管理的时候，全面监控Ku...
-
10
使用 Vmagent 代替 Prometheus 采集监控指标-51CTO.COM 使用 Vmagent 代替 Prometheus 采集监控指标 作者：阳明 2022-05-12 08:01:26 我们以抓取 Kubernetes 集群指标为例说明如何使用...
-
7
采集Java程序JVM信息 创建 Spring Boot Application 应用程序# 进行
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK