作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
目录
- 一.标签概述
- 1.什么是标签
- 2.标签主要有两种表现形式
- 3.Prometheus对数据处理的流程
- 二.relabel_configs修改target标签案例
- 1.为targets自定义打标签案例
- 2.使用target_label新增标签
- 2.relabel_configs替换标签replace案例
- 3.relabel_configs新增标签映射labelmap案例
- 4.relabel_configs删除标签labeldrop案例
- 三.metric_relabel_configs修改metric标签案例
- 1.metric_relabel_configs删除指标
- 2.黑盒监控示例
- 3.K8S示例
一.标签概述
1.什么是标签
标签用于对数据分组和分类，利用标签可以将数据进行过滤筛选。标签管理的常见场景:- 1.删除不必要的指标;- 2.从指标中删除敏感或不需要的标签;- 3.添加，编辑或修改指标的标签值或标签格式;标签的分类:- 默认标签:Prometheus自身内置的标签，格式为"__LABLE__"。如上图所示，典型点如下所示:- "__metrics_path__"- "__address__"- "__scheme__"- "__scrape_interval__"- "__scrape_timeout__"- "instance"- "job"- 应用标签:应用本身内置，尤其是监控特定的服务，会有对应的应用标签，格式一般为"__LABLE"如下图所示，以consul服务为例，典型点如下所示:- "__meta_consul_address"- "__meta_consul_dc"- ...- 自定义标签:指的是用户自定义的标签，我们在定义targets可以自定义。
2.标签主要有两种表现形式
- 私有标签:以"__*"样式存在，用于获取监控目标的默认元数据属性，比如"__scheme__","__address__"，"__metrics_path__"等。- 普通标签:对监控指标进行各种灵活管理操作，常见的操作有删除不必要敏感数据，添加，编辑或修改指标标签纸或者标签格式等。
3.Prometheus对数据处理的流程
- 1.服务发现: 支持静态发现和动态发现，主要是找打到对应的target。- 2.配置: 加载"__scheme__","__address__"，"__metrics_path__"等信息。- 3.重新标记: relabel_configs，主要针对要监控的target的标签。- 4.抓取: 开始抓取数据。- 5.重新标记: metric_relabel_configs，主要针对已经抓取回来的metrics的标签的操作。
二.relabel_configs修改target标签案例
1.为targets自定义打标签案例
如上图所示，我们成为target自定义了标签。测试代码如下:
[root@prometheus-server31 ~]# tail -6 /yinzhengjie/softwares/prometheus-2.53.2.linux-amd64/prometheus.yml- job_name: "yinzhengjie-node-exporter"static_configs:- targets: ["10.0.0.31:9100","10.0.0.41:9100","10.0.0.42:9100","10.0.0.43:9100"]labels:auther: yinzhengjieblog: https://www.cnblogs.com/yinzhengjie
[root@prometheus-server31 ~]#
2.使用target_label新增标签
1.编写代码
[root@prometheus-server31 ~]# tail -7 /yinzhengjie/softwares/prometheus-2.53.2.linux-amd64/prometheus.yml- job_name: "yinzhengjie-node-exporter"static_configs:- targets: ["10.0.0.31:9100","10.0.0.41:9100","10.0.0.42:9100","10.0.0.43:9100"]relabel_configs:- source_labels:- jobtarget_label: jasonyin2020_jobs
[root@prometheus-server31 ~]# 2.验证效果
测试结果如上图所示。
2.relabel_configs替换标签replace案例
1.编写代码
[root@prometheus-server31 ~]# tail -29 /yinzhengjie/softwares/prometheus-2.53.2.linux-amd64/prometheus.yml - job_name: "yinzhengjie-node-exporter"static_configs:- targets: ["10.0.0.31:9100","10.0.0.41:9100","10.0.0.42:9100","10.0.0.43:9100"]labels:auther: yinzhengjieblog: https://www.cnblogs.com/yinzhengjierelabel_configs:# 指定正则表达式匹配成功的label进行标签管理的列表- source_labels:- __scheme__- __address__- __metrics_path__# 表示source_labels对应Label的名称或值进行匹配此处指定的正则表达式。# 此处我们对数据进行了分组，后面replacement会是哟合格"${1}"和"$2"进行引用。regex: "(http|https)(.*)" # 指定用于连接多个source_labels为一个字符串的分隔符，若不指定，默认为分号";"。# 假设源数据如下:# __address__="10.0.0.31:9100"# __metrics_path__="/metrics"# __scheme__="http"# 拼接后操作的结果为: "http10.0.0.31:9100/metrics"separator: ""# 在进行Label替换的时候，可以将原来的source_labels替换为指定修改后的label。# 将来会新加一个标签，标签的名称为"yinzhengjie_prometheus_ep"，值为replacement的数据。target_label: "yinzhengjie_prometheus_ep"# 替换标签时，将target_label对应的值进行修改成此处的值replacement: "${1}://${2}"# 对Label或指标进行管理，场景的动作有replace|keep|drop|lablemap|labeldrop等，默认为replace。action: replace
[root@prometheus-server31 ~]# 2.验证效果
测试结果如上图所示。
3.relabel_configs新增标签映射labelmap案例
1.编写代码
[root@prometheus-server31 ~]# tail -9 /yinzhengjie/softwares/prometheus-2.53.2.linux-amd64/prometheus.yml - job_name: "yinzhengjie-node-exporter"static_configs:- targets: ["10.0.0.31:9100","10.0.0.41:9100","10.0.0.42:9100","10.0.0.43:9100"]relabel_configs:- regex: "(job|app)"replacement: "${1}_yinzhengjie_labelmap_kubernetes"# labelmap一般用于生成新的标签，通常用于驱逐匹配标签名的一部分生成新表现，旧的标签依旧会存在。# 将regex对source label中指定的标签名称进行匹配，而后将匹配到的标签的赋值给replacement字段指定的标签。action: labelmap
[root@prometheus-server31 ~]# 2.验证效果
测试结果如上图所示。
4.relabel_configs删除标签labeldrop案例
1.编写代码
[root@prometheus-server31 ~]# tail /yinzhengjie/softwares/prometheus-2.53.2.linux-amd64/prometheus.yml - job_name: "yinzhengjie-node-exporter"static_configs:- targets: ["10.0.0.31:9100","10.0.0.41:9100","10.0.0.42:9100","10.0.0.43:9100"]relabel_configs:- regex: "(job|app)"replacement: "${1}_yinzhengjie_labelmap_kubernetes"action: labelmap- regex: "(job|app)"# 删除regex匹配到的标签action: labeldrop
[root@prometheus-server31 ~]# 2.验证效果
测试结果如上图所示。
三.metric_relabel_configs修改metric标签案例
1.metric_relabel_configs删除指标
1.编写代码
[root@prometheus-server31 ~]# tail -8 /yinzhengjie/softwares/prometheus-2.53.2.linux-amd64/prometheus.yml - job_name: "yinzhengjie-node-exporter"static_configs:- targets: ["10.0.0.31:9100","10.0.0.41:9100","10.0.0.42:9100","10.0.0.43:9100"]metric_relabel_configs:- source_labels:- __name__regex: "node_cpu_.*"action: drop
[root@prometheus-server31 ~]#
[root@prometheus-server31 ~]# for i in 31 41 42 43; do curl -s 10.0.0.$i:9100/metrics | grep ^node_cpu | wc -l;done
20
20
20
20
[root@prometheus-server31 ~]# 2.验证效果
测试结果如上图所示。温馨提示:如果你已经做过relabel_configs相关实验，建议将Prometheus的数据目录删除清空后测试效果更佳哟~
2.黑盒监控示例
推荐阅读:https://www.cnblogs.com/yinzhengjie/p/18540807
3.K8S示例
参考链接:https://github.com/prometheus/prometheus/blob/release-2.53/documentation/examples/prometheus-kubernetes.ymlhttps://prometheus.io/docs/prometheus/2.53/configuration/configuration/#kubernetes_sd_config源代码如下:
[root@prometheus-server31 ~]# egrep -v "^.*#|^$" /tmp/prometheus-kubernetes.yml
global:keep_dropped_targets: 100
scrape_configs:- job_name: "kubernetes-apiservers"kubernetes_sd_configs:- role: endpointsscheme: httpstls_config:ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crtauthorization:credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenrelabel_configs:- source_labels:[__meta_kubernetes_namespace,__meta_kubernetes_service_name,__meta_kubernetes_endpoint_port_name,]action: keepregex: default;kubernetes;https- job_name: "kubernetes-nodes"scheme: httpstls_config:ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crtauthorization:credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenkubernetes_sd_configs:- role: noderelabel_configs:- action: labelmapregex: __meta_kubernetes_node_label_(.+)- job_name: "kubernetes-cadvisor"scheme: httpsmetrics_path: /metrics/cadvisortls_config:ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crtauthorization:credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenkubernetes_sd_configs:- role: noderelabel_configs:- action: labelmapregex: __meta_kubernetes_node_label_(.+)- job_name: "kubernetes-service-endpoints"kubernetes_sd_configs:- role: endpointsrelabel_configs:- action: labelmapregex: __meta_kubernetes_service_label_(.+)- source_labels: [__meta_kubernetes_namespace]action: replacetarget_label: namespace- source_labels: [__meta_kubernetes_service_name]action: replacetarget_label: service- job_name: "kubernetes-services"metrics_path: /probeparams:module: [http_2xx]kubernetes_sd_configs:- role: servicerelabel_configs:- source_labels: [__address__]target_label: __param_target- target_label: __address__replacement: blackbox-exporter.example.com:9115- source_labels: [__param_target]target_label: instance- action: labelmapregex: __meta_kubernetes_service_label_(.+)- source_labels: [__meta_kubernetes_namespace]target_label: namespace- source_labels: [__meta_kubernetes_service_name]target_label: service- job_name: "kubernetes-ingresses"metrics_path: /probeparams:module: [http_2xx]kubernetes_sd_configs:- role: ingressrelabel_configs:- source_labels:[__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path,]regex: (.+);(.+);(.+)replacement: ${1}://${2}${3}target_label: __param_target- target_label: __address__replacement: blackbox-exporter.example.com:9115- source_labels: [__param_target]target_label: instance- action: labelmapregex: __meta_kubernetes_ingress_label_(.+)- source_labels: [__meta_kubernetes_namespace]target_label: namespace- source_labels: [__meta_kubernetes_ingress_name]target_label: ingress- job_name: "kubernetes-pods"kubernetes_sd_configs:- role: podrelabel_configs:- action: labelmapregex: __meta_kubernetes_pod_label_(.+)- source_labels: [__meta_kubernetes_namespace]action: replacetarget_label: namespace- source_labels: [__meta_kubernetes_pod_name]action: replacetarget_label: pod
[root@prometheus-server31 ~]#
