从0开始搭建Prometheus集群架构(Prometheus联邦模式+Grafana+node_exporter+VirctoriaMetrics远端仓库)

一、环境说明

主机名	主机IP	操作系统
prometheus-server31	10.0.0.31	ubuntu 22.04 LTS
prometheus-server32	10.0.0.32	ubuntu 22.04 LTS
prometheus-server33	10.0.0.33	ubuntu 22.04 LTS
node-exporter41	10.0.0.41	ubuntu 22.04 LTS
node-exporter42	10.0.0.42	ubuntu 22.04 LTS
node-exporter43	10.0.0.43	ubuntu 22.04 LTS

二、服务版本说明

服务名	版本号	下载链接
Prometheus	2.53.4	https://github.com/prometheus/prometheus/releases/download/v2.53.4/prometheus-2.53.4.linux-amd64.tar.gz
Grafana	9.5.21	https://dl.grafana.com/enterprise/release/grafana-enterprise_9.5.21_amd64.deb
no-exporter	1.9.0	https://github.com/prometheus-community/elasticsearch_exporter/releases/download/v1.9.0/elasticsearch_exporter-1.9.0.linux-amd64.tar.gz
VictoriaMetrics	1.93.16	https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.93.16/victoria-metrics-linux-amd64-v1.93.16.tar.gz
consul	1.20.5	https://releases.hashicorp.com/consul/1.20.5/consul_1.20.5_linux_amd64.zip

三、架构说明

• 本集群采用联邦模式部署，基于consul服务实现自动发现，采集Linux服务器的基本指标，将数据存储在VictoriaMetrics远端仓库，Grafana出图展示

• 关于联邦模式

  • 类似于古代帝王的分封制

  • prometheus-server32直接采集node-exporter41的数据

  • prometheus-server33采集consul集群的数据，node-exporter42、node-exporter43自动注册到consul集群

  • prometheus-server31采集prometheus-server32、prometheus-server33采集到数据，也就是对其做一个汇总，并将数据存储到远端仓库

  • Grafana从远端仓库获取数据出图展示

四、集群部署

1.所有prometheus-server节点安装Prometheus

安装过程简单，参考https://www.cnblogs.com/dezyan/p/18794577
我编写了脚本一键部署，安装路径为/dezyan/softwares/prometheus-2.53.4.linux-amd64/
配置了一键启动脚本(systemctl start prometheus)

2.所有node-exporter节点安装node-exporter

安装过程简单，参考https://www.cnblogs.com/dezyan/p/18794577
我编写了脚本一键部署，安装路径为/dezyan/softwares/node_exporter-1.9.0.linux-amd64
配置了一键启动脚本(systemctl start node-exporter.service )

3.部署VictoriaMetrics单机版

• 资源有限，我直接部署在了10.0.0.43节点上

3.1 下载\解压victoriametrics

[root@node-exporter43 ~]# wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.93.16/victoria-metrics-linux-amd64-v1.93.16.tar.gz
[root@node-exporter43 ~]# tar xf victoria-metrics-linux-amd64-v1.93.16.tar.gz  -C /usr/local/bin/

3.2 编写启动脚本

[root@node-exporter43 ~]# cat > /etc/systemd/system/victoria-metrics.service <<EOF
[Unit]
Description=Linux VictoriaMetrics Server
Documentation=https://docs.victoriametrics.com/
After=network.target[Service]
ExecStart=/usr/local/bin/victoria-metrics-prod  \-httpListenAddr=0.0.0.0:8428 \-storageDataPath=/dezyan/data/victoria-metrics \-retentionPeriod=3[Install]
WantedBy=multi-user.target
EOF[root@node-exporter43 ~]# systemctl daemon-reload
[root@node-exporter43 ~]# systemctl enable --now victoria-metrics.service
[root@node-exporter43 ~]# systemctl status victoria-metrics

3.3检查端口是否存活

[root@node-exporter43 ~]# ss -ntl | grep 8428
LISTEN 0      4096         0.0.0.0:8428      0.0.0.0:*

3.4 查看webUI

http://10.0.0.43:8428/

4.部署consul集群

资源有限，直接部署在了node-exporter三个节点上

4.1 下载/解压consul、拷贝到其他节点

[root@node-exporter41 ~]# wget https://releases.hashicorp.com/consul/1.20.5/consul_1.20.5_linux_amd64.zip
[root@node-exporter41 ~]# unzip consul_1.20.5_linux_amd64.zip -d /usr/local/bin/[root@node-exporter41 ~]# scp /usr/local/bin/consul 10.0.0.42:/usr/local/bin
[root@node-exporter41 ~]# scp /usr/local/bin/consul 10.0.0.43:/usr/local/bin

4.2 运行consul集群

指定10.0.0.43作为集群的服务端

服务端43:
[root@node-exporter43 ~]# consul agent -server -bootstrap -bind=10.0.0.43 -data-dir=/dezyan/softwares/consul -client=10.0.0.43 -ui
客户端42:
[root@node-exporter42 ~]# consul agent  -bind=10.0.0.42 -data-dir=/dezyan/softwares/consul -client=10.0.0.42 -ui -retry-join=10.0.0.43
客户端41:
[root@node-exporter41 ~]# consul agent -server -bind=10.0.0.41 -data-dir=/dezyan/softwares/consul -client=10.0.0.41 -ui -retry-join=10.0.0.43

4.3 查看各个节点的监听端口 访问webUI测试

[root@node-exporter41 ~]# ss -ntl | grep 8500
[root@node-exporter42 ~]# ss -ntl | grep 8500
[root@node-exporter43 ~]# ss -ntl | grep 8500http://10.0.0.41:8500/ui/dc1/nodes
http://10.0.0.42:8500/ui/dc1/nodes
http://10.0.0.43:8500/ui/dc1/nodes

5.部署Grafana

参考https://www.cnblogs.com/dezyan/p/18794577

五、集群配置

1.Prometheus-server32节点

1.1编辑配置文件

[root@prometheus-server32 ~]# cd /dezyan/softwares/prometheus-2.53.4.linux-amd64/
[root@prometheus-server32 ~]# vim prometheus.yml- job_name: linux96-node-exporterstatic_configs:- targets: - 10.0.0.41:9100

1.2 重新加载配置

[root@prometheus-server32 ~]# curl -X POST http://10.0.0.32:9090/-/reload

2.Prometheus-server33节点

2.1编辑配置文件

[root@prometheus-server31 ~]# cd /dezyan/softwares/prometheus-2.53.4.linux-amd64/
[root@prometheus-server31 ~]# vim prometheus.yml- job_name: "consul-seriver-discovery"# 配置基于consul的服务发现consul_sd_configs:# 指定consul的服务器地址，若不指定，则默认值为"localhost:8500".- server: 10.0.0.43:8500- server: 10.0.0.42:8500- server: 10.0.0.41:8500relabel_configs:# 匹配consul的源标签字段，表示服务名称- source_labels: [__meta_consul_service]# 指定源标签的正则表达式，若不定义，默认值为"(.*)"regex: consul# 执行动作为删除，默认值为"replace",有效值有多种#   https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_actionaction: drop

2.2 重新加载配置

[root@prometheus-server33 ~]# curl -X POST http://10.0.0.33:9090/-/reload

2.3 被监控节点注册到console集群

[root@node-exporter42 ~]# curl -X PUT -d '{"id":"prometheus-node42","name":"dezyan-prometheus-node42","address":"10.0.0.42","port":9100,"tags":["node-exporter"],"checks": [{"http":"http://10.0.0.42:9100","interval":"5m"}]}' http://10.0.0.42:8500/v1/agent/service/register[root@node-exporter43 ~]# curl -X PUT -d '{"id":"prometheus-node42","name":"dezyan-prometheus-node42","address":"10.0.0.43","port":9100,"tags":["node-exporter"],"checks": [{"http":"http://10.0.0.43:9100","interval":"5m"}]}' http://10.0.0.42:8500/v1/agent/service/register

或者使用Postman这样的工具

PUT http://10.0.0.42:8500/v1/agent/service/register
{"id":"prometheus-node43","name":"dezyan-prometheus-node42","address":"10.0.0.42","port":9100,"tags":["node-exporter"],"checks": [{"http":"http://10.0.0.42:9100","interval":"5m"}]}PUT http://10.0.0.43:8500/v1/agent/service/register
{"id":"prometheus-node43","name":"dezyan-prometheus-node43","address":"10.0.0.43","port":9100,"tags":["node-exporter"],"checks": [{"http":"http://10.0.0.43:9100","interval":"5m"}]}

3.Prometheus-server31节点

3.1编辑配置文件

[root@prometheus-server31 ~]# cd /dezyan/softwares/prometheus-2.53.4.linux-amd64/
[root@prometheus-server31 ~]# vim prometheus.yml
remote_write:- url: http://10.0.0.43:8428/api/v1/write- job_name: "prometheus-federate-32"metrics_path: "/federate"# 用于解决标签的冲突问题，有效值为: true和false，默认值为false# 当设置为true时，将保留抓取的标签以忽略服务器自身的标签。说白了会覆盖原有标签。# 当设置为false时，则不会覆盖原有标签，而是在标点前加了一个"exported_"前缀。honor_labels: trueparams:"match[]":- '{job="promethues"}'- '{__name__=~"job:.*"}'- '{__name__=~"node.*"}'static_configs:- targets:- "10.0.0.32:9090"- job_name: "prometheus-federate-33"metrics_path: "/federate"honor_labels: trueparams:"match[]":- '{job="promethues"}'- '{__name__=~"job:.*"}'- '{__name__=~"node.*"}'static_configs:- targets:- "10.0.0.33:9090"

3.2 重新加载配置

[root@prometheus-server33 ~]# curl -X POST http://10.0.0.31:9090/-/reload

六、在VictoriaMetrics的WebUI查看数据

http://10.0.0.43:8428/

七、配置grafana的数据源及URL

数据源依旧是Prometheus
数据源来源的IP是http://10.0.0.43:8428

八、导入grafana的模板ID并选择数据源出图展示