demo-Prometheus+PushGateway+Grafana监控Flink on YARN作业

Flink 专栏收录该内容
11 篇文章 0 订阅

简述

flink metric用来对外暴露系统内部的一些运行指标,比如flink框架运行时的JVM相关配置,或者基于flink开发的项目。flink提供了Counter, Gauge, Histogram and Meter四种类型的指标。我们通过继承RichFunction拿到MetricGroup,并向其中填充指标。今天主要讲的是使用Prometheus+PushGateway+Grafana监控Flink on YARN作业

[hadoop@hadoop001 lib]$ pwd
/home/hadoop/app/flink/lib
[hadoop@hadoop001 lib]$ ll
-rw-r--r-- 1 hadoop hadoop    103761 Apr 28 15:38 flink-metrics-prometheus-1.9.1.jar

# 配置flink-conf.yaml,进入/home/hadoop/app/flink/conf目录
[hadoop@hadoop001 conf]$ pwd
/home/hadoop/app/flink/conf

metrics.reporter.promgateway.class: org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter
# 这里写PushGateway的主机名与端口号
metrics.reporter.promgateway.host: hadoop001
metrics.reporter.promgateway.port: 9091
# Flink metric在前端展示的标签(前缀)与随机后缀
metrics.reporter.promgateway.jobName: flink-metrics
metrics.reporter.promgateway.randomJobNameSuffix: true
metrics.reporter.promgateway.deleteOnShutdown: false

wget https://github.com/prometheus/prometheus/releases/download/v2.18.0-rc.0/prometheus-2.18.0-rc.0.linux-amd64.tar.gz
# 注意:pushgateway 版本有可能与flink不兼容 ,请自行选择合适的版本
wget https://github.com/prometheus/pushgateway/releases/download/v1.2.0/pushgateway-1.2.0.linux-amd64.tar.gz

  • 3.配置 prometheus 的prometheus.yml文件 scrape_configs
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    static_configs:
    - targets: ['hadoop001:9090']
    #注意格式 空格
  - job_name: 'pushgateway'
    static_configs:
      - targets: ['hadoop001:9091']
        labels:
          instance: 'pushgateway'

  • 4.启动 页面查看是否启动成功 ,监控
nohup ./pushgateway --web.listen-address :9091 > /var/log/pushgateway.log 2>&1 &

nohup ./prometheus --config.file=prometheus.yml > /var/log/prometheus.log 2>&1 &
# 端口号的查看
	netstat -apn | grep -E '9091|3000|9090'

在这里插入图片描述
在这里插入图片描述

wget https://dl.grafana.com/oss/release/grafana-6.7.3.linux-amd64.tar.gz
tar -zxvf grafana-6.7.3.linux-amd64.tar.gz

添加数据源 ,指定要监控的指标 以及添加相关报警规则等
在这里插入图片描述

  • 6.运行flink example ,查看是否监控成功
nc -l 19999
./bin/flink run -m yarn-cluster -p 4 -yjm 1024m -ytm 1024m --yarnname "window-wordcount" ./examples/streaming/SocketWindowWordCount.jar --hostname hadoop001 --port 19999
  • 0
    点赞
  • 0
    评论
  • 1
    收藏
  • 一键三连
    一键三连
  • 扫一扫,分享海报

<p> <span style="font-size:16px;"></span><span style="font-size:20px;"><strong>一、prometheus简介</strong></span> </p> <p> <span style="font-size:16px;"><strong> </strong></span> </p> <p> <strong> Prometheus是一个开源的系统监控和告警系统,现在已经加入到CNCF基金会,成为继k8s之后第二个在CNCF维护管理的项目,在kubernetes容器管理系统中,通常会搭配prometheus进行监控prometheus支持多种exporter采集数据,还支持通过pushgateway进行数据上报,<span style="color:#494949;">Prometheus再</span><span style="color:#494949;">性能上可支撑上万台规模的集群。</span> </strong> </p> <strong> </strong> <p> <strong> <span style="font-size:20px;"><strong><span style="color:#494949;">二、prometheus架构图</span></strong></span> </strong> </p> <strong> </strong> <p> <strong><span style="color:#494949;"><img src="https://img-bss.csdn.net/202004201352387874.png" alt="" /><br /></span> </strong> </p> <strong> </strong> <p> <strong> <span style="font-size:20px;"><strong>三、prometheus组件介绍</strong></span> </strong> </p> <strong> </strong> <p style="text-align:left;"> <strong> <span style="font-size:16px;color:#323232;"></span> </strong> </p> <strong> </strong> <p style="text-align:left;"> <strong> <span style="font-size:16px;color:#323232;">1.Prometheus Server: 用于收集和存储时间序列数据。</span> </strong> </p> <strong> </strong> <p style="text-align:left;"> <strong> <span style="font-size:16px;color:#323232;">2.Client Library: 客户端库,检测应用程序代码,当Prometheus抓取实例的HTTP端点时,客户端库会将所有跟踪的metrics指标的当前状态发送到prometheus server端。</span> </strong> </p> <strong> </strong> <p style="text-align:left;"> <strong> <span style="font-size:16px;color:#323232;">3.Exporters: prometheus支持多种exporter,通过exporter可以采集metrics数据,然后发送到prometheus server端</span> </strong> </p> <strong> </strong> <p style="text-align:left;"> <strong> <span style="font-size:16px;color:#323232;">4.Alertmanager: 从 Prometheus server 端接收到 alerts 后,会进行去重,分组,并路由到相应的接收方,发出报警,常见的接收方式有:电子邮件,微信,钉钉, slack等。</span> </strong> </p> <strong> </strong> <p style="text-align:left;"> <strong> <span style="font-size:16px;color:#323232;">5.Grafana监控仪表盘</span> </strong> </p> <strong> </strong> <p style="text-align:left;"> <strong> <span style="font-size:16px;color:#323232;">6.pushgateway: 各个目标主机可上报数据到pushgatewy,然后prometheus server统一从pushgateway拉取数据。</span> </strong> </p> <strong> </strong> <p> <strong> <span style="font-size:20px;"><strong><span></span></strong></span> </strong> </p> <strong> </strong> <p> <strong> <span style="font-size:20px;"><strong><span>四、课程亮点</span></strong></span><strong><span style="font-size:16px;"><br /></span></strong> </strong> </p> <strong> </strong> <p> <img src="https://img-bss.csdn.net/202004201352522795.png" alt="" /></p> <strong> </strong> <p> <strong> <strong><span style="font-size:16px;"></span></strong> </strong> </p> <strong> </strong> <p> <strong> <span style="font-size:20px;"><strong><span></span></strong></span><span style="font-size:20px;"><strong><span>五、效果图展示</span></strong></span></strong> </p> <p> <strong><span style="font-size:20px;"><strong><span><img src="https://img-bss.csdn.net/202004201353077268.png" alt="" /><br /></span></strong></span></strong> </p> <strong> </strong> <p> <strong><span style="font-size:20px;"><strong><img src="https://img-bss.csdn.net/202004201353245111.png" alt="" /><br /></strong></span></strong> </p> <p> <strong><span style="font-size:20px;"><strong><img src="https://img-bss.csdn.net/202004201353384468.png" alt="" /><br /></strong></span></strong> </p> <p> <strong><span style="font-size:20px;"><strong><img src="https://img-bss.csdn.net/202004201354069425.png" alt="" /><br /></strong></span></strong> </p> <p> <strong><span style="font-size:20px;"><strong><img src="https://img-bss.csdn.net/202004201354193720.png" alt="" /><br /></strong></span></strong> </p> <p> <strong> <span style="font-size:20px;"><strong>六、讲师简介</strong></span> </strong> </p> <strong> </strong> <p> <strong> <span style="font-size:16px;"><span style="color:#000000;">先超(lucky)</span><span style="color:#000000;">:</span></span><span style="color:#000000;font-size:16px;">高级运维工程师、资深DevOps工程师,在互联网上市公司拥有多年一线运维经验,主导过亿级pv项目的架构设计和运维工作</span><span style="color:#000000;font-size:16px;"><br /></span> </strong> </p> <strong> </strong> <p> <strong> <span style="font-size:16px;"><span style="color:#000000;font-size:16px;">主要研究方向:</span><span style="color:#000000;font-size:16px;"><br /></span></span> </strong> </p> <strong> </strong> <p> <strong> <span style="font-size:16px;"><span style="color:#000000;font-size:16px;">1.云计算方向:</span><span style="font-size:40px;color:#000000;"></span><span style="color:#000000;font-size:16px;">容器</span></span> </strong> </p> <strong> </strong> <p> <strong> <span style="font-size:16px;"><span style="color:#000000;font-size:16px;">(kubernetes、docker),虚拟化(kvm、Vmware vSphere),微服务(istio),PaaS(openshift),IaaS(openstack)等</span><span style="font-size:27px;color:#000000;"><br /></span><span style="color:#000000;font-size:16px;">2.系统/运维方向:</span><span style="font-size:40px;color:#000000;"></span><span style="font-size:39px;color:#000000;"></span><span style="color:#000000;font-size:16px;">linux系统下的常用组件(nginx,tomcat,elasticsearch,zookeeper,kafka等),DevOps(Jenkins+gitlab+sonarqube+nexus+k8s),CI/CD,监控(zabbix、prometheusfalcon)等</span><span style="font-size:39px;color:#000000;"><br /></span></span> </strong> </p> <strong> </strong> <p> <strong> <span style="font-size:39px;color:#000000;"></span> </strong> </p> <strong></strong><strong></strong><strong></strong> <p> <strong><span style="font-size:18px;">七、课程大纲</span></strong> </p> <p> <strong><span style="font-size:18px;"><img src="https://img-bss.csdn.net/202004201355254800.png" alt="" /><br /></span></strong> </p> <p> <strong><span style="font-size:24px;"><img src="https://img-bss.csdn.net/202004201355386613.png" alt="" /><br /></span></strong> </p> <p> <strong><span style="font-size:24px;"><img src="https://img-bss.csdn.net/202004201356062136.png" alt="" /><br /></span></strong> </p> <p> <strong><span style="font-size:24px;"><img src="https://img-bss.csdn.net/202004201356218012.png" alt="" /><br /></span></strong> </p>
©️2021 CSDN 皮肤主题: 大白 设计师:CSDN官方博客 返回首页
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、C币套餐、付费专栏及课程。

余额充值