flink restAPI及yarn命令

最新推荐文章于 2023-05-30 14:29:01 发布

wickedByhyk

最新推荐文章于 2023-05-30 14:29:01 发布

阅读量673

点赞数

分类专栏： yarn命令及restful Api 文章标签： flink yarn

本文链接：https://blog.csdn.net/qq_40844739/article/details/125786895

版权

yarn命令及restful Api 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

yarn 操作

source /bigdata
klist -kt /flink/user.keytab
kinit -kt /flink/user.keytab flink_test

查询applicaiton集合：

yarn application -list | grep default

-appStates:ALL/NEW/NEW_SAVING/SUBMITTED/ACCEPTED/RUNNING/FINISHED/FAILED/KILLED
-appTypes: MAPREDUCE/SPARK/FLINK

eg: yarn application -list -appStates ALL

查看application的attempts:

yarn applicationattempt -list

eg: yarn applicationattempt -list application_1653966005529_0021

查看指定attempt所申请创建的containers:

yarn container -list

eg: yarn container -list appattempt_1653966005529_0021_000001

一般containerId 后缀为_000001的是 jobmanager的container,其他为taskmanager的container(暂未看源码)

查看container的日志文件列表:

yarn logs -containerId -show_container_log_info

eg: yarn logs -containerId container_e613_1653966005529_0021_01_000001 -show_container_log_info

查看container的某个日志文件的具体内容:

yarn logs -containerId -log_files

eg: yarn logs -containerId container_e613_1653966005529_0021_01_000001 -log_files taskmanager.log

yarn restful API

获取Application的信息:

yarn.resourcemanager.webapp.https.adress
https:// $R M A d d r / w s / v 1 / c l u s t e r / a p p s /$ ApplicationID

eg: curl -k -i --negotiate -u : “https://192.168.22.3:8088/ws/v1/cluster/apps/application_1638870499459_0001”

https://RMAddr/ws/v1/cluster/apps?state=RUNNING&user=hdfs&queue=default&applicationType=Flink

eg: curl -k -i --negotiate -u : “https://192.168.22.3:8088/ws/v1/cluster/apps?state=RUNNING&user=hdfs&queue=default&applicationType=Flink”

获取Application的container信息:

yarn.nodemanager.webapp.https.adress
http:// $N M A d d r / w s / v 1 / n o d e / a p p s /$ ApplicationID

eg: curl -k -i --negotiate -u : “https://192.168.22.1:8042/ws/v1/node/apps/application_1638870499459_0001”

获取container的详细信息:

获取container的日志文件列表

eg: curl -k -i --negotiate -u : “https://192.168.22.1:8042/ws/v1/node/containers/container_e613_1653966005529_0025_01_000001”

获取日志的具体内容:

eg: curl -k -i --negotiate -u :
“https://192.168.22.1:8042/ws/v1/node/containers/container_e613_1653966005529_0025_01_000001/logs/taskmanager.log”

查看日志：

由yarn container -list 查出的LOG-URL拼接

000001：LOG-URL/jobmanager.log/?start=0
其他：LOG-URL/taskmanager.log/?start=0

flink restful API

https:// $R M A d d r / p r o x y /$ ApplicationID

curl -k -i --negotiate -u : “https://192.168.22.3:8088/proxy/application_1638870499459_0001/jobs”
curl -k -i --negotiate -u : “https://192.168.22.3:8088/proxy/application_1638870499459_0001/taskmanagers”
curl -k -i --negotiate -u : “https://192.168.22.3:8088/proxy/application_1638870499459_0001/jobs/jobid”
curl -k -i --negotiate -u : “https://192.168.22.3:8088/proxy/application_1638870499459_0001/jobs/overview”
curl -k -i --negotiate -u : “https://192.168.22.3:8088/proxy/application_1638870499459_0001/jobs/jobid/accumulators”
curl -k -i --negotiate -u : “https://192.168.22.3:8088/proxy/application_1638870499459_0001/jobs/jobid/exceptions”
curl -k -i --negotiate -u : “https://192.168.22.3:8088/proxy/application_1638870499459_0001/jobs/jobid/checkpoints”
curl -k -i --negotiate -u : “https://192.168.22.3:8088/proxy/application_1638870499459_0001/jobs/jobid/metrics”
curl -k -i --negotiate -u : “https://192.168.22.3:8088/proxy/application_1638870499459_0001/jobs/jobid/execution-result”
curl -k -i --negotiate -u : “https://192.168.22.3:8088/proxy/application_1638870499459_0001/jobs/jobid/vertices/verticesId/”
curl -k -i --negotiate -u : “https://192.168.22.3:8088/proxy/application_1638870499459_0001/jobs/jobid/vertices/verticesId/backpressure”
curl -k -i --negotiate -u : “https://192.168.22.3:8088/proxy/application_1638870499459_0001/jobs/jobid/vertices/verticesId/metrics”
curl -k -i --negotiate -u : “https://192.168.22.3:8088/proxy/application_1638870499459_0001/jobs/jobid/vertices/verticesId/metrics?get=0.currentInputWatermark”
curl -k -i --negotiate -u : “https://192.168.22.3:8088/proxy/application_1638870499459_0001/jobs/jobid/vertices/verticesId/subtasks/metrics”
curl -k -i --negotiate -u : “https://192.168.22.3:8088/proxy/application_1638870499459_0001/jobs/jobid/vertices/verticesId/subtasks/index/metrics”
eg: curl -k -i --negotiate -u : “https://192.168.22.3:8088/proxy/application_1638870499459_0001/jobs/jobid/vertices/verticesId/subtasks/0/metrics”