Flink DataStream RestAPI使用

最新推荐文章于 2024-10-23 17:14:40 发布
最新推荐文章于 2024-10-23 17:14:40 发布
阅读量4k 收藏 6
点赞数 1
分类专栏： Flink 文章标签： Flink Rest API
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/wangpei1949/article/details/100853748
版权

文章目录


基于 Yarn Session模式总结 Rest API在DataStream Job上的使用。

注: 也适用于Yarn Per Job模式。

RestAPI端口设置

可通过flink-conf.yaml中的rest.bind-port参数设置。
注意:

  1. Rest Server和Dashboard Server监听在同一服务器(Active JobManager所在机器)、同一端口。通过不同的URL响应。

  2. rest.bind-port不设置，则Rest Server默认绑定到rest.port端口(8081)。

  3. rest.bind-port可以设置成列表格式如50100,50101，也可设置成范围格式如50100-50200。推荐范围格式，避免端口冲突。

RestAPI使用

注意:

  1. Rest API对Backpressure的监控已经过时。

1. 启动Yarn Session集群

bin/yarn-session.sh --container 3 --slots 2 --queue default --taskManagerMemory 1024 --yarndetached

启动后，在JobManager中可见如下日志:
2019-09-14 13:47:27,163 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Web frontend listening at http://node2:50100.

由日志可知，Rest Server正监听在node2的50100端口。

2. 关闭集群

API: DELETE /cluster

请求:
    curl -X DELETE http://node2:50100/v1/cluster|jq
返回:
    {}

3. 查看集群概览

可以看到当前集群运行的Job数、取消的Job数等等。

API: GET /overview

请求:
    curl -X GET http://node2:50100/v1/overview|jq
返回:
    {
      "taskmanagers": 0,
      "slots-total": 0,
      "slots-available": 0,
      "jobs-running": 1,
      "jobs-finished": 0,
      "jobs-cancelled": 6,
      "jobs-failed": 0,
      "flink-version": "1.9.0",
      "flink-commit": "9c32ed9"
    }

4. 查看Dastboard页面

http://node2:50100/#/overview

5. 查看JobManager上的相关配置

API：GET /jobmanager/config

请求:
    curl -X GET http://node2:50100/v1/jobmanager/config|jq
返回:
    [
      {
        "key": "historyserver.web.address",
        "value": "node2"
      },
      {
        "key": "state.checkpoints.num-retained",
        "value": "20"
      },
      {
        "key": "historyserver.web.port",
        "value": "18082"
      },
      {
        "key": "jobmanager.execution.failover-strategy",
        "value": "region"
      },
      {
        "key": "high-availability.cluster-id",
        "value": "application_1568023533160_0015"
      },
      {
        "key": "jobmanager.rpc.address",
        "value": "node2"
      },
      {
        "key": "FLINK_PLUGINS_DIR",
        "value": "/data/software/flink-1.9.0/plugins"
      },
      {
        "key": "high-availability.zookeeper.path.root",
        "value": "/flink"
      },
      {
        "key": "high-availability.storageDir",
        "value": "hdfs://node1:8020/flink/high_availability"
      },
      {
        "key": "rest.bind-port",
        "value": "50100-50200"
      },
      {
        "key": "io.tmp.dirs",
        "value": "/hadoop/yarn/local/usercache/root/appcache/application_1568023533160_0015"
      },
      {
        "key": "parallelism.default",
        "value": "1"
      },
      {
        "key": "yarn.application-attempts",
        "value": "2"
      },
      {
        "key": "taskmanager.numberOfTaskSlots",
        "value": "2"
      },
      {
        "key": "zookeeper.sasl.disable",
        "value": "true"
      },
      {
        "key": "historyserver.archive.fs.dir",
        "value": "hdfs:///completed-jobs"
      },
      {
        "key": "jobmanager.heap.size",
        "value": "1024m"
      },
      {
        "key": "jobmanager.archive.fs.dir",
        "value": "hdfs:///completed-jobs"
      },
      {
        "key": "web.port",
        "value": "0"
      },
      {
        "key": "classloader.resolve-order",
        "value": "parent-first"
      },
      {
        "key": "web.tmpdir",
        "value": "/tmp/flink-web-951aa23b-c01a-4026-a154-602538873f1d"
      },
      {
        "key": "jobmanager.rpc.port",
        "value": "37294"
      },
      {
        "key": "internal.io.tmpdirs.use-local-default",
        "value": "true"
      },
      {
        "key": "rest.port",
        "value": "8081"
      },
      {
        "key": "high-availability.zookeeper.quorum",
        "value": "node1:2181,node2:2181,node3:2181"
      },
      {
        "key": "internal.cluster.execution-mode",
        "value": "NORMAL"
      },
      {
        "key": "high-availability",
        "value": "zookeeper"
      },
      {
        "key": "rest.address",
        "value": "node2"
      },
      {
        "key": "taskmanager.heap.size",
        "value": "1024m"
      }
    ]

6. 查看JobManager Metrics

API：GET /jobmanager/metrics
    参数:
        A、get: 逗号分隔的Metrics列表。如taskSlotsAvailable,taskSlotsTotal,Status.JVM.Memory.Heap.Used。

请求:
    curl -X GET http://node2:50100/v1/jobmanager/metrics?get=taskSlotsAvailable,taskSlotsTotal,Status.JVM.Memory.Heap.Used|jq
返回:
    [
      {
        "id": "taskSlotsAvailable",
        "value": "0"
      },
      {
        "id": "taskSlotsTotal",
        "value": "2"
      },
      {
        "id": "Status.JVM.Memory.Heap.Used",
        "value": "62225192"
      }
    ]

7. 查看所有TaskManager

API: GET /taskmanagers
    
请求:
    curl -X GET http://node2:50100/v1/taskmanagers|jq
返回:   
    {
      "taskmanagers": [
        {
          "id": "container_e35_1568023533160_0015_01_000007",
          "path": "akka.tcp://flink@node5:45124/user/taskmanager_0",
          "dataPort": 41687,
          "timeSinceLastHeartbeat": 1568519539011,
          "slotsNumber": 2,
          "freeSlots": 0,
          "hardware": {
            "cpuCores": 6,
            "physicalMemory": 12131536896,
            "freeMemory": 362283008,
            "managedMemory": 264241152
          }
        }
      ]
    }

8. 查看某个TaskManager

API: GET /taskmanagers/:taskmanagerid
    参数: 
        A、taskmanagerid: /taskmanagers接口返回的id。

请求:
    curl -X GET http://node2:50100/v1/taskmanagers/container_e35_1568023533160_0015_01_000007|jq
返回:
    {
      "id": "container_e35_1568023533160_0015_01_000007",
      "path": "akka.tcp://flink@node5:45124/user/taskmanager_0",
      "dataPort": 41687,
      "timeSinceLastHeartbeat": 1568519639213,
      "slotsNumber": 2,
      "freeSlots": 0,
      "hardware": {
        "cpuCores": 6,
        "physicalMemory": 12131536896,
        "freeMemory": 362283008,
        "managedMemory": 264241152
      },
      "metrics": {
        "heapUsed": 78424048,
        "heapCommitted": 361234432,
        "heapMax": 361234432,
        "nonHeapUsed": 86401760,
        "nonHeapCommitted": 88735744,
        "nonHeapMax": -1,
        "directCount": 2206,
        "directUsed": 67839321,
        "directMax": 67839319,
        "mappedCount": 0,
        "mappedUsed": 0,
        "mappedMax": 0,
        "memorySegmentsAvailable": 2044,
        "memorySegmentsTotal": 2048,
        "garbageCollectors": [
          {
            "name": "PS_Scavenge",
            "count": 9,
            "time": 116
          },
          {
            "name": "PS_MarkSweep",
            "count": 3,
            "time": 159
          }
        ]
      }
    }

9. 查看某个TaskManager Metrics

API: GET /taskmanagers/:taskmanagerid/metrics
    参数:
        A、get: 逗号分隔的Metrics列表。如taskSlotsAvailable,taskSlotsTotal,Status.JVM.Memory.Heap.Used。
        
请求:
    curl -X GET http://node2:50100/v1/taskmanagers/container_e35_1568023533160_0015_01_000007/metrics?get=Status.JVM.Memory.Heap.Used,Status.JVM.Memory.NonHeap.Used,Status.JVM.Memory.NonHeap.Max|jq
返回:
    [
      {
        "id": "Status.JVM.Memory.NonHeap.Used",
        "value": "101535976"
      },
      {
        "id": "Status.JVM.Memory.Heap.Used",
        "value": "101347576"
      },
      {
        "id": "Status.JVM.Memory.NonHeap.Max",
        "value": "-1"
      }
    ]

10. 上传Jar包

API: POST /jars/upload
    参数:
        A、jarfile: Jar包路径。
    
请求:
    curl -X POST -H "Expect:" -F "jarfile=@/data/apps/flink-read-write-kafka-jar-with-dependencies.jar" http://node2:50100/v1/jars/upload|jq
返回:
    {
      "filename": "/tmp/flink-web-951aa23b-c01a-4026-a154-602538873f1d/flink-web-upload/74ae26fe-7973-4998-8594-23d4e357c577_flink-read-write-kafka-jar-with-dependencies.jar",
      "status": "success"
    }

11. 查看所有Jar包

API: GET /jars

请求:
    curl -X GET http://node2:50100/v1/jars|jq
返回:
    {
      "address": "http://node2:50100",
      "files": [
        {
          #Jar唯一标识
          "id": "74ae26fe-7973-4998-8594-23d4e357c577_flink-read-write-kafka-jar-with-dependencies.jar",
          "name": "flink-read-write-kafka-jar-with-dependencies.jar",
          "uploaded": 1568469312000,
          "entry": []
        }
      ]
    }

12. 删除某个Jar包

API: DELETE  /jars/:jarid
    参数:
        jarid: Jar唯一标识。

请求:
    curl -X DELETE  http://node2:50100/v1/jars/74ae26fe-7973-4998-8594-23d4e357c577_flink-read-write-kafka-jar-with-dependencies.jar|jq
返回:
    {}

13. 根据输入参数查看Job Dataflow Plan

支持Query Parameter和JSON Request两种请求方式。这里，只总结JSON Request方式(推荐)。

API: GET /jars/:jarid/plan
    参数:
        A、jarid: Jar唯一标识。
        B、programArgsList: 应用程序参数列表。
        C、entryClass: 入口类。
        D、parallelism: 并行度。
    
请求:
    curl -X GET -H 'Content-Type: application/json' --data '
    {
        "programArgsList": [
            "--fromKafkaBrokers",
            "node1:6667,node2:6667,node3:6667",
            "--fromKafkaGroup",
            "c4",
            "--fromKafkaTopic",
            "userActionLog",
            "--toKafkaBrokers",
            "node5:9092",
            "--toKafkaTopic",
            "realtime_dashboard"
        ],
        "entryClass": "com.bigdata.flink.ReadWriteKafka",
        "parallelism": 2
    }
    ' http://node2:50100/v1/jars/7184ba9f-b160-439c-b3cb-bd2fa61ca704_flink-read-write-kafka-jar-with-dependencies.jar/plan|jq
返回:
    {
      "plan": {
        "jid": "06b5f5d9a56646f54721b4421b59f811",
        "name": "",
        "nodes": [
          {
            "id": "09078a2a0d570fc10e5b089eeccaaa4b",
            "parallelism": 2,
            "operator": "",
            "operator_strategy": "",
            "description": "Window(TumblingEventTimeWindows(5000), EventTimeTrigger, CustomAggFunction, CustomWindowFunction) -> (Sink: Print to Std. Out, Sink: outputToKafka)",
            "inputs": [
              {
                "num": 0,
                "id": "bd9c186e89cc37a76d776844dfa8ccba",
                "ship_strategy": "HASH",
                "exchange": "pipelined_bounded"
              }
            ],
            "optimizer_properties": {}
          },
          {
            "id": "bd9c186e89cc37a76d776844dfa8ccba",
            "parallelism": 2,
            "operator": "",
            "operator_strategy": "",
            "description": "Source: KafkaSource -> Map: ExtractTransform -> Filter: FilterExceptionData -> Timestamps/Watermarks",
            "optimizer_properties": {}
          }
        ]
      }
    }

14. 向集群提交新Job或从状态恢复Job

支持Query Parameter和JSON Request两种请求方式。这里，只总结JSON Request方式(推荐)。

API: POST /jars/:jarid/run
    参数:
        A、jarid: Jar唯一标识。
        B、programArgsList: 应用程序参数列表。
        C、entryClass: 入口类。
        D、parallelism: 并行度。
        E、allowNonRestoredState: 状态不一致时，是否允许提交。
        F、savepointPath: savepoint路径。
    
请求:
    curl -X POST -H 'Content-Type: application/json' --data '
    {
        "programArgsList": [
            "--fromKafkaBrokers",
            "node1:6667,node2:6667,node3:6667",
            "--fromKafkaGroup",
            "c4",
            "--fromKafkaTopic",
            "userActionLog",
            "--toKafkaBrokers",
            "node5:9092",
            "--toKafkaTopic",
            "realtime_dashboard"
        ],
        "entryClass": "com.bigdata.flink.ReadWriteKafka",
        "parallelism": 2,
        "allowNonRestoredState":true,
        "savepointPath":""
    }
    ' http://node2:50100/v1/jars/7184ba9f-b160-439c-b3cb-bd2fa61ca704_flink-read-write-kafka-jar-with-dependencies.jar/run|jq
    
返回:
    {
      "jobid": "a5e6a799c9b358817e8a5d21534d6948"
    }

15. 查看所有Job

API：GET /jobs

请求:
    curl -X GET http://node2:50100/v1/jobs|jq
返回:
    {
      "jobs": [
        {
          "id": "a5e6a799c9b358817e8a5d21534d6948",
          "status": "RUNNING"
        },
        {
          "id": "33b12aef7b474f306e59938ac4d9c32e",
          "status": "CANCELED"
        },
        {
          "id": "19fe28e89dbd5550b4c3510894204027",
          "status": "CANCELED"
        },
        {
          "id": "1ecad592ae428d685b6f8c7c8bec362a",
          "status": "CANCELED"
        }
      ]
    }

16. 取消某个Job

API：PATCH /jobs/:jobid

请求:
    curl -X PATCH http://node2:50100/v1/jobs/a5e6a799c9b358817e8a5d21534d6948|jq
返回:
    {}

17. 查看某个Job详细信息

API：GET /jobs/:jobid

请求:
    curl -X GET http://node2:50100/v1/jobs/a5e6a799c9b358817e8a5d21534d6948|jq
返回:
    {
      "jid": "a5e6a799c9b358817e8a5d21534d6948",
      "name": "",
      "isStoppable": false,
      "state": "RUNNING",
      "start-time": 1568513921599,
      "end-time": -1,
      "duration": 1454068,
      "now": 1568515375667,
      "timestamps": {
        "FAILING": 0,
        "CANCELED": 0,
        "FINISHED": 0,
        "SUSPENDED": 0,
        "CREATED": 1568513921599,
        "FAILED": 0,
        "RECONCILING": 0,
        "RESTARTING": 0,
        "RUNNING": 1568513921638,
        "CANCELLING": 0
      },
      "vertices": [
        {
          "id": "bd9c186e89cc37a76d776844dfa8ccba",
          "name": "Source: KafkaSource -> Map: ExtractTransform -> Filter: FilterExceptionData -> Timestamps/Watermarks",
          "parallelism": 2,
          "status": "RUNNING",
          "start-time": 1568515358914,
          "end-time": -1,
          "duration": 16753,
          "tasks": {
            "CREATED": 0,
            "RUNNING": 2,
            "FINISHED": 0,
            "SCHEDULED": 0,
            "FAILED": 0,
            "RECONCILING": 0,
            "DEPLOYING": 0,
            "CANCELING": 0,
            "CANCELED": 0
          },
          "metrics": {
            "read-bytes": 0,
            "read-bytes-complete": true,
            "write-bytes": 0,
            "write-bytes-complete": true,
            "read-records": 0,
            "read-records-complete": true,
            "write-records": 605,
            "write-records-complete": true
          }
        },
        {
          "id": "09078a2a0d570fc10e5b089eeccaaa4b",
          "name": "Window(TumblingEventTimeWindows(5000), EventTimeTrigger, CustomAggFunction, CustomWindowFunction) -> (Sink: Print to Std. Out, Sink: outputToKafka)",
          "parallelism": 2,
          "status": "RUNNING",
          "start-time": 1568515358914,
          "end-time": -1,
          "duration": 16753,
          "tasks": {
            "CREATED": 0,
            "RUNNING": 2,
            "FINISHED": 0,
            "SCHEDULED": 0,
            "FAILED": 0,
            "RECONCILING": 0,
            "DEPLOYING": 0,
            "CANCELING": 0,
            "CANCELED": 0
          },
          "metrics": {
            "read-bytes": 30983,
            "read-bytes-complete": true,
            "write-bytes": 0,
            "write-bytes-complete": true,
            "read-records": 605,
            "read-records-complete": true,
            "write-records": 0,
            "write-records-complete": true
          }
        }
      ],
      "status-counts": {
        "CREATED": 0,
        "RUNNING": 2,
        "FINISHED": 0,
        "SCHEDULED": 0,
        "FAILED": 0,
        "RECONCILING": 0,
        "DEPLOYING": 0,
        "CANCELING": 0,
        "CANCELED": 0
      },
      "plan": {
        "jid": "a5e6a799c9b358817e8a5d21534d6948",
        "name": "",
        "nodes": [
          {
            "id": "09078a2a0d570fc10e5b089eeccaaa4b",
            "parallelism": 2,
            "operator": "",
            "operator_strategy": "",
            "description": "Window(TumblingEventTimeWindows(5000), EventTimeTrigger, CustomAggFunction, CustomWindowFunction) -> (Sink: Print to Std. Out, Sink: outputToKafka)",
            "inputs": [
              {
                "num": 0,
                "id": "bd9c186e89cc37a76d776844dfa8ccba",
                "ship_strategy": "HASH",
                "exchange": "pipelined_bounded"
              }
            ],
            "optimizer_properties": {}
          },
          {
            "id": "bd9c186e89cc37a76d776844dfa8ccba",
            "parallelism": 2,
            "operator": "",
            "operator_strategy": "",
            "description": "Source: KafkaSource -> Map: ExtractTransform -> Filter: FilterExceptionData -> Timestamps/Watermarks",
            "optimizer_properties": {}
          }
        ]
      }
    }

18. 查看Job Dataflow Plan

API: GET /jobs/:jobid/plan

请求:
    curl -X GET http://node2:50100/v1/jobs/12c848465ef964bba9d1b1ddf558570a/plan|jq
返回:
    {
      "plan": {
        "jid": "12c848465ef964bba9d1b1ddf558570a",
        "name": "",
        "nodes": [
          {
            "id": "09078a2a0d570fc10e5b089eeccaaa4b",
            "parallelism": 2,
            "operator": "",
            "operator_strategy": "",
            "description": "Window(TumblingEventTimeWindows(5000), EventTimeTrigger, CustomAggFunction, CustomWindowFunction) -> (Sink: Print to Std. Out, Sink: outputToKafka)",
            "inputs": [
              {
                "num": 0,
                "id": "bd9c186e89cc37a76d776844dfa8ccba",
                "ship_strategy": "HASH",
                "exchange": "pipelined_bounded"
              }
            ],
            "optimizer_properties": {}
          },
          {
            "id": "bd9c186e89cc37a76d776844dfa8ccba",
            "parallelism": 2,
            "operator": "",
            "operator_strategy": "",
            "description": "Source: KafkaSource -> Map: ExtractTransform -> Filter: FilterExceptionData -> Timestamps/Watermarks",
            "optimizer_properties": {}
          }
        ]
      }
    }

19. 查看Job Metrics

API: GET /jobs/:jobid/metrics
    参数:
        get: 逗号分隔的metrics列表，如要看uptime,lastCheckpointExternalPath。

请求:
    curl -X GET http://node2:50100/v1/jobs/12c848465ef964bba9d1b1ddf558570a/metrics?get=uptime,lastCheckpointExternalPath|jq
返回:
    [
      {
        "id": "lastCheckpointExternalPath",
        "value": "hdfs://node1:8020/flink/checkpoint/12c848465ef964bba9d1b1ddf558570a/chk-143"
      },
      {
        "id": "uptime",
        "value": "1437983"
      }
    ]

20. 查看Job中不可恢复的异常

API: GET /jobs/:jobid/exceptions

请求:
    curl -X GET http://node2:50100/v1/jobs/12c848465ef964bba9d1b1ddf558570a/exceptions|jq
返回:
    {
      "root-exception": null,
      "timestamp": null,
      "all-exceptions": [],
      "truncated": false
    }

21. 查看Job Checkpoint配置

API: GET /jobs/:jobid/checkpoints/config

请求:
    curl -X GET http://node2:50100/v1/jobs/12c848465ef964bba9d1b1ddf558570a/checkpoints/config|jq
返回:
    {
      "mode": "exactly_once",
      "interval": 10000,
      "timeout": 600000,
      "min_pause": 0,
      "max_concurrent": 1,
      "externalization": {
        "enabled": true,
        "delete_on_cancellation": false
      }
    }

22. 查看Job中的Checkpoint

API: GET /jobs/:jobid/checkpoints

请求:
    curl -X GET http://node2:50100/v1/jobs/12c848465ef964bba9d1b1ddf558570a/checkpoints|jq
返回:
    {
      "counts": {
        "restored": 0,
        "total": 2,
        "in_progress": 0,
        "completed": 2,
        "failed": 0
      },
      "summary": {
        "state_size": {
          "min": 13518,
          "max": 13640,
          "avg": 13579
        },
        "end_to_end_duration": {
          "min": 456,
          "max": 467,
          "avg": 461
        },
        "alignment_buffered": {
          "min": 0,
          "max": 0,
          "avg": 0
        }
      },
      "latest": {
        "completed": {
          "@class": "completed",
          "id": 2,
          "status": "COMPLETED",
          "is_savepoint": false,
          "trigger_timestamp": 1568516147296,
          "latest_ack_timestamp": 1568516147763,
          "state_size": 13640,
          "end_to_end_duration": 467,
          "alignment_buffered": 0,
          "num_subtasks": 4,
          "num_acknowledged_subtasks": 4,
          "tasks": {},
          "external_path": "hdfs://node1:8020/flink/checkpoint/12c848465ef964bba9d1b1ddf558570a/chk-2",
          "discarded": false
        },
        "savepoint": null,
        "failed": null,
        "restored": null
      },
      "history": [
        {
          "@class": "completed",
          "id": 2,
          "status": "COMPLETED",
          "is_savepoint": false,
          "trigger_timestamp": 1568516147296,
          "latest_ack_timestamp": 1568516147763,
          "state_size": 13640,
          "end_to_end_duration": 467,
          "alignment_buffered": 0,
          "num_subtasks": 4,
          "num_acknowledged_subtasks": 4,
          "tasks": {},
          "external_path": "hdfs://node1:8020/flink/checkpoint/12c848465ef964bba9d1b1ddf558570a/chk-2",
          "discarded": false
        },
        {
          "@class": "completed",
          "id": 1,
          "status": "COMPLETED",
          "is_savepoint": false,
          "trigger_timestamp": 1568516137296,
          "latest_ack_timestamp": 1568516137752,
          "state_size": 13518,
          "end_to_end_duration": 456,
          "alignment_buffered": 0,
          "num_subtasks": 4,
          "num_acknowledged_subtasks": 4,
          "tasks": {},
          "external_path": "hdfs://node1:8020/flink/checkpoint/12c848465ef964bba9d1b1ddf558570a/chk-1",
          "discarded": false
        }
      ]
    }

23. 查看Job某次Checkpoint详细信息

API: GET /jobs/:jobid/checkpoints/details/:checkpointid

请求:
    curl -X GET http://node2:50100/v1/jobs/12c848465ef964bba9d1b1ddf558570a/checkpoints/details/33|jq
返回:
    {
      "@class": "completed",
      "id": 33,
      "status": "COMPLETED",
      "is_savepoint": false,
      "trigger_timestamp": 1568516457296,
      "latest_ack_timestamp": 1568516458382,
      "state_size": 13700,
      "end_to_end_duration": 1086,
      "alignment_buffered": 0,
      "num_subtasks": 4,
      "num_acknowledged_subtasks": 4,
      "tasks": {
        "bd9c186e89cc37a76d776844dfa8ccba": {
          "id": 33,
          "status": "COMPLETED",
          "latest_ack_timestamp": 1568516457349,
          "state_size": 2508,
          "end_to_end_duration": 53,
          "alignment_buffered": 0,
          "num_subtasks": 2,
          "num_acknowledged_subtasks": 2
        },
        "09078a2a0d570fc10e5b089eeccaaa4b": {
          "id": 33,
          "status": "COMPLETED",
          "latest_ack_timestamp": 1568516458382,
          "state_size": 11192,
          "end_to_end_duration": 1086,
          "alignment_buffered": 0,
          "num_subtasks": 2,
          "num_acknowledged_subtasks": 2
        }
      },
      "external_path": "hdfs://node1:8020/flink/checkpoint/12c848465ef964bba9d1b1ddf558570a/chk-33",
      "discarded": false
    }

24. 触发Savepoint然后取消Job

API: POST /jobs/:jobid/savepoints

请求:
    curl -X POST -H 'Content-Type: application/json' --data '
    {
        "target-directory":"hdfs://node1:8020/flink/savepoint/",
        "cancel-job":true
    }
    ' http://node2:50100/v1/jobs/12c848465ef964bba9d1b1ddf558570a/savepoints|jq
返回:
    {
      "request-id": "7cc8e3cf54cc9d3c3465102ce2e44ef0"
    }

25. 查看触发的Savepoint状态

API: GET /jobs/:jobid/savepoints/:triggerid
    参数:
        A、jobid: Job ID。
        B、triggerid: /jobs/:jobid/savepoints接口返回的request-id。

请求:
    curl -X GET http://node2:50100/v1/jobs/12c848465ef964bba9d1b1ddf558570a/savepoints/7cc8e3cf54cc9d3c3465102ce2e44ef0|jq
返回:
    {
      "status": {
        "id": "COMPLETED"
      },
      "operation": {
        "location": "hdfs://node1:8020/flink/savepoint/savepoint-12c848-69571045e0b3"
      }
    }

26. 查看Job中的累加器

API: GET /jobs/:jobid/accumulators

请求:
    curl -X GET http://node2:50100/v1/jobs/dc68ddeffded6119d7395c814b9aff17/accumulators|jq
返回:
    {
      "job-accumulators": [],
      "user-task-accumulators": [],
      "serialized-user-task-accumulators": {}
    }

27. 查看Job某个Vertex所有Subtasks信息

API: GET /jobs/:jobid/vertices/:vertexid
    参数:
        A、vertexid: jobs/jobid接口vertices中的id
请求:
    curl -X GET http://node2:50100/v1/jobs/ff9d383001652ab54e4970965fa386ee/vertices/bd9c186e89cc37a76d776844dfa8ccba|jq
返回:
    {
      "id": "bd9c186e89cc37a76d776844dfa8ccba",
      "name": "Source: KafkaSource -> Map: ExtractTransform -> Filter: FilterExceptionData -> Timestamps/Watermarks",
      "parallelism": 2,
      "now": 1568523887783,
      "subtasks": [
        {
          "subtask": 0,
          "status": "RUNNING",
          "attempt": 0,
          "host": "node5:41687",
          "start-time": 1568519471838,
          "end-time": -1,
          "duration": 4415945,
          "metrics": {
            "read-bytes": 0,
            "read-bytes-complete": true,
            "write-bytes": 113598,
            "write-bytes-complete": true,
            "read-records": 0,
            "read-records-complete": true,
            "write-records": 1486,
            "write-records-complete": true
          },
          "start_time": 1568519471838
        },
        {
          "subtask": 1,
          "status": "RUNNING",
          "attempt": 0,
          "host": "node5:41687",
          "start-time": 1568519471840,
          "end-time": -1,
          "duration": 4415943,
          "metrics": {
            "read-bytes": 0,
            "read-bytes-complete": true,
            "write-bytes": 223023,
            "write-bytes-complete": true,
            "read-records": 0,
            "read-records-complete": true,
            "write-records": 2919,
            "write-records-complete": true
          },
          "start_time": 1568519471840
        }
      ]
    }

28. 查看Job某个Vertex Metrics

API: /jobs/:jobid/vertices/:vertexid/metrics
    参数:
        A、vertexid: jobs/jobid接口vertices中的id
        B、get: 逗号分隔的指定metrics名字。
请求:
    curl -X GET http://node2:50100/v1/jobs/ff9d383001652ab54e4970965fa386ee/vertices/bd9c186e89cc37a76d776844dfa8ccba/metrics?get=0.numRecordsInPerSecond,0.buffers.inputQueueLength,0.buffers.outputQueueLength,1.numRecordsInPerSecond,1.buffers.inputQueueLength,1.buffers.outputQueueLength|jq
返回:
    [
      {
        "id": "0.buffers.inputQueueLength",
        "value": "0"
      },
      {
        "id": "1.numRecordsInPerSecond",
        "value": "0.0"
      },
      {
        "id": "0.numRecordsInPerSecond",
        "value": "0.0"
      },
      {
        "id": "1.buffers.inputQueueLength",
        "value": "0"
      },
      {
        "id": "0.buffers.outputQueueLength",
        "value": "2"
      },
      {
        "id": "1.buffers.outputQueueLength",
        "value": "2"
      }
]

29. 查看Job某个Vertex下某Subtasks信息

API: /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskindex
    参数:
        A、vertexid: jobs/jobid接口vertices中的id
        B、subtaskindex: /jobs/:jobid/vertices/:vertexid接口中的subtask。
请求:
    curl -X GET http://node2:50100/v1/jobs/ff9d383001652ab54e4970965fa386ee/vertices/bd9c186e89cc37a76d776844dfa8ccba/subtasks/0|jq
返回:
    {
      "subtask": 0,
      "status": "RUNNING",
      "attempt": 0,
      "host": "node5",
      "start-time": 1568519471838,
      "end-time": -1,
      "duration": 5682174,
      "metrics": {
        "read-bytes": 0,
        "read-bytes-complete": true,
        "write-bytes": 145817,
        "write-bytes-complete": true,
        "read-records": 0,
        "read-records-complete": true,
        "write-records": 1907,
        "write-records-complete": true
      }
    }

30. 查看Job某个Vertex下某个Subtasks Metrics

API: /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskindex/metrics
    参数:
        A、vertexid: jobs/jobid接口vertices中的id
        B、subtaskindex: /jobs/:jobid/vertices/:vertexid接口中的subtask。
        C、get参数: 逗号分隔的metrics列表。
请求:
    curl -X GET http://node2:50100/v1/jobs/ff9d383001652ab54e4970965fa386ee/vertices/bd9c186e89cc37a76d776844dfa8ccba/subtasks/0/metrics?get=Timestamps/Watermarks.currentOutputWatermark,Source__KafkaSource.records-consumed-rate|jq
返回:
    [
      {
        "id": "Source__KafkaSource.records-consumed-rate",
        "value": "0.38687440649949"
      },
      {
        "id": "Timestamps/Watermarks.currentOutputWatermark",
        "value": "1568527125000"
      }
    ]

31. 查看Job某个Vertex下某个Subtasks下某次Attempt信息

API: /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskindex/attempts/:attempt
    参数:
        A、vertexid: jobs/jobid接口vertices中的id
        B、subtaskindex: /jobs/:jobid/vertices/:vertexid接口中的subtask。
        C、attempt: /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskindex接口中的attempt。
请求:
    curl -X GET http://node2:50100/v1/jobs/ff9d383001652ab54e4970965fa386ee/vertices/bd9c186e89cc37a76d776844dfa8ccba/subtasks/1/attempts/0|jq
返回:
    {
      "subtask": 0,
      "status": "RUNNING",
      "attempt": 0,
      "host": "node5",
      "start-time": 1568519471838,
      "end-time": -1,
      "duration": 5682174,
      "metrics": {
        "read-bytes": 0,
        "read-bytes-complete": true,
        "write-bytes": 145817,
        "write-bytes-complete": true,
        "read-records": 0,
        "read-records-complete": true,
        "write-records": 1907,
        "write-records-complete": true
      }
    }
wangpei1949
关注
专栏目录
Flink常见输出（sink）算子使用教程（DataStream API）
SunnyRivers
08-22 81
一个 Flink 程序，其实就是对 DataStream 的各种转换。具体来说，代码基本上都由以下几部分构成，获取执行环境（execution environment）读取数据源（source）定义基于数据的转换操作（transformations）定义计算结果的输出位置（sink）触发程序执行（execute）Flink 作为数据处理框架，最终还是要把计算处理的结果写入外部存储，为外部应用提供支持，如上图所示。
flink实战--metrics监控与rest api
热门推荐
阿华田的博客
05-24 1万+
Flink metrics简介 Flink的metrics是Flink公开的一个度量系统，允许收集flink程序的公开指标到外部系统，我们也可以自定义指标通过metric收集，实际开发时经常需要查看当前程序的运行状况，flink 提供了 UI 界面，有比较详细的统计信息。但是 UI 界面也有不完善的地方，比如想要获取 flink 的实时吞吐。本文将详细介绍如何通过metric......
Flink学习资料
10-09
附件：Flink-on-yarn部署指南 0、Flink基本原理与生产实践 1、Flink基本概念与部署 2、DataStreamAPI介绍与实战 3、Window与Time 4、Connector 5、Flink状态管理与恢复 6、Metrics与监控 7、Flink应用案例介绍 8、Druid基本概念以及架构设计 9、Druid数据存储与写入 10、Druid实践介绍
FlinkRestAPI
m0_57320261的博客
01-08 877
如果输出结果为空，则说明 Flink 客户端没有安装在系统路径中。在这种情况下，您可以通过设置。which flink 找到Flink客户端地址。环境变量来指定 Flink 客户端的路径。
Flink历史服务器
weixin_55787608的博客
08-27 466
在实际线上开发flink任务时，是否存在这样的需求：flink实时任务掉线了，此时我们再去查看yarn的web页面最多也只能查看JobManager 的日志了，像检查点，处理的数据量，水印等指标如果没有Metrics 监控，那么很遗憾这些信息就丢失了，对我们分析flink任务出现的bug，异常退出的原因很不友好，此时flink的HistoryServer出现了。
Flink DataStream API使用(含实例)
wx_start_ag的博客
09-22 825
本文主要通过代码练习熟悉Flink DataStream相关API的功能和使用。读者可完成简单的准备工作后跟着一起完成练习 准备 一台装有 Docker 的 Linux 或 MacOS 计算机。 使用 Docker Compose 启动容器 通过 wget 命令自动下载该 docker-compose.yml 文件，也可以手动下载 mkdir flink-service; cd flink-service; wget https://gitee.com/WX_in_gitee/flink_learn/raw
Flink使用RestApi
jeseva的博客
08-05 5225
flink是一个非常好用的流任务计算框架, 这次我们来试用flink的restApi来提交任务. 1, 上传jar包 public static boolean uploadJar( File jarFile) { RequestBody requestBody = new MultipartBody.Builder() .setType(MultipartBody.FORM) .addFormDataPart("fil
Flink REST API #
CarloPan的博客
05-27 1796
官方文档： https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/rest_api/ https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/rest_api.html /jars Returns a list of all jars previously uploaded via '/jars/upload'. /jars/uplo.
Flink Metrics&REST API 介绍和原理解析
cloudbigdata的博客
05-06 2071
通过学习 Flink Metrics 和 REST 接口，了解 Flink 是如何完成监控的。
Flink系列一：DataStream API 教程
yiweiyi329的博客
10-22 556
本教程主要来自flink1.9教程 设置Maven项目 编写Flink程序 在集群上运行并写入kafka 在本指南中，我们将从头开始，从设置Flink项目到在Flink集群上运行流分析程序。 Wikipedia提供了一个IRC频道，其中记录了对Wiki的所有编辑。我们将在Flink中读取此通道，并计算每个用户在给定时间窗口内编辑的字节数。这很容易使用Flink在几分...
【Flink】通过REST API监控正在运行的作业状态和统计信息
qq_43744420的博客
08-10 2055
通过REST API监控正在运行的作业状态和统计信息
RESTful-API-demo:RESTful API演示
05-02
使用express,MongoDb快速构建Node.js REST API () REST API解决了什么问题（能干什么？） 网络应用-重前端轻后端模式，后端专注于业务，提供前端数据接口。一套API给多客户端提供服务 Resetful API-设计更加规范的API RESTful(Representational State Transfer)-表现层状态转化 每一个URI代表一种资源。为所有“事物”定义ID 客户端和服务器之间，传递这种资源的某种表现层。 客户端通过四个HTTP动词（GET、POST、PUT、DELETE），对服务器端资源进行操作，实现"表现层状态转化"。 设计规范（怎么用？） ####1. 网址中不能有动词，只能有名词，所用的名词往往与数据库的表格名对应。 ####2. 对于资源的具体操作类型，由HTTP动词表示。 * GET: 从服务器取回一个资源 * POST:
bind-restapi：BIND DNS的RESTful json api
02-05
绑定restapi 一种快速简单的RESTful BIND RESTful API，以Ruby / Sinatra编写。 提供使用现有的BIND DNS体系结构添加/删除条目的功能。 我将其写为一种解决方案，使我们的内部云能够通过与我们现有的DNS架构集成来向DNS添加/删除计算机。 使用说明 # cd etc/ # named -c named.conf $ ruby dns.rb 将记录添加到DNS： $ curl -X POST -H 'Content-Type: application/json' -H 'X-Api-Key: secret' -d '{ "hostname":
FlinkRestService(Flink Rest API)
杜海的博客
10-18 942
package com.iot.mapp.std.ruleengine.utils; import java.io.File; import java.util.ArrayList; import java.util.List; import org.apache.commons.lang.StringUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.beans.fact.
Flink Rest API
最新发布
xingyuelhb的博客
10-23 392
通过curl 或者Rest API工具测试web UI对应的接口返回信息。Flink官网API。
Flink基本的API（续）
weixin_30488085的博客
06-10 136
上一篇介绍了编写 Flink 程序的基本步骤，以及一些常见 API，如：map、filter、keyBy 等，重点介绍了 keyBy 方法。本篇将继续介绍 Flink 中常用的 API，主要内容为 指定 transform 函数 Flink 支持的数据类型 累加器 指定 transform 函数 许多 transform 操作需要用户自定义函数来实现，Flink 支持多种自定义 ...
Flink REST API 的使用
magic_kid_2010的专栏
07-24 7236
一、前言 Flink有一个监视 API，可以用来查询正在运行的作业以及最近完成的作业的状态和统计信息。这个监视API由Flink自己的仪表板使用，但也被设计用于定制监视工具。 监视API是一个REST-ful API，它接受HTTP请求并使用JSON数据进行响应。 默认情况下，此服务器监听8081端口，可以在flink-conf.yaml中配置rest.port。注意...
Flink Rest API接口使用
程序人生
09-13 3930
用户可以通过flink提供的Rest API管理应用。Rest API可供用户或脚本直接访问，它可以对外公开有关Flink集群和应用的信息
flink实战-实时计算平台通过api停止流任务
CarloPan的博客
06-12 1314
https://blog.csdn.net/zhangjun5965/article/details/106820591
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值