Trino On K8S(Dockerfile)[建议]

Trino On K8S(Dockerfile)[建议]

构建 Dockerfile,部署 Trino,Paimon Jar包可打在镜像里,K8S节点不用分发。

目前 0.8版本Paimon 仅支持 Trino 420/427

前期准备

基础镜像
docker pull centos:centos7
docker image tag  centos:centos7  10.83.195.6/bigdata/centos:centos7
docker push 10.83.195.6/bigdata/centos:centos7
下载java
# https://adoptium.net/zh-CN/temurin/releases/?os=linux&arch=any&package=jdk

# https://www.azul.com/downloads/?version=java-21-lts&os=centos&package=jdk#zulu
# Trino 444 需要Java 21及以上
构建 Dockerfile
FROM 10.83.195.6/bigdata/centos:centos7

COPY CentOS-Base.repo /etc/yum.repos.d/

# 修改时区
RUN rm -f /etc/localtime && ln -sv /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo "Asia/Shanghai" > /etc/timezone

# RUN yum -y install net-tools

# 使用root用户(容器内id为0),修改values。yaml编排里的spec.template.spec.containers. securityContext.runAsUser: 0
# RUN groupadd --system --gid=1000 admin && useradd --system --home-dir /home/admin --uid=1000 --gid=admin admin

# 创建目录
RUN mkdir -p /opt/apache

# 安装sudo
# RUN yum -y install sudo ; chmod 640 /etc/sudoers

# 给admin添加sudo权限
# RUN echo "admin ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

# 添加 JDK
ADD zulu21.32.17-ca-jdk21.0.2-linux_x64.tar.gz /opt/apache/
ENV JAVA_HOME /opt/apache/zulu21.32.17-ca-jdk21.0.2-linux_x64
ENV PATH $JAVA_HOME/bin:$PATH

# 添加 trino server
ADD trino-server-420.tar.gz /opt/apache/
RUN ln -s /opt/apache/trino-server-420 /opt/apache/trino
RUN mkdir -p /opt/apache/trino/etc/

ENV TRINO_HOME /opt/apache/trino

COPY bootstrap.sh /opt/apache/

RUN chmod +x /opt/apache/bootstrap.sh

RUN mkdir -p  /opt/apache/trino/tmp

# 增加 paimon jar
ADD paimon-trino-422-0.8-20240305.000456-12-plugin.tar.gz /opt/apache/trino/plugin/

# 增加 jmx exporter
COPY jmx_prometheus_javaagent-0.20.0.jar /opt/apache/trino/lib/
COPY jmx_config.yaml /tmp/

# 修改目录属组
# RUN chown -R admin:admin /opt/apache/

WORKDIR /opt/apache/trino/
ENTRYPOINT [ "/opt/apache/bootstrap.sh" ]
# vim jmx_config.yaml

rules:
- pattern: ".*"
编写 bootstrap.sh 脚本
# 最终版

#!/bin/bash

#node_id=`hostname`
#echo "node.id=$node_id" >>  $TRINO_HOME/etc/node.properties
# bash: /opt/apache/trino/etc/node.properties: Read-only file system

# $TRINO_HOME/bin/launcher start 是后台启动,需要前台启动用run
$TRINO_HOME/bin/launcher run -Djava.io.tmpdir=/opt/apache/trino/tmp
#!/bin/bash

cat /opt/apache/hosts | sudo tee -a /etc/hosts   >/dev/null
cat /opt/apache/resolv.conf | sudo tee -a /etc/resolv.conf  >/dev/null

# 修改配置
cat /tmp/node.properties > $TRINO_HOME/etc/node.properties
cat /tmp/config.properties > $TRINO_HOME/etc/config.properties

# 获取主机名当作node_id
node_id=`hostname`
# 替换NODE_ID
sed -i 's/${NODE_ID}/'$node_id'/g' $TRINO_HOME/etc/node.properties

mkdir -p /opt/apache/prestotmp/$node_id

sed -i 's/${NODE_ID}/'$node_id'/g' $TRINO_HOME/etc/config.properties


# 启动trino服务
$TRINO_HOME/bin/launcher start -Djava.io.tmpdir=/opt/apache/trino/tmp/
until [ -f ${TRINO_HOME}/data/var/log/server.log ]
do
   echo 'waiting for presto started...';
   sleep 1
done
tail -F ${TRINO_HOME}/data/var/log/server.log
构建镜像
# -t:指定镜像名称
# . :当前目录Dockerfile
# -f:指定Dockerfile路径
#  --no-cache:不缓存
docker build -t 10.83.195.8:1443/bigdata/trino:420 -f trino-Dockerfile . --no-cache
docker push 10.83.195.8:1443/bigdata/trino:420 

部署 Trino

下载 Helm 源
# https://artifacthub.io/packages/helm/trino/trino
helm repo add trino https://trinodb.github.io/charts/
helm pull trino/trino --version 0.19.0
tar -zxvf trino-0.19.0.tgz
修改配置
# vim values.yaml

# 修改镜像地址
image:
  registry: "10.83.195.8:1443"   
  repository: bigdata/trino   
  tag: "420"
  
server:
  # worker 个数
  workers: 1
  node:
    environment: production
    dataDir: /opt/apache/trino/data
    # pluginDir: /usr/lib/trino/plugin
    pluginDir: /opt/apache/trino/plugin

  config:
    # path: /etc/trino
    path: /opt/apache/trino/etc

# 修改为0,使用root用户启动
securityContext:
  runAsUser: 0
  runAsGroup: 0
启动
# 和 Chart.yaml 同级目录下执行
helm install trino-test ./ -n trino-test --create-namespace

# 在harbor Web 将bigdata项目改成 公开,否则因权限问题拉取镜像失败

# 重新构建镜像后,需要删除原镜像
# ansible -i /opt/ansible/nodes all -m shell -a "docker rmi 10.83.195.8:1443/bigdata/trino:420"

# 拉取新镜像
# ansible -i /opt/ansible/nodes all -m shell -a "docker pull 10.83.195.8:1443/bigdata/trino:420"

# 卸载
# helm uninstall trino-test -n trino-test && ansible -i /opt/ansible/nodes all -m shell -a "docker rmi 10.83.195.8:1443/bigdata/trino:420"

# 查看
kubectl get po -n trino-test

# 查看 pod日志
kubectl logs trino-coordinator-7ff574d48f-s7f45  -n trino-test  --all-containers

# 卸载
# helm uninstall trino-test -n trino-test

kubectl get po -n trino-test

coordinator_name=`kubectl get pods -n trino-test|grep coordinator|awk '{print $1}'`
kubectl exec -it $coordinator_name -n trino-test -- sh

kubectl describe po $coordinator_name  -n trino-test 
kubectl logs $coordinator_name  -n trino-test  --all-containers

# 验证
/data/trino_helm/trino-cli-444-executable.jar --server http://192.168.22.112:8080 --catalog=hive --schema=default --user=admin

select * from system.runtime.nodes;
# 仅用于调试
kubectl edit deploy trino-test-worker -n trino-test

    spec:
      containers:
      - image: 10.83.195.8:1443/bigdata/trino:420
        command:  ['bash','-c','sleep 3600s']

kubectl edit deploy trino-test-coordinator -n trino-test

    spec:
      containers:
      - image: 10.83.195.8:1443/bigdata/trino:420
        command:  ['bash','-c','sleep 3600s']       

增加 Hive Catalog

# vim values.yaml

# 添加 hive catalog
# hive.config.resources 和后面的挂载路径一致
#additionalCatalogs: {}
additionalCatalogs:
  hive: |-
    connector.name=hive
    hive.metastore.uri=thrift://10.83.192.8:9083,thrift://10.83.192.9:9083
    hive.config.resources=/tmp/core-site.xml,/tmp/hdfs-site.xml
挂载 core-site.xml hdfs-site.xml 配置
# vim templates/configmap-hadoop.yaml
# 需要把 core-site.xml hdfs-site.xml 中配置的hostname 换成ip

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "trino.fullname" . }}-hadoop
  labels:
    app.kubernetes.io/name: {{ include "trino.name" . }}
    helm.sh/chart: {{ include "trino.chart" . }}
    app.kubernetes.io/instance: {{ .Release.Name }}
data:
  core-site.xml: |
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration>
      <property>
        <name>fs.defaultFS</name>
        <value>hdfs://bigbigworld</value>
      </property>
      <property>
        <name>ha.zookeeper.quorum</name>
        <value>10.83.192.6:2181,10.83.192.7:2181,10.83.192.8:2181</value>
      </property>
       <property>
        <name>io.file.buffer.size</name>
        <value>131072</value>
      </property>
        <property>
        <name>fs.trash.interval</name>
        <value>4320</value>
      </property>
      <property>
        <name>fs.trash.checkpoint.interval</name>
        <value>60</value>
      </property>
      <property>
        <name>io.native.lib.available</name>
        <value>true</value>
      </property>
      <property>
        <name>net.topology.script.file.name</name>
        <value>/opt/apache/hadoop/etc/hadoop/topology.sh</value>
      </property>
       <property>
        <name>dfs.ha.fencing.methods</name>
        <value>
          sshfence
          shell(/bin/true)
        </value>
      </property>
      <property>
        <name>dfs.ha.fencing.ssh.private-key-files</name>
        <value>file:///home/admin/.ssh/id_rsa</value>
      </property>
      <property>
        <name>dfs.ha.fencing.ssh.connect-timeout</name>
        <value>1000</value>
      </property>
      <property>
        <name>hadoop.proxyuser.admin.hosts</name>
        <value>*</value>
      </property>
      <property>
        <name>hadoop.proxyuser.admin.users</name>
        <value>*</value>
      </property>
      <property>
        <name>hadoop.proxyuser.admin.groups</name>
        <value>*</value>
      </property>
      <property>
        <name>hadoop.proxyuser.hive.hosts</name>
        <value>*</value>
      </property>
      <property>
        <name>hadoop.proxyuser.hive.users</name>
        <value>*</value>
      </property>
      <property>
        <name>hadoop.proxyuser.yarn.groups</name>
        <value>*</value>
      </property>
      <property>
        <name>hadoop.proxyuser.yarn.users</name>
        <value>*</value>
      </property>
      <property>
        <name>hadoop.proxyuser.yarn.groups</name>
        <value>*</value>
      </property>
      <property>
        <name>ipc.server.read.threadpool.size</name>
        <value>5</value>
      </property>

      <property>
        <name>ipc.server.listen.queue.size</name>
        <value>1024</value>
      </property>

    </configuration>

  hdfs-site.xml: |
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration>
      <property>
        <name>dfs.replication</name>
        <value>3</value>
      </property>
      <property>
        <name>dfs.nameservices</name>
        <value>bigbigworld</value>
      </property>
        <property>
        <name>dfs.ha.namenodes.bigbigworld</name>
        <value>nn1,nn2</value>
      </property>
      <property>
        <name>dfs.namenode.rpc-address.bigbigworld.nn1</name>
        <value>10.83.192.6:8020</value>
      </property>
      <property>
        <name>dfs.namenode.rpc-address.bigbigworld.nn2</name>
        <value>10.83.192.7:8020</value>
      </property>
      <property>
        <name>dfs.namenode.http-address</name>
        <value>0.0.0.0:9870</value>
      </property>
      <property>
        <name>dfs.namenode.http-address.bigbigworld.nn1</name>
        <value>10.83.192.6:9870</value>
      </property>
      <property>
        <name>dfs.namenode.http-address.bigbigworld.nn2</name>
        <value>10.83.192.7:9870</value>
      </property>
      <property>
        <name>dfs.namenode.shared.edits.dir</name>
        <value>qjournal://10.83.192.6:8485;10.83.192.7:8485;10.83.192.8:8485/bigbigworld</value>
      </property>
      <property>
        <name>dfs.permissions.enabled</name>
        <value>true</value>
      </property>
      <property>
        <name>dfs.client.failover.proxy.provider.bigbigworld</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
      </property>
      <property>
        <name>dfs.ha.automatic-failover.enabled.bigbigworld</name>
        <value>true</value>
      </property>
      <property>
        <name>dfs.journalnode.edits.dir</name>
        <value>/data1/hadoop/dfs/journalnode</value>
      </property>
      <property>
        <name>dfs.qjournal.write-txns.timeout.ms</name>
        <value>60000</value>
      </property>
      <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:///data1/hadoop/dfs/namenode</value>
      </property>
      <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:///data1/hadoop/dfs/datanode</value>
      </property>
      <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
      </property>
      <property>
        <name>dfs.namenode.handler.count</name>
        <value>192</value>
      </property>
      <property>
        <name>dfs.datanode.handler.count</name>
        <value>96</value>
      </property>
      <property>
        <name>dfs.datanode.max.transfer.threads</name>
        <value>16384</value>
      </property>
      <property>
        <name>dfs.datanode.socket.write.timeout</name>
        <value>480000</value>
      </property>
      <property>
        <name>dfs.client.socket-timeout</name>
        <value>300000</value>
      </property>
      <property>
        <name>dfs.datanode.balance.bandwidthPerSec</name>
        <value>209715200</value>
      </property>
      <property>
        <name>dfs.datanode.balance.max.concurrent.moves</name>
        <value>64</value>
      </property>
      <property>
        <name>dfs.namenode.replication.max-streams</name>
        <value>128</value>
      </property>
      <property>
        <name>dfs.namenode.replication.max-streams-hard-limit</name>
        <value>512</value>
      </property>
      <property>
        <name>dfs.namenode.replication.work.multiplier.per.iteration</name>
        <value>512</value>
      </property>
      <property>
        <name>dfs.hosts</name>
        <value>/opt/apache/hadoop/etc/hadoop/dfs-hosts.includes</value>
      </property>
      <property>
        <name>dfs.hosts.exclude</name>
        <value>/opt/apache/hadoop/etc/hadoop/dfs-hosts.excludes</value>
      </property>
      <property>
        <name>dfs.balancer.moverThreads</name>
        <value>2000</value>
      </property>
      <property>
        <name>dfs.balancer.max-size-to-move</name>
        <value>107374182400</value>
      </property>
      <property>
        <name>dfs.balancer.getBlocks.min-block-size</name>
        <value>1048576</value>
      </property>
      <property>
        <name>dfs.block.invalidate.limit</name>
        <value>2000</value>
      </property>
      <property>
        <name>dfs.namenode.acls.enabled</name>
        <value>true</value>
      </property>
      <property>
        <name>dfs.blockreport.incremental.intervalMsec</name>
        <value>50</value>
      </property>
      <property>
        <name>dfs.namenode.checkpoint.txns</name>
        <value>3000000</value>
      </property>
      <property>
        <name>dfs.qjournal.write-txns.timeout.ms</name>
        <value>90000</value>
      </property>
      <property>
        <name>dfs.qjournal.start-segment.timeout.ms</name>
        <value>90000</value>
      </property>
      <property>
        <name>dfs.qjournal.select-input-streams.timeout.ms</name>
        <value>90000</value>
      </property>
      <property>
        <name>dfs.namenode.audit.log.async</name>
        <value>true</value>
      </property>
      <property>
        <name>dfs.namenode.servicerpc-address.bigbigworld.nn1</name>
        <value>10.83.192.6:8041</value>
      </property>
      <property>
        <name>dfs.namenode.servicerpc-address.bigbigworld.nn2</name>
        <value>10.83.192.7:8041</value>
      </property>
    </configuration>
修改 coordinator/worker configmap
# vim templates/configmap-coordinator.yaml 
# vim templates/configmap-worker.yaml 
# configmap-coordinator.yaml和configmap-worker.yaml的jvm.config 增加如下配置
# HADOOP_USER_NAME为Hadoop集群管理员用户
-DHADOOP_USER_NAME=admin
修改 coordinator/worker deploy
# vim templates/deployment-coordinator.yaml 
# vim templates/deployment-worker.yaml
# deployment-coordinator.yaml和deployment-worker.yaml的volumes、volumeMounts 增加配置

      volumes:
        # 新增如下配置
        - name: core-site
          configMap:
            name: {{ include "trino.fullname" . }}-hadoop
        - name: hdfs-site
          configMap:
            name: {{ include "trino.fullname" . }}-hadoop

          volumeMounts:
            # 新增如下配置
            - mountPath: /tmp/hdfs-site.xml
              name: hdfs-site
              subPath: hdfs-site.xml
            - mountPath: /tmp/core-site.xml
              name: core-site
              subPath: core-site.xml
验证
/data/trino_helm/trino-cli-444-executable.jar --server http://192.168.22.118:8080 --catalog=hive --schema=default --user=admin

select * from system.runtime.nodes;

增加 Paimon Catalog【目前 0.8版本Paimon 仅支持 Trino 420/427】

# vim values.yaml

additionalCatalogs:
  hive: |-
    connector.name=hive
    hive.metastore.uri=thrift://10.83.192.8:9083,thrift://10.83.192.9:9083
    hive.config.resources=/tmp/core-site.xml,/tmp/hdfs-site.xml
  paimon: |-
    connector.name=paimon
    warehouse=hdfs://bigbigworld/user/hive/warehouse
    hive.config.resources=/tmp/core-site.xml,/tmp/hdfs-site.xml
    
    
# Dockerfile 增加 paimon plugins

增加 Ingress 外部访问

# values.yaml

ingress:
  enabled: true
  className: "nginx"
  annotations: {}
  hosts:
    - host: bigdata-dev-trino.ky-tech.com.cn
      paths:
        - path: /
          pathType: Prefix

  tls:
    - secretName: secret-trino-tls
      hosts:
        - bigdata-dev-trino.ky-tech.com.cn


kubectl create secret tls secret-trino-tls --key /data/harbor_helm/stl/ky-tech.com.cn_nginx/ky-tech.com.cn.key --cert /data/harbor_helm/stl/ky-tech.com.cn_nginx/ky-tech.com.cn_bundle.crt -n trino-test

增加 JMX Exporter


  jvm.config: |
    -DHADOOP_USER_NAME=admin
    -Dcom.sun.management.jmxremote.rmi.port=9081
    -javaagent:/opt/apache/trino/lib/jmx_prometheus_javaagent-0.20.0.jar=9082:/tmp/jmx_config.yaml
    
  config.properties: |
    jmx.rmiregistry.port=9080
    jmx.rmiserver.port=9081

Kube Prometheus 集成 JMX 监控

# vim manifests/jmx

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/name: jmx
    cluster: trino-test
  name: trino-test
  namespace: monitoring
spec:
  endpoints:
  - interval: 30s
    scrapeTimeout: 30s
    port: metrcis
    metricRelabelings:
    - sourceLabels: ['__name__']
      regex: 'jvm_(.*)|java_(.*)|trino_(execution|memory|metadata|operator|security|server|sql|failuredetector)_(.*)|process_(.*)'
      action: keep
    relabelings:
    - sourceLabels:
      - __meta_kubernetes_service_label_cluster
      targetLabel: cluster
    - sourceLabels:
      - __address__
      targetLabel: ip
      regex: "(.*):(.*)"
      replacement: $1
  selector:
    matchLabels:
      app.kubernetes.io/name: jmx
      cluster: trino-test
  namespaceSelector:
    matchNames:
    - trino-test
---
apiVersion: v1
kind: Service
metadata:
  name: trino-test-jmx
  labels:
    app.kubernetes.io/name: jmx
    cluster: trino-test
    app: trino
  namespace: trino-test
spec:
  type: ClusterIP
  clusterIP: None
  ports:
  - name: metrcis
    port: 9082
    targetPort: 9082
    protocol: TCP
  selector:
    app: trino

# vim prometheus-clusterRole.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/instance: k8s
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 2.36.1
  name: prometheus-k8s
  namespace: monitoring
rules:
- apiGroups:
  - ""
  resources:
  - nodes/metrics
  - pods
  - services
  - endpoints
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- nonResourceURLs:
  - /metrics
  verbs:
  - get

Grafana 图表 Json

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "datasource",
          "uid": "grafana"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "target": {
          "limit": 100,
          "matchAny": false,
          "tags": [],
          "type": "dashboard"
        },
        "type": "dashboard"
      }
    ]
  },
  "description": "JMX Dashboard for Kubernetes Monitoring with Prometheus",
  "editable": true,
  "fiscalYearStartMonth": 0,
  "gnetId": 11131,
  "graphTooltip": 0,
  "id": 39,
  "iteration": 1716690732539,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "datasource": {
        "type": "datasource",
        "uid": "grafana"
      },
      "description": "",
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "id": 138,
      "options": {
        "content": "",
        "mode": "html"
      },
      "pluginVersion": "8.5.5",
      "title": "数据来源:JMX Exporter ; 采集频率:每30s一次",
      "type": "text"
    },
    {
      "collapsed": false,
      "datasource": {
        "type": "datasource",
        "uid": "grafana"
      },
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 1
      },
      "id": 46,
      "panels": [],
      "title": "总览",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P1809F7CD0C75ACF3"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "fixedColor": "dark-green",
            "mode": "fixed"
          },
          "decimals": 0,
          "mappings": [],
          "max": 68,
          "min": 0,
          "noValue": "0",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "rgba(245, 54, 54, 0.9)",
                "value": null
              },
              {
                "color": "rgba(50, 172, 45, 0.97)",
                "value": 0
              },
              {
                "color": "rgba(237, 129, 40, 0.89)",
                "value": 69
              }
            ]
          },
          "unit": "short"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "worker 节点掉线数量"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "dark-red",
                  "mode": "fixed"
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 0,
        "y": 2
      },
      "id": 1,
      "interval": "30s",
      "options": {
        "displayMode": "gradient",
        "minVizHeight": 10,
        "minVizWidth": 0,
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "last"
          ],
          "fields": "",
          "values": false
        },
        "showUnfilled": true,
        "text": {}
      },
      "pluginVersion": "8.5.5",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "exemplar": true,
          "expr": "trino_failuredetector_HeartbeatFailureDetector_ActiveCount{cluster=\"$cluster\"}",
          "format": "time_series",
          "hide": false,
          "instant": false,
          "interval": "",
          "legendFormat": "worker 节点在线数量",
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "exemplar": true,
          "expr": "count(up{cluster=\"$cluster\"})-count(up{cluster=\"$cluster\"}==1)",
          "hide": false,
          "instant": false,
          "interval": "",
          "legendFormat": "worker 节点掉线数量",
          "refId": "C"
        }
      ],
      "title": "worker 节点数量",
      "type": "bargauge"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P1809F7CD0C75ACF3"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "GB"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 6,
        "w": 4,
        "x": 7,
        "y": 2
      },
      "id": 3,
      "interval": "30s",
      "options": {
        "colorMode": "none",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "pluginVersion": "8.5.5",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "exemplar": true,
          "expr": "trino_failuredetector_HeartbeatFailureDetector_ActiveCount{cluster=\"$cluster\"} * $maxTotalMemoryPerNode",
          "format": "time_series",
          "instant": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "A"
        }
      ],
      "title": "Presto 最大使用内存",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P1809F7CD0C75ACF3"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "max": 1,
          "min": 0,
          "noValue": "0",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 0.9
              }
            ]
          },
          "unit": "percentunit"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 6,
        "w": 4,
        "x": 11,
        "y": 2
      },
      "id": 13,
      "maxDataPoints": 100,
      "options": {
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true
      },
      "pluginVersion": "8.5.5",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "exemplar": true,
          "expr": "sum((trino_memory_MemoryPool_MaxBytes{cluster=~\"$cluster\", endpoint=\"metrcis\"} - trino_memory_MemoryPool_FreeBytes{cluster=~\"$cluster\", endpoint=\"metrcis\"}))/sum(trino_memory_MemoryPool_MaxBytes{cluster=~\"$cluster\", endpoint=\"metrcis\"})",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "A",
          "step": 20
        }
      ],
      "title": "Memory Usage",
      "type": "gauge"
    },
    {
      "aliasColors": {},
      "bars": false,
      "colorBackground": false,
      "colorValue": true,
      "dashLength": 10,
      "dashes": false,
      "datasource": {
        "type": "prometheus",
        "uid": "P1809F7CD0C75ACF3"
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 6,
        "w": 9,
        "x": 15,
        "y": 2
      },
      "hiddenSeries": false,
      "id": 14,
      "legend": {
        "alignAsTable": true,
        "avg": true,
        "current": true,
        "max": true,
        "min": true,
        "rightSide": false,
        "show": true,
        "sortDesc": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "8.5.5",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "$$hashKey": "object:147",
          "alias": "Usage %",
          "bars": true,
          "color": "#6d1f62",
          "legend": false,
          "lines": false,
          "yaxis": 2,
          "zindex": -1
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "exemplar": true,
          "expr": "sum((trino_memory_MemoryPool_MaxBytes{cluster=~\"$cluster\", endpoint=\"metrcis\"} - trino_memory_MemoryPool_FreeBytes{cluster=~\"$cluster\", endpoint=\"metrcis\"}))/sum(trino_memory_MemoryPool_MaxBytes{cluster=~\"$cluster\", endpoint=\"metrcis\"})",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "Usage",
          "refId": "A",
          "step": 20
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "expr": "avg(container_memory_usage_bytes{container=\"tomcat\",pod=\"$pod\"})/avg(kube_pod_container_resource_limits_memory_bytes{container=\"tomcat\",pod=\"$pod\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Usage %",
          "refId": "B",
          "step": 20
        }
      ],
      "thresholds": [],
      "timeRegions": [],
      "title": "总内存平均使用率",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "mode": "time",
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "$$hashKey": "object:441",
          "decimals": 0,
          "format": "percentunit",
          "logBase": 1,
          "show": true
        },
        {
          "$$hashKey": "object:442",
          "decimals": 1,
          "format": "percentunit",
          "label": "",
          "logBase": 1,
          "max": "1",
          "min": "0",
          "show": true
        }
      ],
      "yaxis": {
        "align": false
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": {
        "type": "prometheus",
        "uid": "P1809F7CD0C75ACF3"
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 8
      },
      "hiddenSeries": false,
      "id": 133,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "8.5.5",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "expr": "trino_failuredetector_HeartbeatFailureDetector_ActiveCount{cluster=\"$cluster\"}",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeRegions": [],
      "title": "集群 Worker 节点数",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "mode": "time",
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "$$hashKey": "object:119",
          "format": "short",
          "logBase": 1,
          "show": true
        },
        {
          "$$hashKey": "object:120",
          "format": "short",
          "logBase": 1,
          "show": true
        }
      ],
      "yaxis": {
        "align": false
      }
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P1809F7CD0C75ACF3"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": true,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 8
      },
      "id": 135,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom"
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "pluginVersion": "8.3.3",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "expr": "sum(sum(trino_memory_MemoryPool_MaxBytes{cluster=~\"$cluster\", endpoint=\"metrcis\"}) by(instance)  -sum(trino_memory_MemoryPool_FreeBytes{cluster=~\"$cluster\", endpoint=\"metrcis\"}) by(instance))/1024/1024/1024",
          "refId": "A"
        }
      ],
      "title": "集群内存总使用量",
      "type": "timeseries"
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": {
        "type": "prometheus",
        "uid": "P1809F7CD0C75ACF3"
      },
      "description": "",
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 16
      },
      "hiddenSeries": false,
      "id": 131,
      "legend": {
        "alignAsTable": true,
        "avg": true,
        "current": true,
        "max": true,
        "min": true,
        "rightSide": false,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "8.5.5",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "expr": "max(trino_execution_QueryManager_RunningQueries{cluster=~\"$cluster\", endpoint=\"metrcis\"})",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeRegions": [],
      "title": "正在运行的查询总数",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "mode": "time",
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "$$hashKey": "object:365",
          "format": "short",
          "logBase": 1,
          "show": true
        },
        {
          "$$hashKey": "object:366",
          "format": "short",
          "logBase": 1,
          "show": true
        }
      ],
      "yaxis": {
        "align": false
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": {
        "type": "prometheus",
        "uid": "P1809F7CD0C75ACF3"
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 16
      },
      "hiddenSeries": false,
      "id": 129,
      "legend": {
        "alignAsTable": true,
        "avg": true,
        "current": true,
        "max": true,
        "min": true,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "8.5.5",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "expr": "max(trino_execution_QueryManager_QueuedQueries{cluster=~\"$cluster\", endpoint=\"metrcis\"})",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeRegions": [],
      "title": "任务排队数",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "mode": "time",
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "$$hashKey": "object:193",
          "format": "short",
          "logBase": 1,
          "show": true
        },
        {
          "$$hashKey": "object:194",
          "format": "short",
          "logBase": 1,
          "show": true
        }
      ],
      "yaxis": {
        "align": false
      }
    },
    {
      "collapsed": true,
      "datasource": {
        "type": "datasource",
        "uid": "grafana"
      },
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 24
      },
      "id": 48,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 25
          },
          "hiddenSeries": false,
          "id": 54,
          "legend": {
            "alignAsTable": true,
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "rightSide": true,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "(trino_memory_MemoryPool_MaxBytes{cluster=~\"$cluster\", endpoint=\"metrcis\"} - trino_memory_MemoryPool_FreeBytes{cluster=~\"$cluster\", endpoint=\"metrcis\"})/1024/1024/1024",
              "interval": "",
              "legendFormat": "{{pod}}{{ip}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "已使用内存",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:60",
              "format": "GB",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:61",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 12,
            "y": 25
          },
          "hiddenSeries": false,
          "id": 52,
          "legend": {
            "alignAsTable": true,
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "rightSide": true,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_memory_MemoryPool_MaxBytes{cluster=~\"$cluster\", endpoint=\"metrcis\"}/1024/1024/1024",
              "interval": "",
              "legendFormat": "{{pod}}{{ip}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "最大可使用内存",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:242",
              "format": "GB",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:243",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "description": "",
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 33
          },
          "hiddenSeries": false,
          "id": 126,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "rightSide": true,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "sum(trino_execution_executor_TaskExecutor_TotalSplits{cluster=~\"$cluster\", endpoint=\"metrcis\"})",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "分片执行数量(sum)",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:79",
              "format": "short",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:80",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "description": "",
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 12,
            "y": 33
          },
          "hiddenSeries": false,
          "id": 127,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "rightSide": true,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_execution_executor_TaskExecutor_TotalSplits{cluster=~\"$cluster\", endpoint=\"metrcis\"}",
              "interval": "",
              "legendFormat": "{{pod}}{{ip}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "节点分片执行数量",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:79",
              "format": "short",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:80",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 41
          },
          "hiddenSeries": false,
          "id": 56,
          "legend": {
            "alignAsTable": true,
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "rightSide": true,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_memory_MemoryPool_ReservedRevocableBytes{cluster=~\"$cluster\", endpoint=\"metrcis\"}/1024/1024/1024",
              "interval": "",
              "legendFormat": "{{pod}}{{ip}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "ReservedRevocable",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:577",
              "format": "GB",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:578",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "description": "",
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 12,
            "y": 41
          },
          "hiddenSeries": false,
          "id": 50,
          "legend": {
            "alignAsTable": true,
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "rightSide": true,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_memory_MemoryPool_FreeBytes{cluster=~\"$cluster\", endpoint=\"metrcis\"}/1024/1024/1024",
              "interval": "",
              "legendFormat": "{{pod}}{{ip}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "剩余内存",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:135",
              "format": "GB",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:136",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        }
      ],
      "title": "Worker MemoryPool",
      "type": "row"
    },
    {
      "collapsed": true,
      "datasource": {
        "type": "datasource",
        "uid": "grafana"
      },
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 25
      },
      "id": 88,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 26
          },
          "hiddenSeries": false,
          "id": 108,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_execution_QueryExecution_Executor_TaskCount{cluster=~\"$cluster\", endpoint=\"metrcis\"}",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "Executor.TaskCount",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:643",
              "format": "short",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:644",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 12,
            "y": 26
          },
          "hiddenSeries": false,
          "id": 112,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_execution_QueryExecution_Executor_Terminating{cluster=~\"$cluster\", endpoint=\"metrcis\"}",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "Executor.Terminating",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:757",
              "format": "short",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:758",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 34
          },
          "hiddenSeries": false,
          "id": 106,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_execution_QueryExecution_Executor_Shutdown{cluster=~\"$cluster\", endpoint=\"metrcis\"}",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "Executor.RejectedExecutionHandler",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:586",
              "format": "short",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:587",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 12,
            "y": 34
          },
          "hiddenSeries": false,
          "id": 110,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_execution_QueryExecution_Executor_Terminated{cluster=~\"$cluster\", endpoint=\"metrcis\"}",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "Executor.Terminated",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:700",
              "format": "short",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:701",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 42
          },
          "hiddenSeries": false,
          "id": 104,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_execution_QueryExecution_Executor_QueuedTaskCount{cluster=~\"$cluster\", endpoint=\"metrcis\"}",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "Executor.QueuedTaskCount",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:529",
              "format": "short",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:530",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 12,
            "y": 42
          },
          "hiddenSeries": false,
          "id": 96,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_execution_QueryExecution_Executor_CorePoolSize{cluster=~\"$cluster\", endpoint=\"metrcis\"}",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "CorePoolSize",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:1404",
              "format": "short",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:1405",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 50
          },
          "hiddenSeries": false,
          "id": 102,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_execution_QueryExecution_Executor_PoolSize{cluster=~\"$cluster\", endpoint=\"metrcis\"}",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "Executor.PoolSize",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:472",
              "format": "short",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:473",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 12,
            "y": 50
          },
          "hiddenSeries": false,
          "id": 92,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_execution_QueryExecution_Executor_AllowCoreThreadTimeOut{cluster=~\"$cluster\", endpoint=\"metrcis\"}",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "AllowCoreThreadTimeOut",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:301",
              "format": "short",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:302",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 58
          },
          "hiddenSeries": false,
          "id": 98,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_execution_QueryExecution_Executor_LargestPoolSize{cluster=~\"$cluster\", endpoint=\"metrcis\"}",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "LargestPoolSize",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:1147",
              "format": "short",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:1148",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 12,
            "y": 58
          },
          "hiddenSeries": false,
          "id": 100,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_execution_QueryExecution_Executor_MaximumPoolSize{cluster=~\"$cluster\", endpoint=\"metrcis\"}",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "MaximumPoolSize",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:415",
              "format": "short",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:416",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 66
          },
          "hiddenSeries": false,
          "id": 90,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_execution_QueryExecution_Executor_ActiveCount{cluster=~\"$cluster\", endpoint=\"metrcis\"}",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "Executor.ActiveCount",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:244",
              "format": "short",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:245",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 12,
            "y": 66
          },
          "hiddenSeries": false,
          "id": 94,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_execution_QueryExecution_Executor_CompletedTaskCount{cluster=~\"$cluster\", endpoint=\"metrcis\"}",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "CompletedTaskCount",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:358",
              "format": "short",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:359",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        }
      ],
      "title": "Coodinator QueryExecution",
      "type": "row"
    },
    {
      "collapsed": true,
      "datasource": {
        "type": "datasource",
        "uid": "grafana"
      },
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 26
      },
      "id": 72,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 27
          },
          "hiddenSeries": false,
          "id": 80,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_memory_ClusterMemoryPool_Nodes{cluster=~\"$cluster\", endpoint=\"metrcis\"}",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "worker 节点数",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:2274",
              "format": "short",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:2275",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 12,
            "y": 27
          },
          "hiddenSeries": false,
          "id": 86,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_memory_ClusterMemoryPool_TotalDistributedBytes{cluster=~\"$cluster\", endpoint=\"metrcis\"}/1024/1024/1024",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "TotalDistributed",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:2445",
              "format": "short",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:2446",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 35
          },
          "hiddenSeries": false,
          "id": 82,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_memory_ClusterMemoryPool_ReservedDistributedBytes{cluster=~\"$cluster\", endpoint=\"metrcis\"}/1024/1024/1024",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "ReservedDistributed",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:2331",
              "format": "GB",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:2332",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 12,
            "y": 35
          },
          "hiddenSeries": false,
          "id": 84,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_memory_ClusterMemoryPool_ReservedRevocableDistributedBytes{cluster=~\"$cluster\", endpoint=\"metrcis\"}/1024/1024/1024",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "ReservedRevocableDistributed",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:2388",
              "format": "GB",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:2389",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 43
          },
          "hiddenSeries": false,
          "id": 74,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_memory_ClusterMemoryPool_AssignedQueries{cluster=~\"$cluster\", endpoint=\"metrcis\"}/1024/1024/1024",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "AssignedQueries",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:1606",
              "format": "short",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:1607",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 12,
            "y": 43
          },
          "hiddenSeries": false,
          "id": 76,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_memory_ClusterMemoryPool_BlockedNodes{cluster=~\"$cluster\", endpoint=\"metrcis\"}",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "BlockedNodes",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:2160",
              "format": "short",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:2161",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fieldConfig": {
            "defaults": {
              "unit": "GB"
            },
            "overrides": []
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 51
          },
          "hiddenSeries": false,
          "id": 78,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_memory_ClusterMemoryPool_FreeDistributedBytes{cluster=~\"$cluster\", endpoint=\"metrcis\"}/1024/1024/1024",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "FreeDistributed",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:2217",
              "format": "GB",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:2218",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        }
      ],
      "title": "Coodinator ClusterMemoryPool",
      "type": "row"
    },
    {
      "collapsed": true,
      "datasource": {
        "type": "datasource",
        "uid": "grafana"
      },
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 27
      },
      "id": 58,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 28
          },
          "hiddenSeries": false,
          "id": 60,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_memory_ClusterMemoryManager_ClusterMemoryBytes{cluster=~\"$cluster\", endpoint=\"metrcis\"}/1024/1024/1024",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "集群总内存",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:493",
              "format": "GB",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:494",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 12,
            "y": 28
          },
          "hiddenSeries": false,
          "id": 62,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_memory_ClusterMemoryManager_ClusterTotalMemoryReservation{cluster=~\"$cluster\", endpoint=\"metrcis\"}/1024/1024/1024",
              "interval": "",
              "legendFormat": "",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "ClusterTotalMemoryReservation",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:699",
              "format": "GB",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:700",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 36
          },
          "hiddenSeries": false,
          "id": 68,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_memory_ClusterMemoryManager_QueriesKilledDueToOutOfMemory{cluster=~\"$cluster\", endpoint=\"metrcis\"}/1024/1024/1024",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "QueriesKilledDueToOutOfMemory",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:870",
              "format": "GB",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:871",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 12,
            "y": 36
          },
          "hiddenSeries": false,
          "id": 70,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_memory_ClusterMemoryManager_TotalAvailableProcessors{cluster=~\"$cluster\", endpoint=\"metrcis\"}",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "TotalAvailableProcessors",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:927",
              "format": "short",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:928",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 44
          },
          "hiddenSeries": false,
          "id": 64,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_memory_ClusterMemoryManager_ClusterUserMemoryReservation{cluster=~\"$cluster\", endpoint=\"metrcis\"}/1024/1024/1024",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "ClusterUserMemoryReservation",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:756",
              "format": "GB",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:757",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 12,
            "y": 44
          },
          "hiddenSeries": false,
          "id": 66,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_memory_ClusterMemoryManager_NumberOfLeakedQueries{cluster=~\"$cluster\", endpoint=\"metrcis\"}",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "NumberOfLeakedQueries",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:813",
              "format": "string",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:814",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        }
      ],
      "title": "Coodinator ClusterMemoryManager",
      "type": "row"
    },
    {
      "collapsed": true,
      "datasource": {
        "type": "datasource",
        "uid": "grafana"
      },
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 28
      },
      "id": 114,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 29
          },
          "hiddenSeries": false,
          "id": 116,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "8.3.3",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "P1809F7CD0C75ACF3"
              },
              "exemplar": true,
              "expr": "trino_execution_resourcegroups_InternalResourceGroupManager_QueriesQueuedOnInternal{cluster=~\"$cluster\", endpoint=\"metrcis\"}",
              "interval": "",
              "legendFormat": "{{pod}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeRegions": [],
          "title": "QueriesQueuedOnInternal",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "$$hashKey": "object:850",
              "format": "short",
              "logBase": 1,
              "show": true
            },
            {
              "$$hashKey": "object:851",
              "format": "short",
              "logBase": 1,
              "show": true
            }
          ],
          "yaxis": {
            "align": false
          }
        }
      ],
      "title": "Coodinator InternalResourceGroupManager",
      "type": "row"
    },
    {
      "collapsed": false,
      "datasource": {
        "type": "datasource",
        "uid": "grafana"
      },
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 29
      },
      "id": 124,
      "panels": [],
      "title": "JVM heap",
      "type": "row"
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": {
        "type": "prometheus",
        "uid": "P1809F7CD0C75ACF3"
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 7,
        "w": 24,
        "x": 0,
        "y": 30
      },
      "hiddenSeries": false,
      "id": 118,
      "legend": {
        "alignAsTable": true,
        "avg": false,
        "current": true,
        "max": true,
        "min": false,
        "rightSide": true,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "8.5.5",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "exemplar": true,
          "expr": "jvm_memory_bytes_used{cluster=~\"$cluster\", endpoint=\"metrcis\", area=\"heap\"}/jvm_memory_bytes_max{cluster=~\"$cluster\", area=\"heap\", endpoint=\"metrcis\"}",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 5,
          "legendFormat": "{{pod}}{{ip}}",
          "metric": "jvm_memory_bytes_used",
          "refId": "A",
          "step": 5
        }
      ],
      "thresholds": [],
      "timeRegions": [],
      "title": "Memory Usage(heap)",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "mode": "time",
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "$$hashKey": "object:145",
          "format": "percentunit",
          "logBase": 1,
          "show": true
        },
        {
          "$$hashKey": "object:146",
          "format": "%",
          "logBase": 1,
          "show": true
        }
      ],
      "yaxis": {
        "align": false
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": {
        "type": "prometheus",
        "uid": "P1809F7CD0C75ACF3"
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 7,
        "w": 24,
        "x": 0,
        "y": 37
      },
      "hiddenSeries": false,
      "id": 136,
      "legend": {
        "alignAsTable": true,
        "avg": false,
        "current": true,
        "max": true,
        "min": false,
        "rightSide": true,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "8.5.5",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "exemplar": true,
          "expr": "jvm_memory_bytes_used{cluster=~\"$cluster\", endpoint=\"metrcis\", area=\"heap\"}/1024/1024/1024",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 5,
          "legendFormat": "{{pod}}{{ip}}",
          "metric": "jvm_memory_bytes_used",
          "refId": "A",
          "step": 5
        }
      ],
      "thresholds": [],
      "timeRegions": [],
      "title": "Memory Used(heap)",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "mode": "time",
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "$$hashKey": "object:145",
          "format": "GB",
          "logBase": 1,
          "show": true
        },
        {
          "$$hashKey": "object:146",
          "format": "%",
          "logBase": 1,
          "show": false
        }
      ],
      "yaxis": {
        "align": false
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": {
        "type": "prometheus",
        "uid": "P1809F7CD0C75ACF3"
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 7,
        "w": 24,
        "x": 0,
        "y": 44
      },
      "hiddenSeries": false,
      "id": 139,
      "legend": {
        "alignAsTable": true,
        "avg": false,
        "current": true,
        "max": true,
        "min": false,
        "rightSide": true,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "8.5.5",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "exemplar": true,
          "expr": "jvm_memory_bytes_used{cluster=~\"$cluster\", endpoint=\"metrcis\", area=\"nonheap\"}/1024/1024/1024",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 5,
          "legendFormat": "{{pod}}{{ip}}",
          "metric": "jvm_memory_bytes_used",
          "refId": "A",
          "step": 5
        }
      ],
      "thresholds": [],
      "timeRegions": [],
      "title": "Memory Used(nonheap)",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "mode": "time",
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "$$hashKey": "object:145",
          "format": "GB",
          "logBase": 1,
          "show": true
        },
        {
          "$$hashKey": "object:146",
          "format": "%",
          "logBase": 1,
          "show": false
        }
      ],
      "yaxis": {
        "align": false
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": {
        "type": "prometheus",
        "uid": "P1809F7CD0C75ACF3"
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 7,
        "w": 12,
        "x": 0,
        "y": 51
      },
      "hiddenSeries": false,
      "id": 120,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "8.5.5",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "exemplar": true,
          "expr": "rate(jvm_gc_collection_seconds_sum{cluster=~\"$cluster\", endpoint=\"metrcis\"}[5m])",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 5,
          "legendFormat": "{{pod}}{{ip}}",
          "metric": "jvm_gc_collection_seconds_sum",
          "refId": "A",
          "step": 10
        }
      ],
      "thresholds": [],
      "timeRegions": [],
      "title": "GC time / 5 min. rate",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "mode": "time",
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "$$hashKey": "object:553",
          "format": "s",
          "logBase": 1,
          "show": true
        },
        {
          "$$hashKey": "object:554",
          "format": "short",
          "logBase": 1,
          "show": true
        }
      ],
      "yaxis": {
        "align": false
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": {
        "type": "prometheus",
        "uid": "P1809F7CD0C75ACF3"
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 7,
        "w": 12,
        "x": 12,
        "y": 51
      },
      "hiddenSeries": false,
      "id": 122,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "8.5.5",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P1809F7CD0C75ACF3"
          },
          "exemplar": true,
          "expr": "jvm_gc_collection_seconds_count{cluster=~\"$cluster\", endpoint=\"metrcis\"}",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 5,
          "legendFormat": "{{pod}}{{ip}}",
          "metric": "",
          "refId": "A",
          "step": 10
        }
      ],
      "thresholds": [],
      "timeRegions": [],
      "title": "GC count",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "mode": "time",
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "$$hashKey": "object:198",
          "decimals": 0,
          "format": "short",
          "logBase": 1,
          "show": true
        },
        {
          "$$hashKey": "object:199",
          "format": "short",
          "logBase": 1,
          "show": true
        }
      ],
      "yaxis": {
        "align": false
      }
    }
  ],
  "refresh": "",
  "schemaVersion": 36,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "current": {
          "selected": false,
          "text": "trino-test",
          "value": "trino-test"
        },
        "hide": 0,
        "includeAll": false,
        "label": "cluster",
        "multi": false,
        "name": "cluster",
        "options": [
          {
            "selected": true,
            "text": "trino-test",
            "value": "trino-test"
          }
        ],
        "query": "trino-test",
        "queryValue": "",
        "skipUrlSync": false,
        "type": "custom"
      },
      {
        "current": {
          "selected": false,
          "text": "10.82.192.13|10.82.192.160|10.82.192.191",
          "value": "10.82.192.13|10.82.192.160|10.82.192.191"
        },
        "description": "cip",
        "hide": 2,
        "includeAll": false,
        "label": "cip",
        "multi": false,
        "name": "cip",
        "options": [
          {
            "selected": true,
            "text": "10.82.192.13|10.82.192.160|10.82.192.191",
            "value": "10.82.192.13|10.82.192.160|10.82.192.191"
          }
        ],
        "query": "10.82.192.13|10.82.192.160|10.82.192.191",
        "queryValue": "",
        "skipUrlSync": false,
        "type": "custom"
      },
      {
        "current": {
          "selected": false,
          "text": "22",
          "value": "22"
        },
        "hide": 2,
        "includeAll": false,
        "label": "max-total-memory-per-node",
        "multi": false,
        "name": "maxTotalMemoryPerNode",
        "options": [
          {
            "selected": true,
            "text": "22",
            "value": "22"
          }
        ],
        "query": "22",
        "skipUrlSync": false,
        "type": "custom"
      },
      {
        "current": {
          "isNone": true,
          "selected": false,
          "text": "None",
          "value": ""
        },
        "datasource": {
          "type": "datasource",
          "uid": "grafana"
        },
        "definition": "",
        "hide": 2,
        "includeAll": false,
        "label": "IP",
        "multi": false,
        "name": "IP",
        "options": [],
        "query": {
          "query": "",
          "refId": "Prometheus-OPS-IP-Variable-Query"
        },
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      }
    ]
  },
  "time": {
    "from": "now-1h",
    "to": "now"
  },
  "timepicker": {
    "refresh_intervals": [
      "5s",
      "10s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m",
      "1h",
      "2h",
      "1d"
    ],
    "time_options": [
      "5m",
      "15m",
      "1h",
      "6h",
      "12h",
      "24h",
      "2d",
      "7d",
      "30d"
    ]
  },
  "timezone": "browser",
  "title": "K8S Trino420 JMX 监控1",
  "uid": "aaPB3LYIz1",
  "version": 9,
  "weekStart": ""
}

调整资源 requests 和 limits

# vim values.yaml

  resources:
    limits:
      cpu: 1
      memory: 4Gi
    requests:
      cpu: 0.1
      memory: 1Gi



# 卸载
# helm uninstall trino-test -n trino-test

# 更新
helm upgrade trino-test ./ -n trino-test

kubectl get po -n trino-test

设置Trino Worker 优雅下线

https://trino.io/docs/420/security/file-system-access-control.html#principal-rules

Worker 优雅下线需要设置 System access control

方式1: etc/access-control.properties 增加配置 access-control.name=allow-all

方式2:access-control.name=file security.config-file=etc/rules.json

# vim values.yaml

accessControl:
  type: configmap
  configFile: "rules.json"
  rules:
    rules.json: |-
      {
        "system_information": [
          {
            "role": "admin",
            "allow": ["read", "write"]
          },
          {
            "user": "admin",
            "allow": ["read", "write"]
          }
        ]
      }
# vim templates/deployment-worker.yaml

      volumes:
        {{- if .Values.accessControl }}{{- if eq .Values.accessControl.type "configmap" }}
        - name: access-control-volume
          configMap:
            name: trino-access-control-volume-worker
        {{- end }}{{- end }}
        

          volumeMounts:
            {{- if .Values.accessControl }}{{- if eq .Values.accessControl.type "configmap" }}
            - mountPath: {{ .Values.server.config.path }}/access-control
              name: access-control-volume
            {{- end }}{{- end }}
# vim templates/configmap-worker.yaml

{{- if .Values.accessControl }}{{- if eq .Values.accessControl.type "configmap" }}
  access-control.properties: |
    access-control.name=file
    {{- if .Values.accessControl.refreshPeriod }}
    security.refresh-period={{ .Values.accessControl.refreshPeriod }}
    {{- end }}
    security.config-file={{ .Values.server.config.path }}/access-control/{{ .Values.accessControl.configFile | default "rules.json" }}
{{- end }}{{- end }}

{{- if .Values.accessControl }}{{- if eq .Values.accessControl.type "configmap" }}
apiVersion: v1
kind: ConfigMap
metadata:
  name: trino-access-control-volume-worker
  labels:
    {{- include "trino.labels" . | nindent 4 }}
    app.kubernetes.io/component: worker
data:
  {{- range $key, $val := .Values.accessControl.rules }}
  {{ $key }}: {{ $val | quote }}
  {{- end }}
{{- end }}{{- end }}

---

  • 3
    点赞
  • 5
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
Trino Impala是一种音乐组合,组合集结了来自英国的天才音乐人Trino Green和Impala Johnson。他们以独特的音乐风格和出色的表演才华而闻名。 Trino Impala的音乐融合了多种音乐元素,包括摇滚、放克、灵魂和电子音乐。他们的音乐充满了活力和创新,总能给听众带来耳目一新的感觉。 Trino Impala在音乐上展现了他们对各种音乐流派的深入理解和掌握。他们的歌曲旋律优美动人,创作风格独特。他们以实验性摇滚为基础,将各种不同的音乐元素自由地混合在一起,创造出新颖而又独特的音乐风格。 Trino Green是一位出色的吉他手和作曲家。他的吉他演奏技巧非常高超,能够将吉他的音色变化发挥到极致。他的创作充满了惊喜和创新,并经常利用各种不同的音效和技巧来丰富音乐的表达方式。 Impala Johnson则是一位才华横溢的歌手和键盘手。她的声线清澈动人,唱腔充满灵性。她的键盘演奏技巧独特,能够通过琴键传递出丰富的情感。 Trino Impala的表演气氛十分活跃和狂热。他们的舞台表演充满了能量和活力,总能让观众沉浸在音乐的世界中。他们的音乐受到了广大听众的热爱和追捧,并在国内外音乐界取得了很大的成功。 总之,Trino Impala是一支深受欢迎的音乐组合,他们以独特的音乐风格和卓越的艺术才华在音乐界崭露头角。他们的音乐充满了活力和创新,总能给听众带来美妙的音乐体验。

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值