prometheus + pushgateway + grafana 监控 Flink on Yarn

大数据 专栏收录该内容
102 篇文章 16 订阅

日萌社

人工智能AI:Keras PyTorch MXNet TensorFlow PaddlePaddle 深度学习实战(不定时更新)


配置 prometheus、pushgateway 监控 Flink metrics指标

prometheus + pushgateway + grafana 监控 Flink

prometheus + pushgateway + grafana 监控 Flink on Yarn


Flink 配置 

3台机器:(注意每台机器最好都拷贝jar包和修改配置文件)
	拷贝jar包:cp /data/flink-1.10.1/opt/flink-metrics-prometheus-1.10.1.jar /data/flink-1.10.1/lib
	修改配置文件 vim /data/flink-1.10.1/conf/flink-conf.yaml
		# 这里写PushGateway的主机名host与端口号port
		metrics.reporter.promgateway.class: org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter
		metrics.reporter.promgateway.host: PushGateway的主机名host
		metrics.reporter.promgateway.port: 9091
		metrics.reporter.promgateway.jobName: myJob
		metrics.reporter.promgateway.randomJobNameSuffix: true
		metrics.reporter.promgateway.deleteOnShutdown: false
		metrics.reporter.promgateway.groupingKey: k1=v1;k2=v2
		metrics.reporter.promgateway.interval: 1 SECONDS

prometheus 配置

一台机器安装配置即可
tar zxvf prometheus-2.20.0-rc.0.linux-amd64.tar.gz
mv prometheus-2.20.0-rc.0.linux-amd64 prometheus
vim /home/hadoop/prometheus/prometheus.yml
	# A scrape configuration containing exactly one endpoint to scrape:
	# Here it's Prometheus itself.
	scrape_configs:
	  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
	  - job_name: 'prometheus'
		# metrics_path defaults to '/metrics'
		# scheme defaults to 'http'.
		static_configs:
		- targets: ['prometheus机器的IP域名:9090']
		  labels:
			instance: 'prometheus'
		
	  - job_name: 'pushgateway'
		static_configs:
		- targets: ['pushgateway机器的IP域名:9091']
		  labels:
			instance: 'pushgateway'

pushgateway 安装

一台机器安装即可	
tar zxvf pushgateway-1.2.0.linux-amd64.tar.gz	
mv pushgateway-1.2.0.linux-amd64 pushgateway

Flink on Yarn

Flink on Yarn(无需启动Flink集群,只需要把Flink程序提交到Yarn运行即可,然后在YARN UI界面查看某个程序的Flink详细页面)
	YARN UI界面:http://xxx.xx.x.x:8088/cluster
	Flink UI界面:
		先打开YARN UI界面,然后点击某个Flink on Yarn运行的某个application的详细页面,即可查看该Flink程序的页面。	
		注意还要保证所打开的页面的IP域名是正确的。

查看端口是否被占用linux
	netstat -anp |grep 端口
		看监控状态为LISTEN表示已经被占用
		没有LISTEN那一行,所以就表示没有被占用
启动 pushgateway
	cd /home/hadoop/pushgateway
	./pushgateway &(用了&之后即启动了之后再按一次回车就可以退出打印台)
	http://xxx.xx.xx.xx:9091 打开UI界面
	ps aux|grep pushgateway 
	netstat -anp |grep 9091
 	
启动 prometheus 
	cd /home/hadoop/prometheus
	./prometheus --config.file=prometheus.yml &(用了&之后即启动了之后再按一次回车就可以退出打印台)
	http://xx.xx.xx.xx:9090 打开UI界面
	ps aux|grep prometheus 
	netstat -anp |grep 9090

grafana 安装配置

安装 grafana
	tar zxvf grafana-7.1.0.linux-amd64.tar.gz
	mv grafana-7.1.0 grafana
	cd /home/hadoop/grafana

启动 grafana
	cd /home/hadoop/grafana/bin
	./grafana-server &(用了&之后即启动了之后再按一次回车就可以退出打印台)
	ps aux|grep grafana-server
	netstat -anp |grep 3000
	
登录 grafana
	http://xx.xx.xx.xx:3000 使用默认用户名密码(admin/admin)
登录报错1:
	Unauthorized
	Invalid or expired reset password code
	path=/api/user/password/reset status=400
	path=/api/user/password status=401
	
登录报错2:
	输入账号密码后又回到登录界面

解决:
	设置免密码免用户登录,并且是Admin权限登录
	vim /home/hadoop/grafana/conf/defaults.ini
		[auth.anonymous]
		#启用匿名访问
		enabled = true  #默认false
		#为未经身份验证的用户指定角色
		org_role = Admin #默认Viewer

导入JSON文件


要导入JSON文件

{

  "__inputs": [

    {

      "name": "DS_PROMETHEUS",

      "label": "prometheus",

      "description": "",

      "type": "datasource",

      "pluginId": "prometheus",

      "pluginName": "Prometheus"

    }

  ],

  "__requires": [

    {

      "type": "grafana",

      "id": "grafana",

      "name": "Grafana",

      "version": "4.2.0"

    },

    {

      "type": "panel",

      "id": "graph",

      "name": "Graph",

      "version": ""

    },

    {

      "type": "datasource",

      "id": "prometheus",

      "name": "Prometheus",

      "version": "1.0.0"

    },

    {

      "type": "panel",

      "id": "singlestat",

      "name": "Singlestat",

      "version": ""

    }

  ],

  "annotations": {

    "list": []

  },

  "editable": true,

  "gnetId": 10369,

  "graphTooltip": 0,

  "hideControls": false,

  "id": null,

  "links": [],

  "refresh": "5s",

  "rows": [

    {

      "collapse": false,

      "height": 337,

      "panels": [

        {

          "aliasColors": {},

          "bars": false,

          "datasource": "${DS_PROMETHEUS}",

          "fill": 1,

          "id": 1,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "span": 3,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "flink_jobmanager_Status_JVM_CPU_Load",

              "intervalFactor": 10,

              "legendFormat": "{{instance}}",

              "metric": "flink_jobmanager_Status_JVM_CPU_Load",

              "refId": "A",

              "step": 20

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "JobManager CPU Load",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ]

        },

        {

          "aliasColors": {},

          "bars": false,

          "datasource": "${DS_PROMETHEUS}",

          "fill": 1,

          "id": 2,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "span": 3,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "flink_taskmanager_Status_JVM_CPU_Load",

              "hide": false,

              "intervalFactor": 2,

              "legendFormat": "{{instance}}",

              "metric": "flink_taskmanager_Status_JVM_CPU_Load",

              "refId": "A",

              "step": 4

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "TaskManager CPU Load",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ]

        },

        {

          "aliasColors": {},

          "bars": false,

          "datasource": "${DS_PROMETHEUS}",

          "fill": 1,

          "id": 5,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "span": 3,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "flink_jobmanager_Status_JVM_Memory_Direct_MemoryUsed",

              "intervalFactor": 10,

              "legendFormat": "{{instance}}",

              "metric": "flink_jobmanager_Status_JVM_Memory_Direct_MemoryUsed",

              "refId": "A",

              "step": 20

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "JobManager Memory Used",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ]

        },

        {

          "aliasColors": {},

          "bars": false,

          "datasource": "${DS_PROMETHEUS}",

          "fill": 1,

          "id": 6,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "span": 3,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "flink_taskmanager_Status_JVM_Memory_Direct_MemoryUsed",

              "hide": false,

              "intervalFactor": 2,

              "legendFormat": "{{instance}}",

              "metric": "flink_taskmanager_Status_JVM_Memory_Direct_MemoryUsed",

              "refId": "A",

              "step": 4

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "TaskManager Memory Used",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ]

        }

      ],

      "repeat": null,

      "repeatIteration": null,

      "repeatRowId": null,

      "showTitle": false,

      "title": "Dashboard Row",

      "titleSize": "h6"

    },

    {

      "collapse": false,

      "height": 276,

      "panels": [

        {

          "cacheTimeout": null,

          "colorBackground": false,

          "colorValue": true,

          "colors": [

            "rgba(245, 54, 54, 0.9)",

            "rgba(255, 255, 255, 0.89)",

            "rgba(50, 172, 45, 0.97)"

          ],

          "datasource": "${DS_PROMETHEUS}",

          "format": "none",

          "gauge": {

            "maxValue": null,

            "minValue": 0,

            "show": false,

            "thresholdLabels": false,

            "thresholdMarkers": true

          },

          "hideTimeOverride": false,

          "id": 3,

          "interval": null,

          "links": [],

          "mappingType": 1,

          "mappingTypes": [

            {

              "name": "value to text",

              "value": 1

            },

            {

              "name": "range to text",

              "value": 2

            }

          ],

          "maxDataPoints": 100,

          "nullPointMode": "connected",

          "nullText": null,

          "postfix": "",

          "postfixFontSize": "50%",

          "prefix": "",

          "prefixFontSize": "50%",

          "rangeMaps": [

            {

              "from": "null",

              "text": "N/A",

              "to": "null"

            }

          ],

          "span": 3,

          "sparkline": {

            "fillColor": "rgba(31, 118, 189, 0.18)",

            "full": false,

            "lineColor": "rgb(31, 120, 193)",

            "show": true

          },

          "targets": [

            {

              "expr": "flink_jobmanager_taskSlotsAvailable",

              "hide": false,

              "intervalFactor": 2,

              "legendFormat": "",

              "metric": "flink_jobmanager_taskSlotsAvailable",

              "refId": "A",

              "step": 20

            }

          ],

          "thresholds": "",

          "title": "Taskslots available",

          "type": "singlestat",

          "valueFontSize": "80%",

          "valueMaps": [

            {

              "op": "=",

              "text": "N/A",

              "value": "null"

            }

          ],

          "valueName": "current"

        },

        {

          "cacheTimeout": null,

          "colorBackground": false,

          "colorValue": true,

          "colors": [

            "rgba(245, 54, 54, 0.9)",

            "rgba(255, 255, 255, 0.89)",

            "rgba(50, 172, 45, 0.97)"

          ],

          "datasource": "${DS_PROMETHEUS}",

          "format": "none",

          "gauge": {

            "maxValue": null,

            "minValue": 0,

            "show": false,

            "thresholdLabels": false,

            "thresholdMarkers": true

          },

          "hideTimeOverride": false,

          "id": 4,

          "interval": null,

          "links": [],

          "mappingType": 1,

          "mappingTypes": [

            {

              "name": "value to text",

              "value": 1

            },

            {

              "name": "range to text",

              "value": 2

            }

          ],

          "maxDataPoints": 100,

          "nullPointMode": "connected",

          "nullText": null,

          "postfix": "",

          "postfixFontSize": "50%",

          "prefix": "",

          "prefixFontSize": "50%",

          "rangeMaps": [

            {

              "from": "null",

              "text": "N/A",

              "to": "null"

            }

          ],

          "span": 3,

          "sparkline": {

            "fillColor": "rgba(31, 118, 189, 0.18)",

            "full": false,

            "lineColor": "rgb(31, 120, 193)",

            "show": true

          },

          "targets": [

            {

              "expr": "flink_jobmanager_taskSlotsTotal",

              "hide": false,

              "intervalFactor": 2,

              "legendFormat": "",

              "metric": "flink_jobmanager_taskSlotsTotal",

              "refId": "A",

              "step": 20

            }

          ],

          "thresholds": "",

          "title": "Taskslots total",

          "type": "singlestat",

          "valueFontSize": "80%",

          "valueMaps": [

            {

              "op": "=",

              "text": "N/A",

              "value": "null"

            }

          ],

          "valueName": "current"

        },

        {

          "cacheTimeout": null,

          "colorBackground": false,

          "colorValue": true,

          "colors": [

            "rgba(245, 54, 54, 0.9)",

            "rgba(255, 255, 255, 0.89)",

            "rgba(50, 172, 45, 0.97)"

          ],

          "datasource": "${DS_PROMETHEUS}",

          "format": "none",

          "gauge": {

            "maxValue": null,

            "minValue": 0,

            "show": false,

            "thresholdLabels": false,

            "thresholdMarkers": true

          },

          "hideTimeOverride": false,

          "id": 7,

          "interval": null,

          "links": [],

          "mappingType": 1,

          "mappingTypes": [

            {

              "name": "value to text",

              "value": 1

            },

            {

              "name": "range to text",

              "value": 2

            }

          ],

          "maxDataPoints": 100,

          "nullPointMode": "connected",

          "nullText": null,

          "postfix": "",

          "postfixFontSize": "50%",

          "prefix": "",

          "prefixFontSize": "50%",

          "rangeMaps": [

            {

              "from": "null",

              "text": "N/A",

              "to": "null"

            }

          ],

          "span": 3,

          "sparkline": {

            "fillColor": "rgba(251, 129, 76, 0.18)",

            "full": false,

            "lineColor": "rgb(193, 31, 31)",

            "show": true

          },

          "targets": [

            {

              "expr": "flink_jobmanager_numRegisteredTaskManagers",

              "hide": false,

              "intervalFactor": 2,

              "legendFormat": "",

              "metric": "flink_jobmanager_numRegisteredTaskManagers",

              "refId": "A",

              "step": 20

            }

          ],

          "thresholds": "",

          "title": "# of TaskManagers",

          "type": "singlestat",

          "valueFontSize": "80%",

          "valueMaps": [

            {

              "op": "=",

              "text": "N/A",

              "value": "null"

            }

          ],

          "valueName": "current"

        },

        {

          "cacheTimeout": null,

          "colorBackground": false,

          "colorValue": true,

          "colors": [

            "rgba(245, 54, 54, 0.9)",

            "rgba(255, 255, 255, 0.89)",

            "rgba(50, 172, 45, 0.97)"

          ],

          "datasource": "${DS_PROMETHEUS}",

          "format": "none",

          "gauge": {

            "maxValue": null,

            "minValue": 0,

            "show": false,

            "thresholdLabels": false,

            "thresholdMarkers": true

          },

          "hideTimeOverride": false,

          "id": 8,

          "interval": null,

          "links": [],

          "mappingType": 1,

          "mappingTypes": [

            {

              "name": "value to text",

              "value": 1

            },

            {

              "name": "range to text",

              "value": 2

            }

          ],

          "maxDataPoints": 100,

          "nullPointMode": "connected",

          "nullText": null,

          "postfix": "",

          "postfixFontSize": "50%",

          "prefix": "",

          "prefixFontSize": "50%",

          "rangeMaps": [

            {

              "from": "null",

              "text": "N/A",

              "to": "null"

            }

          ],

          "span": 3,

          "sparkline": {

            "fillColor": "rgba(251, 129, 76, 0.18)",

            "full": false,

            "lineColor": "rgb(193, 31, 31)",

            "show": true

          },

          "targets": [

            {

              "expr": "flink_jobmanager_numRunningJobs",

              "hide": false,

              "intervalFactor": 2,

              "legendFormat": "",

              "metric": "flink_jobmanager_numRunningJobs",

              "refId": "A",

              "step": 20

            }

          ],

          "thresholds": "",

          "title": "# of Running Jobs",

          "type": "singlestat",

          "valueFontSize": "80%",

          "valueMaps": [

            {

              "op": "=",

              "text": "N/A",

              "value": "null"

            }

          ],

          "valueName": "current"

        }

      ],

      "repeat": null,

      "repeatIteration": null,

      "repeatRowId": null,

      "showTitle": false,

      "title": "Dashboard Row",

      "titleSize": "h6"

    },

    {

      "collapse": false,

      "height": 255,

      "panels": [

        {

          "aliasColors": {},

          "bars": false,

          "datasource": "${DS_PROMETHEUS}",

          "fill": 1,

          "id": 9,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "span": 6,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "flink_taskmanager_Status_JVM_GarbageCollector_G1_Young_Generation_Time",

              "intervalFactor": 2,

              "legendFormat": "{{instance}} Young Gen Time",

              "metric": "flink_taskmanager_Status_JVM_GarbageCollector_G1_Young_Generation_Count",

              "refId": "A",

              "step": 2

            },

            {

              "expr": "flink_taskmanager_Status_JVM_GarbageCollector_G1_Old_Generation_Time",

              "intervalFactor": 2,

              "legendFormat": "{{instance}} Old Gen Time",

              "metric": "flink_taskmanager_Status_JVM_GarbageCollector_G1_Young_Generation_Count",

              "refId": "B",

              "step": 2

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "TaskManagers Garbage Collection",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ]

        },

        {

          "aliasColors": {},

          "bars": false,

          "datasource": "${DS_PROMETHEUS}",

          "fill": 1,

          "id": 10,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "span": 6,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "flink_jobmanager_Status_JVM_GarbageCollector_Copy_Time",

              "intervalFactor": 2,

              "legendFormat": "{{instance}} GC Copy Time",

              "metric": "flink_jobmanager_Status_JVM_GarbageCollector_Copy_Time",

              "refId": "A",

              "step": 2

            },

            {

              "expr": "flink_jobmanager_Status_JVM_GarbageCollector_MarkSweepCompact_Time",

              "intervalFactor": 2,

              "legendFormat": "{{instance}} GC MarkSweep Time",

              "metric": "flink_jobmanager_Status_JVM_GarbageCollector_MarkSweepCompact_Time",

              "refId": "B",

              "step": 2

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "JobManager Garbage Collection",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ]

        }

      ],

      "repeat": null,

      "repeatIteration": null,

      "repeatRowId": null,

      "showTitle": false,

      "title": "Dashboard Row",

      "titleSize": "h6"

    }

  ],

  "schemaVersion": 14,

  "style": "dark",

  "tags": [

    "flink"

  ],

  "templating": {

    "list": []

  },

  "time": {

    "from": "now-15m",

    "to": "now"

  },

  "timepicker": {

    "refresh_intervals": [

      "5s",

      "10s",

      "30s",

      "1m",

      "5m",

      "15m",

      "30m",

      "1h",

      "2h",

      "1d"

    ],

    "time_options": [

      "5m",

      "15m",

      "1h",

      "6h",

      "12h",

      "24h",

      "2d",

      "7d",

      "30d"

    ]

  },

  "timezone": "browser",

  "title": "Flink Dashboard",

  "version": 19,

  "description": "Flink dashboard using the Prometheus exporter. https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#prometheus-orgapacheflinkmetricsprometheusprometheusreporter "

}

 

  • 0
    点赞
  • 1
    评论
  • 1
    收藏
  • 一键三连
    一键三连
  • 扫一扫,分享海报

©️2021 CSDN 皮肤主题: 自定义皮肤 设计师:あずにゃん 返回首页
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、C币套餐、付费专栏及课程。

余额充值