CDH Manager API 操作Yarn资源池

最新推荐文章于 2024-08-22 09:30:20 发布
最新推荐文章于 2024-08-22 09:30:20 发布
阅读量3.5k 收藏 4
点赞数 2
分类专栏： 大数据笔记 文章标签： cdh yarn
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/shirukai/article/details/85212833
版权

CDH Manager API 操作Yarn 资源池

版本说明：CDH 5.11.0

API版本：v16

1 API 权限认证

在使用REST API 访问 CDH Manager 提供的相关接口的时候，需要进行权限认证。认证的类型是 Basic Auth。在java编程中将如下所示生成authorization，然后将该认证信息添加到请求头信息中即可。

private static String generateAuth(String userName, String password) {
    return "Basic " + new String(Base64.getEncoder().encode((userName + ":" + password).getBytes()));
}

在PostMan中测试可以通过如下设置添加认证信息

2 获取集群信息

API: http://192.168.66.168:7180/api/v16/clusters

请求类型：GET

参数：无

认证方式：Basic Auth

响应：

{
    "items": [
        {
            "name": "cluster",
            "displayName": "Cluster 1",
            "version": "CDH5",
            "fullVersion": "5.11.0",
            "maintenanceMode": false,
            "maintenanceOwners": [],
            "clusterUrl": "http://cdh-manager:7180/cmf/clusterRedirect/cluster",
            "hostsUrl": "http://cdh-manager:7180/cmf/clusterRedirect/cluster/hosts",
            "entityStatus": "CONCERNING_HEALTH",
            "uuid": "2b718bbd-d712-4b63-bd28-35b8f1e6bf35"
        }
    ]
}

3 获取Yarn的配置信息

API: http://192.168.66.168:7180/api/v16/clusters/{clusterName}/services/yarn/config

请求类型：GET

参数：

参数描述
clusterName集群名称

认证方式：无需

响应：

{
    "items": [
        {
            "name": "hdfs_service",
            "value": "hdfs",
            "sensitive": false
        },
        {
            "name": "yarn_fs_scheduled_allocations",
            "value": "{\"defaultFairSharePreemptionThreshold\":null,\"defaultFairSharePreemptionTimeout\":null,\"defaultMinSharePreemptionTimeout\":null,\"defaultQueueSchedulingPolicy\":\"fair\",\"queueMaxAMShareDefault\":null,\"queueMaxAppsDefault\":null,\"queuePlacementRules\":[{\"create\":true,\"name\":\"specified\",\"queue\":null,\"rules\":null},{\"create\":null,\"name\":\"nestedUserQueue\",\"queue\":null,\"rules\":[{\"create\":true,\"name\":\"default\",\"queue\":\"users\",\"rules\":null}]},{\"create\":null,\"name\":\"default\",\"queue\":null,\"rules\":null}],\"queues\":[{\"aclAdministerApps\":\"*\",\"aclSubmitApps\":\" \",\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"root\",\"queues\":[{\"aclAdministerApps\":\"*\",\"aclSubmitApps\":\"*\",\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"default\",\"queues\":[],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaDefaultQueryOptions\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxChildResources\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":1.0}],\"schedulingPolicy\":\"drf\",\"type\":null},{\"aclAdministerApps\":\"*\",\"aclSubmitApps\":\"*\",\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"users\",\"queues\":[],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaDefaultQueryOptions\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxChildResources\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":1.0}],\"schedulingPolicy\":\"drf\",\"type\":\"parent\"},{\"aclAdministerApps\":\"*\",\"aclSubmitApps\":\"*\",\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"analyst-srk\",\"queues\":[],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaDefaultQueryOptions\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxChildResources\":null,\"maxResources\":{\"memory\":4194304,\"vcores\":4},\"maxRunningApps\":null,\"minResources\":{\"memory\":2097152,\"vcores\":2},\"scheduleName\":\"default\",\"weight\":1.0}],\"schedulingPolicy\":\"drf\",\"type\":null}],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaDefaultQueryOptions\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxChildResources\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":1.0}],\"schedulingPolicy\":\"drf\",\"type\":null}],\"userMaxAppsDefault\":null,\"users\":[]}",
            "sensitive": false
        },
        {
            "name": "zookeeper_service",
            "value": "zookeeper",
            "sensitive": false
        }
    ]
}

将name为"yarn_fs_scheduled_allocations"的value值json化展开，得到如下信息，这些信息就是Yarn的资源池配置信息。

{
    "defaultQueueSchedulingPolicy": "fair",
    "userMaxAppsDefault": null,
    "queueMaxAppsDefault": null,
    "queueMaxAMShareDefault": null,
    "queuePlacementRules": [
        {
            "name": "specified",
            "create": true,
            "rules": null,
            "queue": null
        },
        {
            "name": "nestedUserQueue",
            "create": null,
            "rules": [
                {
                    "name": "default",
                    "create": true,
                    "rules": null,
                    "queue": "users"
                }
            ],
            "queue": null
        },
        {
            "name": "default",
            "create": null,
            "rules": null,
            "queue": null
        }
    ],
    "queues": [
        {
            "fairSharePreemptionThreshold": null,
            "queues": [
                {
                    "fairSharePreemptionThreshold": null,
                    "queues": [],
                    "aclSubmitApps": "*",
                    "schedulablePropertiesList": [
                        {
                            "impalaDefaultQueryMemLimit": null,
                            "scheduleName": "default",
                            "impalaMaxMemory": null,
                            "impalaDefaultQueryOptions": null,
                            "weight": 1,
                            "maxChildResources": null,
                            "minResources": null,
                            "impalaMaxRunningQueries": null,
                            "maxRunningApps": null,
                            "maxAMShare": null,
                            "impalaQueueTimeout": null,
                            "maxResources": null,
                            "impalaMaxQueuedQueries": null
                        }
                    ],
                    "name": "default",
                    "aclAdministerApps": "*",
                    "allowPreemptionFrom": null,
                    "type": null,
                    "fairSharePreemptionTimeout": null,
                    "minSharePreemptionTimeout": null,
                    "schedulingPolicy": "drf"
                },
                {
                    "fairSharePreemptionThreshold": null,
                    "queues": [],
                    "aclSubmitApps": "*",
                    "schedulablePropertiesList": [
                        {
                            "impalaDefaultQueryMemLimit": null,
                            "scheduleName": "default",
                            "impalaMaxMemory": null,
                            "impalaDefaultQueryOptions": null,
                            "weight": 1,
                            "maxChildResources": null,
                            "minResources": null,
                            "impalaMaxRunningQueries": null,
                            "maxRunningApps": null,
                            "maxAMShare": null,
                            "impalaQueueTimeout": null,
                            "maxResources": null,
                            "impalaMaxQueuedQueries": null
                        }
                    ],
                    "name": "users",
                    "aclAdministerApps": "*",
                    "allowPreemptionFrom": null,
                    "type": "parent",
                    "fairSharePreemptionTimeout": null,
                    "minSharePreemptionTimeout": null,
                    "schedulingPolicy": "drf"
                },
                {
                    "fairSharePreemptionThreshold": null,
                    "queues": [],
                    "aclSubmitApps": "*",
                    "schedulablePropertiesList": [
                        {
                            "impalaDefaultQueryMemLimit": null,
                            "scheduleName": "default",
                            "impalaMaxMemory": null,
                            "impalaDefaultQueryOptions": null,
                            "weight": 1,
                            "maxChildResources": null,
                            "minResources": {
                                "memory": 2097152,
                                "vcores": 2
                            },
                            "impalaMaxRunningQueries": null,
                            "maxRunningApps": null,
                            "maxAMShare": null,
                            "impalaQueueTimeout": null,
                            "maxResources": {
                                "memory": 4194304,
                                "vcores": 4
                            },
                            "impalaMaxQueuedQueries": null
                        }
                    ],
                    "name": "analyst-srk",
                    "aclAdministerApps": "*",
                    "allowPreemptionFrom": null,
                    "type": null,
                    "fairSharePreemptionTimeout": null,
                    "minSharePreemptionTimeout": null,
                    "schedulingPolicy": "drf"
                }
            ],
            "aclSubmitApps": " ",
            "schedulablePropertiesList": [
                {
                    "impalaDefaultQueryMemLimit": null,
                    "scheduleName": "default",
                    "impalaMaxMemory": null,
                    "impalaDefaultQueryOptions": null,
                    "weight": 1,
                    "maxChildResources": null,
                    "minResources": null,
                    "impalaMaxRunningQueries": null,
                    "maxRunningApps": null,
                    "maxAMShare": null,
                    "impalaQueueTimeout": null,
                    "maxResources": null,
                    "impalaMaxQueuedQueries": null
                }
            ],
            "name": "root",
            "aclAdministerApps": "*",
            "allowPreemptionFrom": null,
            "type": null,
            "fairSharePreemptionTimeout": null,
            "minSharePreemptionTimeout": null,
            "schedulingPolicy": "drf"
        }
    ],
    "defaultFairSharePreemptionThreshold": null,
    "defaultFairSharePreemptionTimeout": null,
    "defaultMinSharePreemptionTimeout": null,
    "users": []
}

从下图可以看出，根资源池root下有三个子资源池。

然后展开三个子资源池的数据，可以看到它们分别为：default、users、analyst-srk。

4 动态设置Yarn的资源池

动态设置Yarn的资源池，其实就是修改Yarn的config信息，然后刷新资源池。修改yarn的config信息的API与上述查询配置的API相同，只是想请求类型改为PUT，然后添加认证头信息即可。

如，我要添加在root资源池中，创建一个子资源池srk-test，并指定最小的资源：内存为2g核数为2核。最大的资源：内存为4g，核数为4核。其它的参数，可以参考界面设置。需要将如下信息，添加到上述的json中。

{
    "fairSharePreemptionThreshold": null,
    "queues": [],
    "aclSubmitApps": "*",
    "schedulablePropertiesList": [
        {
            "impalaDefaultQueryMemLimit": null,
            "scheduleName": "default",
            "impalaMaxMemory": null,
            "impalaDefaultQueryOptions": null,
            "weight": 1,
            "maxChildResources": null,
            "minResources": {
                "memory": 2097152,
                "vcores": 2
            },
            "impalaMaxRunningQueries": null,
            "maxRunningApps": null,
            "maxAMShare": null,
            "impalaQueueTimeout": null,
            "maxResources": {
                "memory": 4194304,
                "vcores": 4
            },
            "impalaMaxQueuedQueries": null
        }
    ],
    "name": "srk-test",
    "aclAdministerApps": "*",
    "allowPreemptionFrom": null,
    "type": null,
    "fairSharePreemptionTimeout": null,
    "minSharePreemptionTimeout": null,
    "schedulingPolicy": "drf"
}

API: http://192.168.66.168:7180/api/v16/clusters/{clusterName}/services/yarn/config

请求类型：PUT

参数：

{
    "items": [
        {
            "name": "hdfs_service",
            "value": "hdfs",
            "sensitive": false
        },
        {
            "name": "yarn_fs_scheduled_allocations_draft",
            "value": {\"defaultQueueSchedulingPolicy\": \"fair\",\"userMaxAppsDefault\": null,\"queueMaxAppsDefault\": null,\"queueMaxAMShareDefault\": null,\"queuePlacementRules\": [{\"name\": \"specified\",\"create\": true,\"rules\": null,\"queue\": null},{\"name\": \"nestedUserQueue\",\"create\": null,\"rules\": [  {\"name\": \"default\",\"create\": true,\"rules\": null,\"queue\": \"users\"  }],\"queue\": null},{\"name\": \"default\",\"create\": null,\"rules\": null,\"queue\": null}],\"queues\": [{\"fairSharePreemptionThreshold\": null,\"queues\": [  {\"fairSharePreemptionThreshold\": null,\"queues\": [],\"aclSubmitApps\": \"*\",\"schedulablePropertiesList\": [  {\"impalaDefaultQueryMemLimit\": null,\"scheduleName\": \"default\",\"impalaMaxMemory\": null,\"impalaDefaultQueryOptions\": null,\"weight\": 1,\"maxChildResources\": null,\"minResources\": null,\"impalaMaxRunningQueries\": null,\"maxRunningApps\": null,\"maxAMShare\": null,\"impalaQueueTimeout\": null,\"maxResources\": null,\"impalaMaxQueuedQueries\": null  }],\"name\": \"default\",\"aclAdministerApps\": \"*\",\"allowPreemptionFrom\": null,\"type\": null,\"fairSharePreemptionTimeout\": null,\"minSharePreemptionTimeout\": null,\"schedulingPolicy\": \"drf\"  },  {\"fairSharePreemptionThreshold\": null,\"queues\": [],\"aclSubmitApps\": \"*\",\"schedulablePropertiesList\": [  {\"impalaDefaultQueryMemLimit\": null,\"scheduleName\": \"default\",\"impalaMaxMemory\": null,\"impalaDefaultQueryOptions\": null,\"weight\": 1,\"maxChildResources\": null,\"minResources\": null,\"impalaMaxRunningQueries\": null,\"maxRunningApps\": null,\"maxAMShare\": null,\"impalaQueueTimeout\": null,\"maxResources\": null,\"impalaMaxQueuedQueries\": null  }],\"name\": \"users\",\"aclAdministerApps\": \"*\",\"allowPreemptionFrom\": null,\"type\": \"parent\",\"fairSharePreemptionTimeout\": null,\"minSharePreemptionTimeout\": null,\"schedulingPolicy\": \"drf\"  },  {\"fairSharePreemptionThreshold\": null,\"queues\": [],\"aclSubmitApps\": \"*\",\"schedulablePropertiesList\": [  {\"impalaDefaultQueryMemLimit\": null,\"scheduleName\": \"default\",\"impalaMaxMemory\": null,\"impalaDefaultQueryOptions\": null,\"weight\": 1,\"maxChildResources\": null,\"minResources\": {  \"memory\": 2097152,  \"vcores\": 2},\"impalaMaxRunningQueries\": null,\"maxRunningApps\": null,\"maxAMShare\": null,\"impalaQueueTimeout\": null,\"maxResources\": {  \"memory\": 4194304,  \"vcores\": 4},\"impalaMaxQueuedQueries\": null  }],\"name\": \"analyst-srk\",\"aclAdministerApps\": \"*\",\"allowPreemptionFrom\": null,\"type\": null,\"fairSharePreemptionTimeout\": null,\"minSharePreemptionTimeout\": null,\"schedulingPolicy\": \"drf\"  },  {\"fairSharePreemptionThreshold\": null,\"queues\": [],\"aclSubmitApps\": \"*\",\"schedulablePropertiesList\": [  {\"impalaDefaultQueryMemLimit\": null,\"scheduleName\": \"default\",\"impalaMaxMemory\": null,\"impalaDefaultQueryOptions\": null,\"weight\": 1,\"maxChildResources\": null,\"minResources\": {  \"memory\": 2097152,  \"vcores\": 2},\"impalaMaxRunningQueries\": null,\"maxRunningApps\": null,\"maxAMShare\": null,\"impalaQueueTimeout\": null,\"maxResources\": {  \"memory\": 4194304,  \"vcores\": 4},\"impalaMaxQueuedQueries\": null  }],\"name\": \"srk-test\",\"aclAdministerApps\": \"*\",\"allowPreemptionFrom\": null,\"type\": null,\"fairSharePreemptionTimeout\": null,\"minSharePreemptionTimeout\": null,\"schedulingPolicy\": \"drf\"  }],\"aclSubmitApps\": \" \",\"schedulablePropertiesList\": [  {\"impalaDefaultQueryMemLimit\": null,\"scheduleName\": \"default\",\"impalaMaxMemory\": null,\"impalaDefaultQueryOptions\": null,\"weight\": 1,\"maxChildResources\": null,\"minResources\": null,\"impalaMaxRunningQueries\": null,\"maxRunningApps\": null,\"maxAMShare\": null,\"impalaQueueTimeout\": null,\"maxResources\": null,\"impalaMaxQueuedQueries\": null  }],\"name\": \"root\",\"aclAdministerApps\": \"*\",\"allowPreemptionFrom\": null,\"type\": null,\"fairSharePreemptionTimeout\": null,\"minSharePreemptionTimeout\": null,\"schedulingPolicy\": \"drf\"}],\"defaultFairSharePreemptionThreshold\": null,\"defaultFairSharePreemptionTimeout\": null,\"defaultMinSharePreemptionTimeout\": null,\"users\": [] },
            "sensitive": false
        },
        {
            "name": "zookeeper_service",
            "value": "zookeeper",
            "sensitive": false
        }
    ]
}

认证方式：Basic Auth

响应：

{
    "items": [
        {
            "name": "hdfs_service",
            "value": "hdfs",
            "sensitive": false
        },
        {
            "name": "yarn_fs_scheduled_allocations",
            "value": "{\"defaultFairSharePreemptionThreshold\":null,\"defaultFairSharePreemptionTimeout\":null,\"defaultMinSharePreemptionTimeout\":null,\"defaultQueueSchedulingPolicy\":\"fair\",\"queueMaxAMShareDefault\":null,\"queueMaxAppsDefault\":null,\"queuePlacementRules\":[{\"create\":true,\"name\":\"specified\",\"queue\":null,\"rules\":null},{\"create\":null,\"name\":\"nestedUserQueue\",\"queue\":null,\"rules\":[{\"create\":true,\"name\":\"default\",\"queue\":\"users\",\"rules\":null}]},{\"create\":null,\"name\":\"default\",\"queue\":null,\"rules\":null}],\"queues\":[{\"aclAdministerApps\":\"*\",\"aclSubmitApps\":\" \",\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"root\",\"queues\":[{\"aclAdministerApps\":\"*\",\"aclSubmitApps\":\"*\",\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"default\",\"queues\":[],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaDefaultQueryOptions\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxChildResources\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":1.0}],\"schedulingPolicy\":\"drf\",\"type\":null},{\"aclAdministerApps\":\"*\",\"aclSubmitApps\":\"*\",\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"users\",\"queues\":[],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaDefaultQueryOptions\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxChildResources\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":1.0}],\"schedulingPolicy\":\"drf\",\"type\":\"parent\"},{\"aclAdministerApps\":\"*\",\"aclSubmitApps\":\"*\",\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"analyst-srk\",\"queues\":[],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaDefaultQueryOptions\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxChildResources\":null,\"maxResources\":{\"memory\":4194304,\"vcores\":4},\"maxRunningApps\":null,\"minResources\":{\"memory\":2097152,\"vcores\":2},\"scheduleName\":\"default\",\"weight\":1.0}],\"schedulingPolicy\":\"drf\",\"type\":null}],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaDefaultQueryOptions\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxChildResources\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":1.0}],\"schedulingPolicy\":\"drf\",\"type\":null}],\"userMaxAppsDefault\":null,\"users\":[]}",
            "sensitive": false
        },
        {
            "name": "yarn_fs_scheduled_allocations_draft",
            "value": "{\"defaultQueueSchedulingPolicy\": \"fair\",\"userMaxAppsDefault\": null,\"queueMaxAppsDefault\": null,\"queueMaxAMShareDefault\": null,\"queuePlacementRules\": [{\"name\": \"specified\",\"create\": true,\"rules\": null,\"queue\": null},{\"name\": \"nestedUserQueue\",\"create\": null,\"rules\": [  {\"name\": \"default\",\"create\": true,\"rules\": null,\"queue\": \"users\"  }],\"queue\": null},{\"name\": \"default\",\"create\": null,\"rules\": null,\"queue\": null}],\"queues\": [{\"fairSharePreemptionThreshold\": null,\"queues\": [  {\"fairSharePreemptionThreshold\": null,\"queues\": [],\"aclSubmitApps\": \"*\",\"schedulablePropertiesList\": [  {\"impalaDefaultQueryMemLimit\": null,\"scheduleName\": \"default\",\"impalaMaxMemory\": null,\"impalaDefaultQueryOptions\": null,\"weight\": 1,\"maxChildResources\": null,\"minResources\": null,\"impalaMaxRunningQueries\": null,\"maxRunningApps\": null,\"maxAMShare\": null,\"impalaQueueTimeout\": null,\"maxResources\": null,\"impalaMaxQueuedQueries\": null  }],\"name\": \"default\",\"aclAdministerApps\": \"*\",\"allowPreemptionFrom\": null,\"type\": null,\"fairSharePreemptionTimeout\": null,\"minSharePreemptionTimeout\": null,\"schedulingPolicy\": \"drf\"  },  {\"fairSharePreemptionThreshold\": null,\"queues\": [],\"aclSubmitApps\": \"*\",\"schedulablePropertiesList\": [  {\"impalaDefaultQueryMemLimit\": null,\"scheduleName\": \"default\",\"impalaMaxMemory\": null,\"impalaDefaultQueryOptions\": null,\"weight\": 1,\"maxChildResources\": null,\"minResources\": null,\"impalaMaxRunningQueries\": null,\"maxRunningApps\": null,\"maxAMShare\": null,\"impalaQueueTimeout\": null,\"maxResources\": null,\"impalaMaxQueuedQueries\": null  }],\"name\": \"users\",\"aclAdministerApps\": \"*\",\"allowPreemptionFrom\": null,\"type\": \"parent\",\"fairSharePreemptionTimeout\": null,\"minSharePreemptionTimeout\": null,\"schedulingPolicy\": \"drf\"  },  {\"fairSharePreemptionThreshold\": null,\"queues\": [],\"aclSubmitApps\": \"*\",\"schedulablePropertiesList\": [  {\"impalaDefaultQueryMemLimit\": null,\"scheduleName\": \"default\",\"impalaMaxMemory\": null,\"impalaDefaultQueryOptions\": null,\"weight\": 1,\"maxChildResources\": null,\"minResources\": {  \"memory\": 2097152,  \"vcores\": 2},\"impalaMaxRunningQueries\": null,\"maxRunningApps\": null,\"maxAMShare\": null,\"impalaQueueTimeout\": null,\"maxResources\": {  \"memory\": 4194304,  \"vcores\": 4},\"impalaMaxQueuedQueries\": null  }],\"name\": \"analyst-srk\",\"aclAdministerApps\": \"*\",\"allowPreemptionFrom\": null,\"type\": null,\"fairSharePreemptionTimeout\": null,\"minSharePreemptionTimeout\": null,\"schedulingPolicy\": \"drf\"  },  {\"fairSharePreemptionThreshold\": null,\"queues\": [],\"aclSubmitApps\": \"*\",\"schedulablePropertiesList\": [  {\"impalaDefaultQueryMemLimit\": null,\"scheduleName\": \"default\",\"impalaMaxMemory\": null,\"impalaDefaultQueryOptions\": null,\"weight\": 1,\"maxChildResources\": null,\"minResources\": {  \"memory\": 2097152,  \"vcores\": 2},\"impalaMaxRunningQueries\": null,\"maxRunningApps\": null,\"maxAMShare\": null,\"impalaQueueTimeout\": null,\"maxResources\": {  \"memory\": 4194304,  \"vcores\": 4},\"impalaMaxQueuedQueries\": null  }],\"name\": \"srk-test\",\"aclAdministerApps\": \"*\",\"allowPreemptionFrom\": null,\"type\": null,\"fairSharePreemptionTimeout\": null,\"minSharePreemptionTimeout\": null,\"schedulingPolicy\": \"drf\"  }],\"aclSubmitApps\": \" \",\"schedulablePropertiesList\": [  {\"impalaDefaultQueryMemLimit\": null,\"scheduleName\": \"default\",\"impalaMaxMemory\": null,\"impalaDefaultQueryOptions\": null,\"weight\": 1,\"maxChildResources\": null,\"minResources\": null,\"impalaMaxRunningQueries\": null,\"maxRunningApps\": null,\"maxAMShare\": null,\"impalaQueueTimeout\": null,\"maxResources\": null,\"impalaMaxQueuedQueries\": null  }],\"name\": \"root\",\"aclAdministerApps\": \"*\",\"allowPreemptionFrom\": null,\"type\": null,\"fairSharePreemptionTimeout\": null,\"minSharePreemptionTimeout\": null,\"schedulingPolicy\": \"drf\"}],\"defaultFairSharePreemptionThreshold\": null,\"defaultFairSharePreemptionTimeout\": null,\"defaultMinSharePreemptionTimeout\": null,\"users\": [] }",
            "sensitive": false
        },
        {
            "name": "zookeeper_service",
            "value": "zookeeper",
            "sensitive": false
        }
    ]
}

查看CHD manager 可以看出我们的资源池配置已经更新上去了

5 刷新动态资源池

配置更新上去之后，我们需要刷新动态资源池，才可以是资源池生效。

API: http://192.168.66.168:7180/api/v16/clusters/{clusterName}/commands/poolsRefresh

请求类型：POST

参数：无

认证方式：Basic Auth

响应：

{
    "id": 1066,
    "name": "RefreshPools",
    "startTime": "2018-10-22T08:13:58.669Z",
    "active": true,
    "clusterRef": {
        "clusterName": "cluster"
    }
}
shirukai
关注
专栏目录
通过CDH的API接口获得yarn的在线用户数
xiaoyutongxue6的博客
09-01 1338
package com.bdp.microservice.index.cluster.service.impl; import java.util.ArrayList; import java.util.Base64; import java.util.HashMap; import java.util.HashSet; import java.util.Iterator; import jav...
0354-如何使用Java调用CM的API动态配置Yarn资源池
Hadoop_SC的博客
03-08 1210
温馨提示：如果使用电脑查看图片不清晰，可以使用手机打开文章单击文中的图片放大查看高清原图。 Fayson的github： https://github.com/fayson/cdhproject 提示：代码块部分可以左右滑动查看噢 1.文档编写目的 用户在使用CDH集群大数据平台时会有需求在自己的统一管理平台上通过API接口能够动态的设置Yarn资源池，Cloudera Manager提供了...
Hadoop YARN API Python 客户端使用教程
最新发布
gitblog_00097的博客
08-22 350
Hadoop YARN API Python 客户端使用教程 项目地址:https://gitcode.com/gh_mirrors/had/hadoop-yarn-api-python-client 项目介绍 Hadoop YARN API Python 客户端是一个用于与Hadoop YARN（Yet Another Resource Negotiator）进行交互的Python库。YARN是...
CDH动态资源池--yarn资源的任务队列
北京小辉
01-03 8617
需求：1、让不同的application任务，在不同的队列中运行任务2、这些队列是并行执行的，相互不干扰目录：1、队列的配置2、队列运行命令3、队列运行展现1、队列的配置2、队列运行命令3、队列运行展现
0353-如何使用curl命令调用CM的API动态配置Yarn资源池
Hadoop_SC的博客
03-08 975
温馨提示：如果使用电脑查看图片不清晰，可以使用手机打开文章单击文中的图片放大查看高清原图。 Fayson的github： https://github.com/fayson/cdhproject 提示：代码块部分可以左右滑动查看噢 1.文档编写目的 在使用CDH集群大数据平台过程中，用户会有需求在自己的统一管理平台上通过API接口能够动态的设置Yarn资源池，Cloudera Manager...
Cloudera的yarn任务监控api
08-14
Hadoop Yarn’s API和Hadoop MapReduce’s API使用总结
Hive启用Sentry后的Yarn资源池限制配置
测试环境中，CM和CDH版本为5.15，有多个测试用户，每个用户都有各自的YARN资源池分配策略。例如，usera的作业会被自动分配到root.testa队列，userc的作业会分配到root.testb队列，而usere的作业则默认分配到root....
0426-如何在Cloudera Manager中配置Yarn放置规则
Hadoop_SC的博客
02-22 735
温馨提示：如果使用电脑查看图片不清晰，可以使用手机打开文章单击文中的图片放大查看高清原图。 Fayson的github： https://github.com/fayson/cdhproject 提示：代码块部分可以左右滑动查看噢 1.文档编写目的 Hadoop集群管理员希望能对集群Yarn作业的资源进行控制。根据不同的业务组或不同的用户，对Yarn的资源池进行划分，达到资源管控、任务管控的...
CDH Yarn 调度资源指南 - CDH6.0.x 详解
weixin_30672295的博客
01-07 559
Yarn 工作架构 最近随着集群大家开始频繁使用集群，资源调度的问题越发的凸显出来。需要更加深入的了解 yarn 资源调度的原理，以及到底在背后做了一些什么事情。 来看一下下面这张图。 yarn 里面有两个大的角色，这个也很容易在 yarn 里面的配置看到。 一个是 Resource Manager 类似于 Master 角色负责任务分配，他会往 Node Manager 分配一个 A...
CDH-Cloudera Manager API 详解
zcs2312852665的博客
01-12 1654
Cloudera Manager是一个用于管理和监控Hadoop集群的强大工具。它提供了一套REST API，使用户能够通过编程方式与Cloudera Manager进行交互，并执行各种操作，如创建集群、添加服务、配置角色等。
如何启用和配置YARN
m0_53157282的博客
09-26 1634
Apache Hadoop的YARN（Yet Another Resource Negotiator）是一个集群资源管理器，用于分配和管理集群中的资源。本文将介绍如何启用和配置YARN。
hadoop 资源配置与队列设置
YI依仔的博客
10-16 5679
yarn 集群资源设置资源 （1）yarn.nodemanager.resource.memory-mb 配置nodemanager节点的可用物理内存，默认是8192（MB） ，如这台服务器有16G 可以考虑分配12G给nodemanager，留下4g给系统内存或者其他程序。 （2）yarn.nodemanager.resource.cpu-vcores 表示该节点上YARN可使用的虚拟C...
Cloudera Manager、CDH零基础入门、线路指导
u012107806的专栏
04-11 913
原创地址：http://www.aboutyun.com/thread-9219-1-1.html 问题导读： 1.什么是cloudera CM 、CDH? 2.CDH、CM有哪些版本？ 3.CDH、CM有哪些安装方式？ 4.CDH如何开发？   我们知道cloudera CDH 是为简化hadoop的安装，也对对hadoop做了一些封装。那么我们就像尝试学习clou
CDH中yarn的动态资源池的相关配置
ChengYanan的博客
06-11 813
CDH Yarn资源队列划分管理 场景：根据不同项目或不同用户，对yarn资源队列进行划分，达到资源管控，任务管控的目的 CDH版本：5.x 配置： 1 yarn资源队列参数设置： (1)yarn.scheduler.fair.user-as-default-queue false 解释：当设置为 true 时，如果未指定池名称，Fair Scheduler 将会使用用户名作为默认...
YARN REST API
扳掘de博客
08-10 5126
先附官网地址> 查看指定队列的所有任务： GET http://<rm http address:port>/ws/v1/cluster/apps?queue=dev > 查看指定任务的详细信息： GET http://<rm http address:port>/ws/v1/cluster/apps/application_1326821518301_0005 > 监控任务： curl 'http
[Hadoop基础]-- 执行yarn命令报权限问题错误
欢迎来到我的博客，一起探索代码里的世界！
08-08 4445
# yarn application -kill application_1469066281048_9314   16/08/08 10:39:23 INFO client.RMProxy: Connecting to ResourceManager at master-27.dev.cluster.enn.cn/10.1.150.7:8032 16/08/08 10:39:24 WARN...
hadoop Yarn 编程API
weixin_34178244的博客
04-02 710
客户端编程库： 所在jar包： org.apache.hadoop.yarn.client.YarnClient 使用方法： 1 定义一个YarnClient实例： private YarnClient client； 2 构造一个Yarn客户端句柄并初始化 this.client = YarnClient.createYarnClient(); client.in...
cdh yarn nodemanager 可用资源分配
我的博客
01-24 1820
yarn resourcemanager 的webUI上显示了当前集群可以的内存资源，这个内存可以配置的， 修改yarn.nodemanager.resource.memory-mb这个参数配置
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值