Java代码监控Hadoop集群(支持HA集群)

最新推荐文章于 2024-07-25 09:04:29 发布
最新推荐文章于 2024-07-25 09:04:29 发布
阅读量2.4k 收藏 9
点赞数
分类专栏： Hadoop 文章标签： hadoop
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_44455388/article/details/105528894
版权
HDFS相关监控指标
public class HdfsSummary {

    //"name": "Hadoop:service=NameNode,name=NameNodeInfo"
    //总空间大小 GB
    private double total;

    //hdfs已使用的空间大小 GB
    private double dfsUsed;

    //hdfs已使用空间百分比
    private double percentUsed;

    //hdfs空闲空间 GB
    private double dfsFree;

    //hdfs是否处于safemode
    private String safeMode;

    //非hdfs空间大小 GB
    private double nonDfsUsed;

    //集群该namespace的hdfs使用容量大小
    private double blockPoolUsedSpace;

    //集群该namespace的hdfs使用容量所占百分比
    private double percentBlockPoolUsed;
    private double percentRemaining;

    //集群总的block数
    private int totalBlocks;

    //集群总的文件数
    private int totalFiles;

    //集群丢失的block数量
    private int missingBlocks;

    //处于可用状态的datanode汇总
    private List<DataNodeInfo> liveDataNodeInfos;

    //处于不可用状态的datanode汇总
    private List<DataNodeInfo> deadDataNodeInfos;


    //"name": "Hadoop:service=NameNode,name=FSNamesystemState"
    //处于可用状态的datanode数量
    private int numLiveDataNodes;

    //处于不可用状态的datanode数量
    private int numDeadDataNodes;

    //坏盘的数量
    private int volumeFailuresTotal;

    public void printInfo() {

        System.out.println("HDFS SUMMARY INFO");
        System.out.println(String
                .format("totalBlocks:%s\ntotalFiles:%s\nnumLiveDataNodes:%s", totalBlocks, totalFiles,
                        numLiveDataNodes));
        liveDataNodeInfos.forEach(node -> {
            System.out.println(
                    String.format("nodeName:%s\nnumBlocks:%s", node.getNodeName(), node.getNumBlocks()));
        });
        System.out.println("----------------------");
    }

    public double getTotal() {
        return total;
    }

    public void setTotal(double total) {
        this.total = total;
    }

    public double getDfsUsed() {
        return dfsUsed;
    }

    public void setDfsUsed(double dfsUsed) {
        this.dfsUsed = dfsUsed;
    }

    public double getPercentUsed() {
        return percentUsed;
    }

    public void setPercentUsed(double percentUsed) {
        this.percentUsed = percentUsed;
    }

    public double getDfsFree() {
        return dfsFree;
    }

    public void setDfsFree(double dfsFree) {
        this.dfsFree = dfsFree;
    }

    public String getSafeMode() {
        return safeMode;
    }

    public void setSafeMode(String safeMode) {
        this.safeMode = safeMode;
    }

    public double getNonDfsUsed() {
        return nonDfsUsed;
    }

    public void setNonDfsUsed(double nonDfsUsed) {
        this.nonDfsUsed = nonDfsUsed;
    }

    public double getBlockPoolUsedSpace() {
        return blockPoolUsedSpace;
    }

    public void setBlockPoolUsedSpace(double blockPoolUsedSpace) {
        this.blockPoolUsedSpace = blockPoolUsedSpace;
    }

    public double getPercentBlockPoolUsed() {
        return percentBlockPoolUsed;
    }

    public void setPercentBlockPoolUsed(double percentBlockPoolUsed) {
        this.percentBlockPoolUsed = percentBlockPoolUsed;
    }

    public double getPercentRemaining() {
        return percentRemaining;
    }

    public void setPercentRemaining(double percentRemaining) {
        this.percentRemaining = percentRemaining;
    }

    public int getTotalBlocks() {
        return totalBlocks;
    }

    public void setTotalBlocks(int totalBlocks) {
        this.totalBlocks = totalBlocks;
    }

    public int getTotalFiles() {
        return totalFiles;
    }

    public void setTotalFiles(int totalFiles) {
        this.totalFiles = totalFiles;
    }

    public int getMissingBlocks() {
        return missingBlocks;
    }

    public void setMissingBlocks(int missingBlocks) {
        this.missingBlocks = missingBlocks;
    }

    public List<DataNodeInfo> getLiveDataNodeInfos() {
        return liveDataNodeInfos;
    }

    public void setLiveDataNodeInfos(
            List<DataNodeInfo> liveDataNodeInfos) {
        this.liveDataNodeInfos = liveDataNodeInfos;
    }

    public List<DataNodeInfo> getDeadDataNodeInfos() {
        return deadDataNodeInfos;
    }

    public void setDeadDataNodeInfos(
            List<DataNodeInfo> deadDataNodeInfos) {
        this.deadDataNodeInfos = deadDataNodeInfos;
    }

    public int getNumLiveDataNodes() {
        return numLiveDataNodes;
    }

    public void setNumLiveDataNodes(int numLiveDataNodes) {
        this.numLiveDataNodes = numLiveDataNodes;
    }

    public int getNumDeadDataNodes() {
        return numDeadDataNodes;
    }

    public void setNumDeadDataNodes(int numDeadDataNodes) {
        this.numDeadDataNodes = numDeadDataNodes;
    }

    public int getVolumeFailuresTotal() {
        return volumeFailuresTotal;
    }

    public void setVolumeFailuresTotal(int volumeFailuresTotal) {
        this.volumeFailuresTotal = volumeFailuresTotal;
    }
}
DataNode相关监控指标
public class DataNodeInfo {

    //datanode的hostname
    private String nodeName;

    //datanode的ip地址
    private String nodeAddr;

    //datanode的上次链接数量
    private int lastContact;

    //datanode上hdfs的已用空间 GB
    private double usedSpace;

    //datanode的状态
    private String adminState;

    //datanode上非hdfs的空间大小 GB
    private double nonDfsUsedSpace;

    //datanode上的总空间大小
    private double capacity;

    //datanode的block
    private int numBlocks;
    private double remaining;
    private double blockPoolUsed;
    private double blockPoolUsedPerent;

    public String getNodeName() {
        return nodeName;
    }

    public void setNodeName(String nodeName) {
        this.nodeName = nodeName;
    }

    public String getNodeAddr() {
        return nodeAddr;
    }

    public void setNodeAddr(String nodeAddr) {
        this.nodeAddr = nodeAddr;
    }

    public int getLastContact() {
        return lastContact;
    }

    public void setLastContact(int lastContact) {
        this.lastContact = lastContact;
    }

    public double getUsedSpace() {
        return usedSpace;
    }

    public void setUsedSpace(double usedSpace) {
        this.usedSpace = usedSpace;
    }

    public String getAdminState() {
        return adminState;
    }

    public void setAdminState(String adminState) {
        this.adminState = adminState;
    }

    public double getNonDfsUsedSpace() {
        return nonDfsUsedSpace;
    }

    public void setNonDfsUsedSpace(double nonDfsUsedSpace) {
        this.nonDfsUsedSpace = nonDfsUsedSpace;
    }

    public double getCapacity() {
        return capacity;
    }

    public void setCapacity(double capacity) {
        this.capacity = capacity;
    }

    public int getNumBlocks() {
        return numBlocks;
    }

    public void setNumBlocks(int numBlocks) {
        this.numBlocks = numBlocks;
    }

    public double getRemaining() {
        return remaining;
    }

    public void setRemaining(double remaining) {
        this.remaining = remaining;
    }

    public double getBlockPoolUsed() {
        return blockPoolUsed;
    }

    public void setBlockPoolUsed(double blockPoolUsed) {
        this.blockPoolUsed = blockPoolUsed;
    }

    public double getBlockPoolUsedPerent() {
        return blockPoolUsedPerent;
    }

    public void setBlockPoolUsedPerent(double blockPoolUsedPerent) {
        this.blockPoolUsedPerent = blockPoolUsedPerent;
    }
}
JsonUtil工具类
public class JsonUtil {

    private static ObjectMapper objMapper = new ObjectMapper();

    static {
        objMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
        objMapper.configure(DeserializationFeature.FAIL_ON_IGNORED_PROPERTIES, false);
    }

    public static String toJson(Object obj) throws IOException {
        return objMapper.writeValueAsString(obj);
    }

    public static String toJson(Object obj, boolean prettyFormat) throws IOException {
        String json = objMapper.writeValueAsString(obj);
        if (prettyFormat) {
            Object jsonObj = objMapper.readValue(json, Object.class);
            json = objMapper.writerWithDefaultPrettyPrinter().writeValueAsString(jsonObj);
        }
        return json;
    }

    public static <T> T fromJson(Class<T> clazz, String json) throws IOException {
        return objMapper.readValue(json, clazz);
    }

    public static <T> List<T> fromJsonList(Class<T> clazz, String json) throws IOException {
        return objMapper.readValue(json, objMapper.getTypeFactory()
                .constructCollectionType(List.class, clazz));
    }

    public static <K, V> Map<K, V> fromJsonMap(Class<K> keyClazz, Class<V> valueClazz, String json)
            throws IOException{
        return objMapper.readValue(json, objMapper.getTypeFactory()
                .constructMapType(HashMap.class, keyClazz, valueClazz));
    }

}
StatefulHttpClient
public class StatefulHttpClient {

    private HttpClientContext context;
    private CloseableHttpClient httpclient;
    private int requestTimeOut = 15;
    private int sessionTimeOut = 60;

    public StatefulHttpClient(int sessionTimeOut, int requestTimeOut,
                              HttpHost proxy) {
        initCookieStore();
        this.sessionTimeOut = sessionTimeOut;
        this.requestTimeOut = requestTimeOut;
        RequestConfig.Builder requestConfigBuilder = RequestConfig.custom()
                .setConnectTimeout(this.requestTimeOut * 1000).setSocketTimeout(this.requestTimeOut * 1000);
        if (proxy != null) {
            requestConfigBuilder.setProxy(proxy);
        }
        httpclient = HttpClientBuilder.create()
                .setDefaultRequestConfig(requestConfigBuilder.build()).build();
    }

    public StatefulHttpClient(HttpHost proxy) {
        initCookieStore();
        RequestConfig.Builder requestConfigBuilder = RequestConfig.custom()
                .setConnectTimeout(10000).setSocketTimeout(this.requestTimeOut * 1000);
        if (proxy != null) {
            requestConfigBuilder.setProxy(proxy);
        }
        httpclient = HttpClientBuilder.create()
                .setDefaultRequestConfig(requestConfigBuilder.build()).build();
    }

    public int getRequestTimeOut() {
        return requestTimeOut;
    }

    public void setRequestTimeOut(int requestTimeOut) {
        this.requestTimeOut = requestTimeOut;
    }

    public int getSessionTimeOut() {
        return sessionTimeOut;
    }

    public void setSessionTimeOut(int sessionTimeOut) {
        this.sessionTimeOut = sessionTimeOut;
    }

    private void initCookieStore() {
        PublicSuffixMatcher publicSuffixMatcher = PublicSuffixMatcherLoader.getDefault();
        Registry<CookieSpecProvider> cookieSpecReg = RegistryBuilder.<CookieSpecProvider>create()
                .register(CookieSpecs.DEFAULT, new DefaultCookieSpecProvider(publicSuffixMatcher))
                .register(CookieSpecs.STANDARD, new RFC6265CookieSpecProvider(publicSuffixMatcher)).build();
        CookieStore cookieStore = new BasicCookieStore();

        context = HttpClientContext.create();
        context.setCookieSpecRegistry(cookieSpecReg);
        context.setCookieStore(cookieStore);
    }


    /**
     * http get.
     *
     * @param clazz clazz
     * @param url url
     * @param queryParam queryParam
     * @param headers headers
     * @param <T> T
     * @return T
     * @throws IOException e
     */
    @SuppressWarnings("unchecked")
    public <T> T get(Class<T> clazz, String url,
                     Map<String, String> queryParam, Map<String, String> headers) throws IOException {
        HttpGet get = new HttpGet(url);
        if (headers != null && headers.size() > 0) {
            for (String key : headers.keySet()) {
                get.setHeader(key, headers.get(key));
            }
        }
        try {
            if (queryParam != null && queryParam.size() > 0) {
                URIBuilder builder = new URIBuilder(get.getURI());
                for (String key : queryParam.keySet()) {
                    builder.addParameter(key, queryParam.get(key));
                }
                get.setURI(builder.build());
            }

        } catch (Exception exception) {
            exception.printStackTrace();

        }
        HttpEntity entity = null;
        try {
            HttpResponse response = httpclient.execute(get, context);
            int statusCode = response.getStatusLine().getStatusCode() / 100;
            entity = response.getEntity();
            String result = EntityUtils.toString(response.getEntity());
            if (statusCode == 4 || statusCode == 5) {
                throw new IOException(result);
            }

            if (String.class.equals(clazz)) {
                return (T) result;
            }
            if (result == null || result.length() == 0) {
                return null;
            }

            return JsonUtil.fromJson(clazz, result);
        } catch (Exception e) {
        	//active宕机，会抛出异常，这时访问standBy
            T t = getStandBy(clazz, "http://192.168.72.142:50070/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo", null, null);
            return t;
        }
    }


    /**
     * http get.
     *
     * @param clazz clazz
     * @param url url
     * @param queryParam queryParam
     * @param headers headers
     * @param <T> T
     * @return T
     * @throws IOException e
     */
    @SuppressWarnings("unchecked")
    public <T> T getStandBy(Class<T> clazz, String url,
                     Map<String, String> queryParam, Map<String, String> headers) throws IOException {
        HttpGet get = new HttpGet(url);
        if (headers != null && headers.size() > 0) {
            for (String key : headers.keySet()) {
                get.setHeader(key, headers.get(key));
            }
        }
        try {
            if (queryParam != null && queryParam.size() > 0) {
                URIBuilder builder = new URIBuilder(get.getURI());
                for (String key : queryParam.keySet()) {
                    builder.addParameter(key, queryParam.get(key));
                }
                get.setURI(builder.build());
            }

        } catch (Exception exception) {
            exception.printStackTrace();

        }
        HttpEntity entity = null;
        try {
            HttpResponse response = httpclient.execute(get, context);
            int statusCode = response.getStatusLine().getStatusCode() / 100;
            entity = response.getEntity();
            String result = EntityUtils.toString(response.getEntity());
            if (statusCode == 4 || statusCode == 5) {
                throw new IOException(result);
            }

            if (String.class.equals(clazz)) {
                return (T) result;
            }
            if (result == null || result.length() == 0) {
                return null;
            }

            return JsonUtil.fromJson(clazz, result);
        } finally {
            if (entity != null) {
                EntityUtils.consume(entity);
            }
        }
    }


    /**
     * http post.
     *
     * @param clazz clazz
     * @param url url
     * @param queryParam queryParam
     * @param headers headers
     * @param <T> T
     * @return T
     * @throws IOException e
     */
    @SuppressWarnings("unchecked")
    public <T> T post(Class<T> clazz, String url,
                      Map<String, String> queryParam, Map<String, String> headers,
                      String body) throws IOException {

        HttpPost post = new HttpPost(url);
        if (headers != null && headers.size() > 0) {
            for (String key : headers.keySet()) {
                post.setHeader(key, headers.get(key));
            }
        }
        try {
            if (queryParam != null && queryParam.size() > 0) {
                URIBuilder builder = new URIBuilder(post.getURI());
                for (String key : queryParam.keySet()) {
                    builder.addParameter(key, queryParam.get(key));
                }
                post.setURI(builder.build());
            }
        } catch (Exception exception) {
            exception.printStackTrace();
        }

        if (body != null) {
            HttpEntity entity = new StringEntity(body,
                    ContentType.APPLICATION_JSON);
            post.setEntity(entity);
        }
        HttpEntity entity = null;
        try {
            HttpResponse response = httpclient.execute(post, context);
            int statusCode = response.getStatusLine().getStatusCode() / 100;
            entity = response.getEntity();
            String result = EntityUtils.toString(response.getEntity());
            if (statusCode == 4 || statusCode == 5) {
                throw new IOException(result);
            }
            if (Void.class.equals(clazz)) {
                return null;
            }
            if (String.class.equals(clazz)) {
                return (T) result;
            }
            if (result == null || result.length() == 0) {
                return null;
            }

            return JsonUtil.fromJson(clazz, result);
        } finally {
            if (entity != null) {
                EntityUtils.consume(entity);
            }
        }
    }

    /**
     * http upload.
     *
     * @param clazz clazz
     * @param url url
     * @param queryParam queryParam
     * @param headers headers
     * @param filePaths filepaths
     * @param <T> T
     * @return T
     * @throws IOException e
     */
    @SuppressWarnings("unchecked")
    public <T> T upload(Class<T> clazz, String url,
                        Map<String, String> queryParam, Map<String, String> headers, List<String> filePaths)
            throws IOException {

        HttpPost post = new HttpPost(url);
        if (headers != null && headers.size() > 0) {
            for (String key : headers.keySet()) {
                post.setHeader(key, headers.get(key));
            }
        }
        try {
            if (queryParam != null && queryParam.size() > 0) {
                URIBuilder builder = new URIBuilder(post.getURI());
                for (String key : queryParam.keySet()) {
                    builder.addParameter(key, queryParam.get(key));
                }
                post.setURI(builder.build());
            }
        } catch (Exception exception) {
            exception.printStackTrace();
        }

        if (filePaths != null) {
            MultipartEntityBuilder builder = MultipartEntityBuilder.create();
            builder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);
            for (String path : filePaths) {
                File file = new File(path);
                builder.addBinaryBody("file", file, ContentType.DEFAULT_BINARY, file.getName());
            }
            HttpEntity entity = builder.build();
            post.setEntity(entity);
        }

        HttpEntity entity = null;
        try {
            HttpResponse response = httpclient.execute(post, context);
            int statusCode = response.getStatusLine().getStatusCode() / 100;
            entity = response.getEntity();
            String result = EntityUtils.toString(response.getEntity());
            if (statusCode == 4 || statusCode == 5) {
                throw new IOException(result);
            }
            if (String.class.equals(clazz)) {
                return (T) result;
            }
            if (result == null || result.length() == 0) {
                return null;
            }

            return JsonUtil.fromJson(clazz, result);
        } finally {
            if (entity != null) {
                EntityUtils.consume(entity);
            }
        }
    }

    /**
     * http download.
     *
     * @param url url
     * @param queryParam queryParam
     * @param headers headers
     * @param filePath filepath
     * @throws IOException e
     */
    @SuppressWarnings("unchecked")
    public void download(String url,
                         Map<String, String> queryParam, Map<String, String> headers, String filePath)
            throws IOException {

        HttpPost post = new HttpPost(url);
        if (headers != null && headers.size() > 0) {
            for (String key : headers.keySet()) {
                post.setHeader(key, headers.get(key));
            }
        }
        try {
            if (queryParam != null && queryParam.size() > 0) {
                URIBuilder builder = new URIBuilder(post.getURI());
                for (String key : queryParam.keySet()) {
                    builder.addParameter(key, queryParam.get(key));
                }
                post.setURI(builder.build());
            }
        } catch (Exception exception) {
            exception.printStackTrace();
        }

        HttpEntity entity = null;
        try {
            HttpResponse response = httpclient.execute(post, context);
            int statusCode = response.getStatusLine().getStatusCode() / 100;
            entity = response.getEntity();
            if (statusCode == 4 || statusCode == 5) {
                throw new IOException("");
            }
            BufferedInputStream bis = new BufferedInputStream(entity.getContent());
            BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(new File(filePath)));
            int inByte;
            while ((inByte = bis.read()) != -1) {
                bos.write(inByte);
            }
            bis.close();
            bos.close();
        } finally {
            if (entity != null) {
                EntityUtils.consume(entity);
            }
        }
    }

    /**
     * http downloadzip.
     *
     * @param url url
     * @param queryParam queryParam
     * @param headers headers
     * @param body body
     * @param filePath filepath
     * @throws IOException e
     */
    @SuppressWarnings("unchecked")
    public void downloadzip(String url,
                            Map<String, String> queryParam, Map<String, String> headers, String body, String filePath)
            throws IOException {

        HttpPost post = new HttpPost(url);
        if (headers != null && headers.size() > 0) {
            for (String key : headers.keySet()) {
                post.setHeader(key, headers.get(key));
            }
        }
        try {
            if (queryParam != null && queryParam.size() > 0) {
                URIBuilder builder = new URIBuilder(post.getURI());
                for (String key : queryParam.keySet()) {
                    builder.addParameter(key, queryParam.get(key));
                }
                post.setURI(builder.build());
            }
        } catch (Exception exception) {
            exception.printStackTrace();
        }
        if (body != null) {
            HttpEntity entity = new StringEntity(body,
                    ContentType.APPLICATION_JSON);
            post.setEntity(entity);
        }
        HttpEntity entity = null;
        try {
            HttpResponse response = httpclient.execute(post, context);
            int statusCode = response.getStatusLine().getStatusCode() / 100;
            entity = response.getEntity();
            if (statusCode == 4 || statusCode == 5) {
                throw new IOException("");
            }
            BufferedInputStream bis = new BufferedInputStream(entity.getContent());
            BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(new File(filePath)));
            int inByte;
            while ((inByte = bis.read()) != -1) {
                bos.write(inByte);
            }
            bis.close();
            bos.close();
        } finally {
            if (entity != null) {
                EntityUtils.consume(entity);
            }
        }
    }

    /**
     * http delete.
     *
     * @param clazz clazz
     * @param url url
     * @param queryParam queryParam
     * @param headers headers
     * @param <T> T
     * @return T
     * @throws IOException e
     */
    @SuppressWarnings("unchecked")
    public <T> T delete(Class<T> clazz, String url,
                        Map<String, String> queryParam, Map<String, String> headers,
                        String body) throws IOException {

        GavialHttpDelete delete = new GavialHttpDelete(url);
        if (headers != null && headers.size() > 0) {
            for (String key : headers.keySet()) {
                delete.setHeader(key, headers.get(key));
            }
        }
        try {
            if (queryParam != null && queryParam.size() > 0) {
                URIBuilder builder = new URIBuilder(delete.getURI());
                for (String key : queryParam.keySet()) {
                    builder.addParameter(key, queryParam.get(key));
                }
                delete.setURI(builder.build());
            }

        } catch (Exception exception) {
            exception.printStackTrace();

        }
        if (body != null) {
            HttpEntity entity = new StringEntity(body,
                    ContentType.APPLICATION_JSON);
            delete.setEntity(entity);
        }
        HttpEntity entity = null;
        try {
            HttpResponse response = httpclient.execute(delete, context);
            int statusCode = response.getStatusLine().getStatusCode() / 100;
            entity = response.getEntity();
            String result = EntityUtils.toString(response.getEntity());
            if (statusCode == 4 || statusCode == 5) {
                throw new IOException(result);
            }

            if (String.class.equals(clazz)) {
                return (T) result;
            }
            if (result == null || result.length() == 0) {
                return null;
            }

            return JsonUtil.fromJson(clazz, result);
        } finally {
            if (entity != null) {
                EntityUtils.consume(entity);
            }
        }
    }

    private static class GavialHttpDelete extends HttpEntityEnclosingRequestBase {

        static final String METHOD_NAME = "DELETE";

        GavialHttpDelete(final String uri) {
            super();
            setURI(URI.create(uri));
        }

        public GavialHttpDelete(final URI uri) {
            super();
            setURI(uri);
        }

        public GavialHttpDelete() {
            super();
        }

        public String getMethod() {
            return METHOD_NAME;
        }
    }

}
获取到的json数据
public class MonitorMetrics {
    // beans为通过jmx所返回的json串中最起始的key
    // 结构为{"beans":[{"":"","":"",...}]}
    List<Map<String, Object>> beans = new ArrayList<>();

    public List<Map<String, Object>> getBeans() {
        return beans;
    }

    public void setBeans(List<Map<String, Object>> beans) {
        this.beans = beans;
    }

    public Object getMetricsValue(String name) {
        if (beans.isEmpty()) {
            return null;
        }
        return beans.get(0).getOrDefault(name, null);
    }
}
HadoopUtil
public class HadoopUtil {

    public static long mbLength = 1048576L;
    public static long gbLength = 1073741824L;
    //active状态的nameNode地址
    public static final String hadoopJmxServerUrl = "http://192.168.72.141:50070";
    public static final String jmxServerUrlFormat = "%s/jmx?qry=%s";
    public static final String nameNodeInfo = "Hadoop:service=NameNode,name=NameNodeInfo";
    public static final String fsNameSystemState = "Hadoop:service=NameNode,name=FSNamesystemState";

    public static HdfsSummary getHdfsSummary(StatefulHttpClient client) throws IOException {
        HdfsSummary hdfsSummary = new HdfsSummary();
        String namenodeUrl = String.format(jmxServerUrlFormat, hadoopJmxServerUrl, nameNodeInfo);
        MonitorMetrics monitorMetrics = client.get(MonitorMetrics.class, namenodeUrl, null, null);
        hdfsSummary.setTotal(doubleFormat(monitorMetrics.getMetricsValue("Total"), gbLength));
        hdfsSummary.setDfsFree(doubleFormat(monitorMetrics.getMetricsValue("Free"), gbLength));
        hdfsSummary.setDfsUsed(doubleFormat(monitorMetrics.getMetricsValue("Used"), gbLength));
        hdfsSummary.setPercentUsed(doubleFormat(monitorMetrics.getMetricsValue("PercentUsed")));
        hdfsSummary.setSafeMode(monitorMetrics.getMetricsValue("Safemode").toString());
        hdfsSummary.setNonDfsUsed(doubleFormat(monitorMetrics.getMetricsValue("NonDfsUsedSpace"), gbLength));
        hdfsSummary.setBlockPoolUsedSpace(doubleFormat(monitorMetrics.getMetricsValue("BlockPoolUsedSpace"), gbLength));
        hdfsSummary.setPercentBlockPoolUsed(doubleFormat(monitorMetrics.getMetricsValue("PercentBlockPoolUsed")));
        hdfsSummary.setPercentRemaining(doubleFormat(monitorMetrics.getMetricsValue("PercentRemaining")));
        hdfsSummary.setTotalBlocks((int) monitorMetrics.getMetricsValue("TotalBlocks"));
        hdfsSummary.setTotalFiles((int) monitorMetrics.getMetricsValue("TotalFiles"));
        hdfsSummary.setMissingBlocks((int) monitorMetrics.getMetricsValue("NumberOfMissingBlocks"));

        String liveNodesJson = monitorMetrics.getMetricsValue("LiveNodes").toString();
        String deadNodesJson = monitorMetrics.getMetricsValue("DeadNodes").toString();
        List<DataNodeInfo> liveNodes = dataNodeInfoReader(liveNodesJson);
        List<DataNodeInfo> deadNodes = dataNodeInfoReader(deadNodesJson);
        hdfsSummary.setLiveDataNodeInfos(liveNodes);
        hdfsSummary.setDeadDataNodeInfos(deadNodes);

        String fsNameSystemStateUrl = String.format(jmxServerUrlFormat, hadoopJmxServerUrl, fsNameSystemState);
        MonitorMetrics hadoopMetrics = client.get(MonitorMetrics.class, fsNameSystemStateUrl, null, null);
        hdfsSummary.setNumLiveDataNodes((int) hadoopMetrics.getMetricsValue("NumLiveDataNodes"));
        hdfsSummary.setNumDeadDataNodes((int) hadoopMetrics.getMetricsValue("NumDeadDataNodes"));
        hdfsSummary.setVolumeFailuresTotal((int) hadoopMetrics.getMetricsValue("VolumeFailuresTotal"));

        return hdfsSummary;
    }

    public static List<DataNodeInfo> dataNodeInfoReader(String jsonData) throws IOException {
        List<DataNodeInfo> dataNodeInfos = new ArrayList<DataNodeInfo>();
        Map<String, Object> nodes = JsonUtil.fromJsonMap(String.class, Object.class, jsonData);
        for (Map.Entry<String, Object> node : nodes.entrySet()) {
            Map<String, Object> info = (HashMap<String, Object>) node.getValue();
            String nodeName = node.getKey().split(":")[0];
            DataNodeInfo dataNodeInfo = new DataNodeInfo();
            dataNodeInfo.setNodeName(nodeName);
            dataNodeInfo.setNodeAddr(info.get("infoAddr").toString().split(":")[0]);
            dataNodeInfo.setLastContact((int) info.get("lastContact"));
            dataNodeInfo.setUsedSpace(doubleFormat(info.get("usedSpace"), gbLength));
            dataNodeInfo.setAdminState(info.get("adminState").toString());
            dataNodeInfo.setNonDfsUsedSpace(doubleFormat(info.get("nonDfsUsedSpace"), gbLength));
            dataNodeInfo.setCapacity(doubleFormat(info.get("capacity"), gbLength));
            dataNodeInfo.setNumBlocks((int) info.get("numBlocks"));
            dataNodeInfo.setRemaining(doubleFormat(info.get("remaining"), gbLength));
            dataNodeInfo.setBlockPoolUsed(doubleFormat(info.get("blockPoolUsed"), gbLength));
            dataNodeInfo.setBlockPoolUsedPerent(doubleFormat(info.get("blockPoolUsedPercent")));

            dataNodeInfos.add(dataNodeInfo);
        }

        return dataNodeInfos;
    }

    public static DecimalFormat df = new DecimalFormat("#.##");

    public static double doubleFormat(Object num, long unit) {
        double result = Double.parseDouble(String.valueOf(num)) / unit;
        return Double.parseDouble(df.format(result));
    }

    public static double doubleFormat(Object num) {
        double result = Double.parseDouble(String.valueOf(num));
        return Double.parseDouble(df.format(result));
    }
}
测试:
public class MonitorApp {

    public static void main(String[] args) throws IOException, JSONException {

        //查询Hadoop信息
        StatefulHttpClient client = new StatefulHttpClient(null);
        HdfsSummary hdfsSummary = HadoopUtil.getHdfsSummary(client);
        hdfsSummary.printInfo();

        //查询HBASE信息
//        HbaseSummary hbaseSummary = HBaseUtil.getHbaseSummary(client);
//        hbaseSummary.printInfo();

        //查询application的信息
//        List<YarnApplication> list = YarnUtils.fetchApp();
//        for (YarnApplication yarnApplication:list){
//            System.out.println(yarnApplication.toString());
//        }

        //查询yarnClusterMetrics信息
//        YarnClusterMetrics yarnClusterMetrics = YarnUtils.fetchClusterMetrics(client);
//        Integer numActiveNMs = yarnClusterMetrics.getNumActiveNMs();
//        System.out.println(numActiveNMs);

        //查询队列资源使用情况
//        QueueMetrics queueMetrics = YarnUtils.fetchQueueMetrics(client);
//        Integer availableMB = queueMetrics.getAvailableMB();
//        System.out.println(availableMB);
    }
}
我在北国不背锅
关注
专栏目录
java 监控hadoop webui数据
MengShi_的博客
10-31 366
import java.io.*; import java.text.SimpleDateFormat; import com.alibaba.fastjson.JSONArray; import com.alibaba.fastjson.JSONObject; import javafx.scene.chart.PieChart; import org.apache.commons.http...
java 连接hadoop集群_JAVA API连接HDFS HA集群
weixin_39815286的博客
02-25 1074
使用JAVA API连接HDFS时我们需要使用NameNode的地址,开启HA后,两个NameNode可能会主备切换,如果连接的那台主机NameNode挂掉了,连接就会失败. HDFS提供了nameservices的方式进行访问,这样只要有一个NameNode活着,都可以正常访问.HDFS NameNode HA在没有HA的环境中,通常使用NameNode hostname访问HDFS的URL.h...
java程序hadoop集群监控,java api操作远程hadoop集群
weixin_34221197的博客
03-22 300
先给个在集群根目录下建立test目录的例子：Stringuri="hdfs://ip地址:端口号";Configurationconf = newConfiguration();FileSystemfs = FileSystem.get(URI.create(uri),conf);StringpathString = "/test";booleanexists = fs.exists(newPat...
Hadoop 重要监控指标
最新发布
weixin_41489908的博客
07-25 826
NameNode 指标：包括 NameNode 的状态、内存使用、文件系统操作等。DataNode 指标：包括 DataNode 的状态、磁盘使用、数据块操作等。ResourceManager 指标：包括 ResourceManager 的状态、应用程序队列、资源使用等。NodeManager 指标：包括 NodeManager 的状态、容器使用、资源分配等。JobHistoryServer 指标：包括历史作业的状态、执行时间、资源消耗等。
java程序hadoop集群监控_hadoop集群监控工具ambari安装
weixin_36128337的博客
03-01 246
Apache Ambari是对Hadoop进行监控、管理和生命周期管理的开源项目。它也是一个为Hortonworks数据平台选择管理组建的项目。Ambari向Hadoop MapReduce、HDFS、 HBase、Pig, Hive、HCatalog以及Zookeeper提供服务。最近准备装ambari，在网上找了许久，没找到比较系统的ambari安装过程，于是，就根据官网进行了安装，下面是我推...
集群监控 java_Java 集群高可用监控(结合阿里SLB)脚本
weixin_39930557的博客
02-12 45
欢迎点评，大家一起来优化计划思路：　　只有在mysql slave java 进程状态都正常的情况下才允许nginx 运行， 否则就干掉它， 负载用的是阿里的SLB#bin/bash#邮件函数 我怕看不见就定义了发送十条mysql_kill_ngx(){ps -ef|grep nginx|grep -v grep |awk '{print $2}'|xargs -i{} kill -9 {}p...
java编写监控redis集群,Redis集群监控RedisClusterManager
weixin_39821378的博客
03-12 219
Redis集群监控RedisClusterManagerRedisClusterManager监控Redis集群1、集群监控功能2、集群状态查询3、集群节点管理功能从节点重新设置主节点主从切换槽迁移4、集群数据管理创建集群1、打开Manager首页，点击右上角的Create Cluster2、若没有在集群中注册redis打包文件，则点击Import Resource,并且上传Resource3、若...
java 连接hadoop集群_hadoop集群访问——Hadoop客户端访问、Java API访问
weixin_35724881的博客
02-13 1466
上一篇中介绍了hadoop集群搭建方式，本文介绍集群的访问。集群的访问方式有两种：hadoop客户端访问，Java API访问。一、集群客户端访问Hadoop采用C/S架构，可以通过客户端对集群进行操作，其实在前面搭建的集群环境中，每个集群节点都可以作为一个客户端进行集群访问，但是一般场景下，会将集群服务器作为整体，从外部设置客户端对集群进行访问。为了能从集群服务器外部访问，需要一台与集群服务器在...
HadoopHA集群部署、规划HadoopHA集群教学课件.pptx
06-20
【Hadoop HA 集群部署详解】 在大数据领域，Hadoop HA（高可用性）是确保服务持续可用的关键技术，特别是在...正确配置和管理这些组件，能确保Hadoop集群在面对单点故障时仍能提供稳定的服务，提高整体系统的可靠性。
hadoop集群信息
01-11
搭建高可用的Hadoop集群，基于NFS共享磁盘的namenode配置，使用zookeeper进行主节点推举
hadoop——HA集群部署
m0_65809235的博客
06-28 1268
各节点布置NameNodeDataNodeNameNodeDataNode。
Hadoop1.2.1配置Ganlia监控
02-26
Hadoop1.2.1配置Ganlia监控
基于SpringBoot的Hadoop-Yarn资源监控系统源码.zip
05-24
基于SpringBoot的Hadoop-Yarn资源监控系统源码.zip
【生活现场】从生日请客到hdfs工作原理解析
yeluoxiang的专栏
01-14 398
郑重声明：漫画人物和故事情节均是原创，未经同意禁止任何形式的转载，侵权将进入法律程序。大佬除外。欢迎大家转发文章，谢谢！小史是一个非科班的程序员，虽然学的是电子专业，但是...
hadoop-集群管理（2）——内存设置
weixin_34415923的博客
03-08 375
1. 内存 hadoop为各个守护进程（namenode,secondarynamenode,jobtracker,datanode,tasktracker）统一分配的内存在hadoop-env.sh中设置，参数为HADOOP_HEAPSIZE，默认为1000M。 大部分情况下，这个统一设置的值可能并不适合。例如对于namenode节点，1000M的内存只能存储几百万个文件的数据块的引用。如果...
Yarn metrics参数说明
骑猪喝咖啡
08-01 4680
获取Yarn jmx信息： curl -i http://node01:8088/jmx Hadoop:service=ResourceManager,name=FSOpDurations Hadoop:service=ResourceManager,name=JvmMetrics Hadoop:service=ResourceManager,name=ClusterMetri
安装hadoop时第一次启动datanode成功，以后启动均失败的解决方法
hl156的博客
03-21 2466
1.去查看datanode的日志. 如果提示JAVA加载路径错误，则去配置JAVA的环境配置。 若提示的错误是：Java.io.IOException:Incompatible clusterIDs 则是因为datanode与namenode中的clusterID匹配，具体原因是，每次格式化namenode都会生成一个新的clusterID, 如果只格式化了namenode，没有格式化此datanode， 就会出现”java.io.IOException: Incompatible namespaceI
hadoop常见错误及解决办法！
热门推荐
沉底的石头
05-30 1万+
1：Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out  Answer： 程序里面需要打开多个文件，进行分析，系统一般默认数量是1024，（用ulimit -a可以看到）对于正常使用是够了，但是对于程序来讲，就太少了。 修改办法： 修改2个文件。        /etc/security/limits.
解决Hadoop集群hdfs无法启动DataNode的问题
brokyz
03-25 4776
问题描述： 在hadoop启动hdfs的之后，使用jps命令查看运行情况时发现hdfs的DataNode并没有打开。 笔者出现此情况前曾使用hdfs namenode -format格式化了hdfs 如有三个hadoop集群，分别为hadoop102，hadoop103，hadoop104 其问题情况如下 可见三个机器均没有将DataNode启动。 问题分析： 进入hadoop根目录的logs中查看DataNode的日志文件 可以看见报错信息 java.io.IOException: Incompa
Hadoop2.2.0集群HA配置：QJM实现
在启动Hadoop集群时，需要启动JournalNode、ZooKeeper、NameNode（一个作为Active，一个作为Standby）以及DataNode。一旦系统运行，你可以通过监控工具检查NameNode的状态，以及JournalNode的同步情况。 值得注意的...
评论 11
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值