2018年03月_houzhizhen

12月 11月 10月 09月 08月 07月 06月 05月 04月 03月 02月 01月

原创 hadoop-env.sh loaded twice when start namenode

hadoop-config.shhadoop-config.sh source the hadoop-env.shif [ -f "${HADOOP_CONF_DIR}/hadoop-env.sh" ]; then . "${HADOOP_CONF_DIR}/hadoop-env.sh"fihadoop-daemon.shAt top of hadoop-daemon.sh,...

2018-03-23 15:57:47 331

原创 SharedCacheManager does not start up caused by webapps/sharedcache not found in CLASSPATH

The webapp directory for the SharedCacheManager is missing and the SCM fails to start up with the following:2017-01-22 00:14:25,162 INFO org.apache.hadoop.service.AbstractService: Service SharedCa...

2018-03-21 17:50:41 299

转载 linux top命令

top linux下的任务管理器top命令是Linux下常用的性能分析工具，能够实时显示系统中各个进程的资源占用状况，类似于Windows的任务管理器。top是一个动态显示过程,即可以通过用户按键来不断刷新当前状态.如果在前台执行该命令,它将独占前台,直到用户终止该程序为止.比较准确的说,top命令提供了实时的对系统处理器的状态监视.它将显示系统中CPU最“敏感”的任务列表.该命令可以按CPU...

2018-03-20 17:44:43 555

转载 11. iostat 监视I/O子系统

11. iostat 监视I/O子系统iostat是I/O statistics（输入/输出统计）的缩写，用来动态监视系统的磁盘操作活动。11.1. 命令格式iostat[参数][时间][次数]11.2. 命令功能通过iostat方便查看CPU、网卡、tty设备、磁盘、CD-ROM 等等设备的活动情况, 负载信息。11.3. 命令参数-C 显示CPU使用情况...

2018-03-20 17:31:55 166

原创 The process NodeManager clean Application log and intermediate data like map output

Content ListContent ListNodeStatusUpdaterImpl.startStatusUpdater.new Runnable.runCMgrCompletedAppsEvent’s event type is ContainerManagerEventType.FINISH_APPSNodeManager.serviceInitContainer...

2018-03-14 11:56:54 241

转载 /proc/meminfo

This is one of the more commonly used files in the /proc/ directory, as it reports a large amount of valuable information about the systems RAM usage.The following sample /proc/meminfo virtual file ...

2018-03-07 17:51:41 235

转载 /proc/stat解析

一. 概述1.1 CPU时间cpu指标含义 user 用户态时间 nice 用户态时间(低优先级，nice>0) system 内核态时间 idle 空闲时间 iowait I/O等待时间 irq 硬中断 softirq 软中断 iowait时间是不可靠值，理由如下：CPU不会等待I/O执行完成，而iowait是等待I/O完成的时间...

2018-03-07 17:36:34 28860

原创 Hadoop hdfs does not update modificationTime when add block while the linux system does

ModificationTime on HDFSWhen file is created on hdfs, the system set modification time, then the modification time does not update until close it. In addition to close a file, the modification ti...

2018-03-07 16:13:39 285

原创 YARN 状态机的原理和使用

有了异步处理，为什么还需要状态机？可以保存对象当前的状态。当接收到一个事件后，可以根据不同的状态，可以方便的决定怎么处理。如接收到用户发送的KILL事件时：当Job的状态为正在初始化时，只要把启动的进程kill就可以。当Job的状态为正在运行，并且是提交到yarn的作业时，除了要把启动的进程kill，还需要调用yarn application -kill 把 yarn上运行的作...

2018-03-05 15:48:09 2473

原创 YARN Dispatcher的原理和应用

AsyncDispatcher ArchitectureAsyncDispatcher ExampleEnum AgentEventTypepublic enum AgentEventType { REGISTER, UNREGISTER, EXPIRED, HEARTBEAT, JOBLAUNCH, JOBKILL}...

2018-03-02 17:59:56 1182

原创 YARN Service的原理及应用

Service TypeAbstratServiceCompositeServiceService Exampleclass Agent { private final int id; public Agent(int id) { this.id = id; } // hashCode and equals method mus...

2018-03-02 17:10:21 2049 1

Hive 性能测试程序，https://github.com/hortonworks/hive-testbench 编译

2023-02-24

Tez 的设计者在 Hadoop 大会上的分享

1. Tez 的设计思想和解决的问题。

2022-03-15

Transactional Operations in Apache Hive:Present and Future

Hive 事务设计

2022-03-01

The Log-Structured Merge-Tree (LSM-Tree).pdf

2021-06-22

Distributed cycle detection in large-scale sparse graphs.pdf

分布式大规模环检测的计算方法。In this paper we present a distributed algorithm for detecting cycles in large-scale directed graphs, along with its correctness proof and analysis. The algorithm is then extended to find strong components in directed graphs. We indicate an application to detecting cycles in number theoretic functions such as the proper divisor function. Our prototype implementation of the cycle detection algorithm, when applied to the proper divisor function, detects all sociable groups of numbers (cycles in the proper divisor function) up to 107.

2020-08-11

空空如也

TA创建的收藏夹 TA关注的收藏夹

TA关注的人