houzhizhen的专栏

【代码】Hadoop 远程 debug。

mvn package -Pdist,native -DskipTests -Dtar -Dmaven.javadoc.skip=true Caused by: org.apache.maven.plugin.MojoExecutionException: CMake failed with error code 1 at org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.runCMake (CompileMojo.java:172)

Start ZookeeperEnable Hadoop Registry (Enabled by Default)# Enable api-servicevim yarn-site.xml<property> <description>Enable services rest api on Resourcemanager.</description> <name> yarn.webapp.api-service.

停止hadoop服务停止YARN以hadoop账号在主控服务器执行以下命令：./upgrade.sh cmd hosts "yarn-daemon.sh stop nodemanager"yarn-daemon.sh stop resourcemanager./upgrade.sh cmd hosts "hadoop-daemon.sh stop datanode"hadoop-daemon.sh stop namenode/usr/local/hadoop/etc/hadoop/core-

package org.apache.hadoop.hdfs.namenode;import java.io.BufferedInputStream;import java.io.File;import java.io.FileInputStream;import java.io.IOException;import java.lang.management.GarbageCollector

class SerializableWritable[T <: Writable](@transient var t: T) extends Serializable { def value: T = t override def toString: String = t.toString private def writeObject(out: ObjectOutputStream): Un

nn1为active，nn2为standby 首先下载2.6.3的hadoop源码进行编译，形成hadoop-2.6.3.tar.gz 安装新版本的hadoop，从log-server上分发新版的hadoop并解压         cd /letv/setupHadoop         ./upgrade.sh distribute cluster_nodes hadoo

UserGroupInformation is used in the following way.final UserGroupInformation loginUgi = UserGroupInformation.getLoginUser();The getLoginUser method is simple. public synchronized static UserGroupIn

/** * A short circuited method to get a block writer that can write data directly to disk. * The Block will be appended to the File specified by filename. Callers should handle error * cases.

找到以下文件/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java找到以下方法： public synchronized List> chooseUnderReplicatedBlocks(找到以下代

一,配置history server1.配置history server,在etc/hadoop/mapred-site.xml中配置以下内容. mapreduce.jobhistory.address localhost:10020 mapreduce.jobhistory.webapp.address localhost:198

Congiruration staticWhen class Congiruration is loaded, static code area is called. It adds core-default.xml and core-site.xml as default resource. static{ //print deprecation warning if hadoop-si

FileSink initialize writer stream in init method, and the writer is not changed since after, so the metrics file can not rotate.```/** * A metrics sink that writes to a file */@InterfaceAud

Git And HadoopA lot of people use Git with Hadoop because they have their own patches to make to Hadoop, and Git helps them manage it. GitHub provide some good lessons on git at http://learn.github.co...

Dev Environment SetupHere are some things you will need to build and test Hadoop. Be prepared to invest some time to set up a working Hadoop dev environment. Try getting the project to build and test ...

[houzhizhen@localhost hadoop]$ sh proto_ls.sh -rw-r--r--. 1 houzhizhen root 2741 1月   4 16:09 ./hadoop-2.7.5-src/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/acl.proto-rw-r--r--. 1 houzhizhen ro

上周周会领导让研究kerberos，要在我们的大集群中使用，研究任务指派给了我。这周的话也是用测试集群大概的做了一遍。目前为止的研究还比较粗糙，网上众多资料都是CDH的集群，而我们的集群是不是用的CDH，所以在集成kerberos的过程中有一些不同之处。测试环境是由5台机器搭建的集群，hadoop版本是2.7.2。5台机器host分别是 rm1、rm2、test-nn1、test-nn2、

TestMetrics import org.apache.hadoop.metrics2.annotation.Metric;import org.apache.hadoop.metrics2.annotation.Metrics;import org.apache.hadoop.metrics2.lib.MetricsRegistry;import org.apache.hadoop.m

DataNodeMetricsTake DataNodeMetrics as an example, the type of DefaultMetricsSystem.instance() is MetricsSystemImpl. public static DataNodeMetrics create(Configuration conf, String dnName) { Strin

/** * <p>A builder of {@link LoadingCache} and {@link Cache} instances having any combination of the * following features: * * <ul> * <li>automatic loading of entries into the cache * <li>least-r

/** * This util class provides a method to register an MBean using * our standard naming convention as described in the doc * for {link {@link #register(String, String, Object)} */@InterfaceAudie

使用脚本实现Hadoop 2.6 RackAwareness的步骤 在etc/hadoop/core-site.xml中添加以下内容: topology.script.file.name /usr/local/hadoop/bin/rackawareness.sh /usr/l/ocal/hadoop/bin/rackawareness.sh的内容如下:

Node 代表一个子结点或者一个中间结点。/** The interface defines a node in a network topology. * A node may be a leave representing a data node or an inner * node representing a datacenter or rack. * Each data has

IntrusiveCollection是一个链表，链表中的元素自己负责存储前一个元素和下一个元素的指针。如果列表中有很多元素，或者元素在很多列表中，这样可以节省很多内存。头定义如下：/** * Implements an intrusive doubly-linked list. * * An intrusive linked list is one in which the e

使用Hadoop ACL 控制访问权限一、HDFS访问控制hdfs-site.xml设置启动acl dfs.permissions.enabled true   dfs.namenode.acls.enabled true  core-site.xml设置用户组默认权限.fs.permissions.umask-m

以hadoop用户执行以下操作:1.修改etc/hadoop/hdfs-site.xml,在dfs.namenode.shared.edits.dir 配置项中增加相应的新的journal的地址和端口.2.把etc/hadoop/hdfs-site.xml分发到集群中各服务器.3.把现有journal中的数据目录拷贝到新journal服务器.4.在新journal服务器中执行had

本文讲解了hadoop 2.7的nfs安装方法。

今天系统维护，hdfs haadmin -failover nn1 nn2 失败。用jps查看java进程，发现没有DFSZKFailoverController进程。打开zkfc的日志，发现以下内容：2015-03-06 14:24:38,262 FATAL org.apache.hadoop.ha.ZKFailoverController: Unable to start fa

一、查看hadoop/conf/mapred-site.xml找到以下代码，根据配置所示，要把新服务器的地址放到文件adminedHosts中，不要在  /usr/local/hadoop/conf/blackHosts中。    mapred.black.hosts    /usr/local/hadoop/conf/blackHosts        m

1.查看20台服务器环境。发现没有hadoop用户，hosts已经更改。2.下载hadoop2.5.2.查看zookeeper版本，发现是zookeeper 3.4.6，和线上不一致。[root@hadoopNN1 ~]# cd /data/hadoop/data1/usr/local/setupSpark/vim newslaves #把要部署的服务器列表放到此文件scp h

一、准备工作1.Unix系统2.JDK1.7_253.Maven3.0.54.Findbugs2.0.2（若运行Findbugs）5.ProtocolBuffer2.5.06.CMake2.6或更高版本（若编译native库）(一)安装Maven 3.0.51.下载地址http://maven.apache.org/download.cgi

使用10.180.92.198作为备份机修改/etc/hosts.allow添加nn1.把/root/.ssh/authorized_keys中添加nn1的root的id_rsa.pub[root@hadoopNN1 ~]# cd /home/hadoop/setupecho "10.180.92.198" > newslaves./upgrade.sh distribut

1.10.200.91.164停止服务  hbase-daemon.sh stop master  hadoop-daemon.sh stop namenode  hadoop-daemon.sh stop zkfc2.重启之后，设置10.200.91.164参数  vim conf/hadoop-env.sh  hbase-daemon.sh start ma

前言前阵子，公司的hadoop从hadoop1.02升级到hadoop2.4.1，记录下升级的步骤和遇到的问题，和大家分享，希望别人可以少走一些弯路技术选型  当前使用版本：     apache hadoop 1.0.2      hive 0.10  升级目标版本     Apache hadoop 2.4.1     Hive 0.13 升级风险点

hftp默认是打开的,允许以浏览器的方式访问和下载文件,以此方式下,可以读取所有文件,留下了安全隐患.测试如下/user/hive/warehouse/cdntest.db/selfreadonly/hosts的上级目录selfreadonly的所有者是zhouyang,权限是700,但以xiangtao用户在浏览器中输入以下地址,就能下载.http://localhost:500

存放PendingBlockInfo/** * An object that contains information about a block that * is being replicated. It records the timestamp when the * system started replicating the most recent copy

LightWeightGSet的作用用一个数组来存储元素，并且用链表来解决冲突。不能rehash，所以内部数组永远不用改变大小。此类不支持空元素。此类也不是线程安全的。有两个类型参数，第一个用于查找元素，第二个类型参数必须是第一个类型参数的子类，并且必须实现LinkedElement接口。/** * A low memory footprint {@link GSet} implementa

以hadoop fs -chown data_sum:data_sum /test/input为例，讲解决EditLog的执行过程。　　当客户端执行以上命令时，通过RPC调用，服务器端执行NameNodeRpcServer.setOwner方法,代码如下： @Override // ClientProtocol public void setOwner(String src, Str

JvmPauseMonitor，此类建立一个简单的线程。在此线程中，在循环中运行sleep一段时间方法，如果sleep花费的时间比传递给sleep方法的时间长，就意味着JVM或者宿主机已经出现了停顿处理现象，可能会导致其它问题，如果这种停顿被监测出来，线程会打印一个消息。/** * Class which sets up a simple thread which runs in a loo

INodeMap利用LightWeightGSet来存储INode对象，为什么可以存储INode ID和INode的映射呢？因为INode的equals和hashCode方法都已经重载，并且只根据id判断，那么知道id之后，其它字段为空的Inode对象，就可以得到映射的对象。equals和hashCode 方法如下： @Override public final boolean eq