hadoop记录

单机hadoop2.8.3
1、hadoop运行文件：hadoop-env.sh， 增加
export JAVA_HOME=/opt/software/jdk1.8.0_40

2、修改core-site.xml
<configuration>
	<property>
			<name>fs.defaultFS</name>
			<value>hdfs://192.168.6.99:9000</value>
	</property>
	<property>
		<name>hadoop.proxyuser.hadoop.groups</name>
		<value>hadoop</value>
	</property>
	<property>
		<name>hadoop.proxyuser.hadoop.hosts</name>
		<value>192.168.6.99,127.0.0.1,localhost</value>
	</property>
</configuration>

3、修改hdfs-site.xml
<configuration>
    <property>
        <name>dfs.nameservices</name>
        <value>hadoop-cluster</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:///data/hadoop/hdfs/nn</value>
    </property>
    <property>
        <name>dfs.namenode.checkpoint.dir</name>
        <value>file:///data/hadoop/hdfs/snn</value>
    </property>
    <property>
        <name>dfs.namenode.checkpoint.edits.dir</name>
        <value>file:///data/hadoop/hdfs/snn</value>
    </property>
    <property>
        <name>dfs.namenode.data.dir</name>
        <value>file:///data/hadoop/hdfs/dn</value>
    </property>
    <property><name>dfs.datanode.max.transfer.threads</name> <value>8192</value></property>
</configuration>

4、免密登录

5、hadoop停止与气动
第一次启动需要格式化：hdfs namenode -format
启动：start-dfs.sh
--jps会看到NameNode和DataNode节点
--http://localhost:50070/dfshealth.html#tab-overview
停止：stop-dfs.sh

5、文件读取（关闭安全模式：hadoop dfsadmin -safemode leave 
hadoop fs 类linux命令
hadoop fs -ls /




namenode--中心服务器
datanoce--存储数据


6 namenode安全模式离开：hdfs dfsadmin -safemode leave

HIVE安装

1、hive-site.xml配置文件
<configuration>
    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://127.0.0.1:3306/hive?createDatabaseIfNotExist=true&amp;useSSL=false</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>root</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>123456</value>
    </property>
    <property>
      <name>datanucleus.autoCreateSchema</name>
      <value>true</value>
    </property>

    <property>
         <name>datanucleus.fixedDatastore</name>
         <value>true</value>
    </property>

    <property>
        <name>datanucleus.autoCreateTables</name>
        <value>true</value>
    </property>
    <property>
        <name>hive.metastore.schema.verification</name>
        <value>false</value>
    </property>
	
	<property>
		<name>hive.metastore.warehouse.dir</name>
		<value>hdfs://192.168.12.58:9000/user/hive/warehouse</value>
		<description>location of default database for thewarehouse</description> 
	</property>
</configuration>

2、启动hive
hive
命令：！shell命令

show tables语句报错SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
执行：set hive.vectorized.execution.enabled=true; 

3.启动hivede metastroe后台进程。执行 nohup hive --service metastore &  启动hive的service metastore后台进程。
4、启动hive远程服务端口：nohup hive --service hiveserver2 & 提供10000端口



create table student(
id int ,
name string,
password string
)
row format delimited fields terminated by '\t'
;

tab建隔开
hive> load data local inpath '/home/hadoop/data/student.txt' into table student;


beeline连接数据库
!connect jdbc:hive2://192.168.12.58:10000/fits

set hive.cli.print.header=true; // 打印列名

spark提交任务
一、开启spark集群：
1、./start-master
2、./start-slave.sh spark://localhost:7077 (-m 1G -c 4)
3、任务提交：
spark-submit --master spark://192.168.12.58:7077 --class com.terry.HadoopTest /fits/project/client1/build/libs/client-1.0-SNAPSHOT.jar