Hadoop & Spark & Hive & HBase

1. bin/hdfs namenode -format
2. sbin/start-dfs.sh

  http://localhost:50070/

1. bin/hdfs dfs -mkdir /user
2. bin/hdfs dfs -mkdir /user/<username>

these are for testing:

1. bin/hdfs dfs -put etc/hadoop input
2. bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.4.jar grep input output 'dfs[a-z.]+'
3. bin/hdfs dfs -cat output/*

testing results:

1. 6 dfs.audit.logger
2. 4 dfs.class
3. 3 dfs.server.namenode.
4. 2 dfs.period
5. 2 dfs.audit.log.maxfilesize
6. 2 dfs.audit.log.maxbackupindex
7. 1 dfsmetrics.log
8. 1 dfsadmin
9. 1 dfs.servers
10. 1 dfs.replication
11. 1 dfs.file

YARN: 

ResourceManager

1. ./sbin/start-yarn.sh

Http://localhost:8088/

HistoryServer

1. ./sbin/mr-jobhistory-daemon.sh start historyserver

http://localhost:19888/

Spark:

http://spark.apache.org/docs/1.6.2/

start:  

1. ./sbin/start-master.sh

http://localhost:8080/

start worker:

1. ./sbin/start-slaves.sh spark://<your-computer-name>:7077  

You will see:

• Alive Workers: 1

http://localhost:8080/

This is for testing:

1. ./bin/spark-shell --master spark://<your-computer-name>:7077

You will see the scala shell.

use : q to quit.

To see the history:

1. ./sbin/start-history-server.sh

http://localhost:18080/

Hive:

https://cwiki.apache.org/confluence/display/Hive/GettingStarted

https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2

http://www.360doc.com/content/16/0411/19/2795334_549791350.shtml

Bug:

in mysql 5.7 you should use :

1. jdbc:mysql://localhost:3306/hivedb?useSSL=false&createDatabaseIfNotExist=true

start hiveserver2:

1.  nohup hiveserver2 &

http://localhost:10002/

Bug：

1. User:  is not allowed to impersonate anonymous (state=,code=0)

https://community.hortonworks.com/questions/42483/user-hive-is-not-allowed-to-impersonate-anonymous.html

http://stackoverflow.com/questions/31228420/how-to-run-hive-on-spark-job-from-beeline-or-any-jdbc-client

See more:

https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html

Hwi 界面Bug：

1. HWI WAR file not found at

pack the war file yourself, then copy it to the right place, then add needed setting into hive-site.xml

http://blog.csdn.net/gao634209276/article/details/51426371

http://blog.csdn.net/duguduchong/article/details/8852425

1. Problem: failed to create task or type componentdef
2. Or:

3. Could not create task or type of type: componentdef

sudo apt-get install libjasperreports-java

sudo apt-get install ant

_________________________________________________________________________________not finished

自定义配置：

http://blog.csdn.net/reesun/article/details/8556078

数据库连接软件：

默认 用户名就是登录账号 密码为空

http://blog.sina.com.cn/s/blog_76923bd80102wi3g.html

语法

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL

more info:

http://stackoverflow.com/questions/35476468/what-is-the-difference-between-the-hive-metastore-in-derby-vs-the-one-in-hive-wa

http://www.2cto.com/database/201408/325554.html

HBase

http://hbase.apache.org/book.html#quickstart

1. ./bin/start-hbase.sh 

http://localhost:16010/

HBase & Hive

SparkSQL

Spark SQL架构如下图所示:

  http://lib.csdn.net/article/spark/33925

phoenix

1. queryserver.py start
   
2. jdbc:phoenix:thin:url=http://localhost:8765;serialization=PROTOBUF
   

Or:

1. phoenix-sqlline.py localhost:2181

来自为知笔记(Wiz)

转载于:https://www.cnblogs.com/jins-note/p/9513419.html