学习笔记 集群安装

购买集群
终端 代码以下载jdk1.8 压缩文件。完成后解压。
1.File->New->Project->Maven Project->Next 勾选 create->com.huweiMRWordCount
2.Window->Preferences->Java->Compiler->level 1.8->Apply and close
Window->Preferences->Java->|nstalledJREs->Add->standard VM->Next->配置->Finish->勾选新增 jdk->Apply3.
4…初始页面 JRE System Library【J2SE-1.5】->Build Path->configure->Add Library->JRE System Library->Next ->workspace->finish
5.Remove【J2SE-1.5】->Apply and close配置 pom 文件 ctrl+s
6.MRWordCount 的/src/main/java 新建 package,在 com.huawei.mapreduce 包下新建 MRWordCount的Java class,编写业务代码,ctrl+s
19行:string[]splited=line.split(“”);//请在此处编写代码，将 1ir
FLinkne110orld[root@node-master1Em
22行:Text k2=new Text(word);//请在此处编写代码
23行:LongWritable v2=new LongWritable(1);//请在此处编写代码
33行:long count= 0L;//请在此处编写代码，为count 完成初始化
37行:LongWritablev3=new LongWritable(count);//请在此处编写代码49 行:job.setMapperClass(MyMapper.class);//请在此处编写代码，填入括号内55 行:iob.setReducerClass(MyReducer.class);//请在此处编写代码，填入括号内7.MRWordCount->RUN AS->Maven install
16行:JavaRDDtextFile=jsc.textFile(args[0]);//请考生完成该友步骤代码编写，注意路径需以 args[819行:JavaRDDwords=textFile.flatMap(line->Arrays.asList(line.split("“)).iterator());25行:JavaPairRDD<String,Long>result= wordsAndOne.reduceByKey((v1,v2)->v1+v2 );
28行:result.saveAsTextFile(args[1])
Scp ~/eclipse-workspace/MRWordcount/target/MRWordcount-1.0-SNAPSHoT.jar root@xxx.xxx.xxx.xxx:/root8.输入密码运行并登录ssh root@xxx.xxx.xxx.xxx 输入密码
1s
可选:cd~/eclipse-workspace/MRWordCount/target/
9.vim words.txt i进入编辑模式 tab分隔 esc退出 :wg 回车保存退出
hdfs dfs -mkdir /user/FinalExam
hdfs dfs -put words.txt /user/FinalExam
10. yarn jar MRWordCount-1.0-SNApsHoT,jar, com.huawei .mapreduce.MRNordcount /user/FinalExam/words.txt /user/mr output11.hdfs dfs ls /user/mr output
12,hdfs dfs-cat /user/mr output/part-r-00000复制到面part-r-00080 文档
---------ニ-------------
---- ------- ---- - ----------,
----------Scp ~/eclipse-workspace/SparkWordcount/target/Sparkwordcount-1.0-SNAPsHoT.jar root@xxx.xx: /root2.输入密码 运行并登录sshroot@xxx.xxx.xxx.xxx 输入密码
可选:cd ~/eclipse-workspace/SparkWordCount/target/ls
spark-submit–class com,huawei,spark,sparkWordcount-master yarn-client SparkNordCount-1.0-SNAPSHoT.jar p/user/FinalExam/words.txt /user/spark output
m13.hdfs dfs -ls /user/spark output14.hdfs dfs -cat /user/spark_output/part-00800复制到面part-00000 文档

spark的链接处理

wget https://sandbox-expriment-files.obs.cn-north-1.myhuaweicloud.com/hccdp/HCCDP/jdk-8u341-linux-x64.tar.gz

tar -zxvf jdk-8u341-linux-x64.tar.gz

wget https://sandbox-expriment-files.obs.cn-north-1.myhuaweicloud.com/hccdp/HCCDP/apache-maven-3.6.0.tar.gz

tar -zxvf apache-maven-3.6.0.tar.gz

wget https://sandbox-expriment-files.obs.cn-north-1.myhuaweicloud.com/hccdp/HCCDP/scala-2.11.8.tgz

tar -zxf scala-2.11.8.tgz

vim ~/.bashrc

export JAVA_HOME=/home/user/jdk1.8.0_341 export JRE_HOME=$JAV{A}_{H}OME/jreexportCLASSPATH=.:${JAVA_HOME}/lib:$JR{E}_{H}OME/libexportSCAL{A}_{H}OME=/home/user/scala-\mathrm{2.11.8}exportPATH=${JAVA_HOME}/bin:$SCAL{A}_{H}OME/bin:$PATH

source ~/.bashrc

java -version scala -version

安装idea

wget https://sandbox-expriment-files.obs.cn-north-1.myhuaweicloud.com/hccdp/ideaIC.tar.gz
1.6安装IDEA打开xfce命令行，利用下列命令下载IDEA：
拷贝代码wget https://sandbox-expriment-files.obs.cn-north-1.myhuaweicloud.com/hccdp/ideaIC.tar.gz
下载完成后，利用如下命令进行解压：
拷贝代码tar -zxvf ideaIC.tar.gz解压需要一定时间，请耐心等待约1分钟。解压完成后，利用下列命令安装idea：
拷贝代码/home/user/idea-IC/bin/idea.sh运行后出现下列界面，勾选I confirm，然后点击Continue：在新窗口点击Don’t Send，即可完成安装。

1.7修改Maven配置等待安装完毕，出现IDEA界面，在界面左侧找到Customize标签，点击该标签。在Customize窗口下找到All settings，点击弹出配置窗口。在新窗口左侧找到“Build, Execution, Deployment”，展开该标签，选择“Build Tools”-“Maven”标签进入如下界面。直接编辑“Maven home path”，修改为如下路径：
拷贝代码/home/user/apache-maven-3.6.0修改“User settings file”为相关配置路径（记得点选右侧的Override）：
拷贝代码/home/user/apache-maven-3.6.0/conf/setting.xml修改完成后，结果如下：确认无误后点击OK即可。1.8安装Scala插件在IDEA初始界面找到Plugins标签，单击打开。选择Marketplace，在搜索框中输入scala，完成scala插件安装。安装scala插件后，点击绿色的Restart IDE重启IntelliJ IDEA。点击New Project，如果在Language标签下显示Scala，表示已成功安装