开发第一个入门级spark 应用程序
第一测试数据:
A0008062015_48c47a4cf0a4f79f pConnection_18 1440648000.043241000 1540648000.045089000 211.88.25.111 17848 0 0 211.88.25.173 8888 7 295 noabort 0.000000000 0.000000000 0 0 ---SF-|http://localhost
A0008062015_48c47a4cf0a4f79f pConnection_18 1440648000.043241000 1640648000.045089000 211.88.25.111 17848 0 0 212.88.25.173 8888 7 295 noabort 0.000000000 0.000000000 0 0 ---SF-|url
A0008062015_48c47a4cf0a4f79f pConnection_18 1440648000.043241000 1440648000.045089000 211.88.25.111 17848 0 0 213.88.25.173 8888 7 295 noabort 0.000000000 0.000000000 0 0 ---SF-|url
A0008062015_48c47a4cf0a4f79f pConnection_18 1440648000.043241000 1440648000.045089000 211.88.25.111 17848 0 0 214.88.25.173 8888 7 295 noabort 0.000000000 0.000000000 0 0 ---SF-|url
A0008062015_48c47a4cf0a4f79f pConnection_18 1440648000.043241000 1440648000.045089000 211.88.25.111 17848 0 0 215.88.25.173 8888 7 295 noabort 0.000000000 0.000000000 0 0 ---SF-|url
A0008062015_48c47a4cf0a4f79f pConnection_18 1440648000.043241000 1440648000.045089000 211.88.25.111 17848 0 0 216.88.25.173 8888 7 295 noabort 0.000000000 0.000000000 0 0 ---SF-|url
A0008062015_48c47a4cf0a4f79f pConnection_18 1440648000.043241000 1440648000.045089000 211.88.25.111 17848 0 0 217.88.25.173 8888 7 295 noabort 0.000000000 0.000000000 0 0 ---SF-|url
A0008062015_48c47a4cf0a4f79f pConnection_18 1440648000.043241000 1440648000.045089000 211.88.25.111 17848 0 0 218.88.25.173 8888 7 295 noabort 0.000000000 0.000000000 0 0 ---SF-|url
A0008062015_48c47a4cf0a4f79f pConnection_18 1440648000.043241000 1440648000.045089000 211.88.25.111 17848 0 0 211.88.25.173 8888 7 295 noabort 0.000000000 0.000000000 0 0 ---SF-|url
A0008062015_48c47a4cf0a4f79f pConnection_18 1440648000.043241000 1440648000.045089000 211.88.25.111 17848 0 0 211.88.25.173 8888 7 295 noabort 0.000000000
第二程序:
主函数入口
3 打成jar 文件运行。 File----project Structure-----Arcfacts 进行打包操作 详细如下图所示:
然后点击Apply 并ok。
4 编译程序
4 (spark 集群启动的环境下)
在终端输入: spark-submit --master spark://192.168.1.118:7077 --name WorldCount --class com.hq.WorldCount --executor-memory 2G --total-executor-cores 2 ~/spark_test/spark_yw.jar hdfs://192.168.1.118:9000/yw/b.txt hdfs://192.168.1.118:9000/yw/vvv
看到如下结果。就ok了。