java spark wordcount_在IDEA中编写Spark的WordCount程序

1 [root@master hadoop]# bin/spark-submit --class com.bie.WordCount --master spark://master:7077 --executor-memory 700M --total-executor-cores 2 /home/hadoop/data_hadoop/sparkWordCount-1.0-SNAPSHOT.jar hdfs://master:9000/wordcount.txt hdfs://master:9000/output

2 bash: bin/spark-submit: No such file or directory3 [root@master hadoop]# cd spark-1.6.1-bin-hadoop2.6/

4 [root@master spark-1.6.1-bin-hadoop2.6]# bin/spark-submit --class com.bie.WordCount --master spark://master:7077 --executor-memory 700M --total-executor-cores 2 /home/hadoop/data_hadoop/sparkWordCount-1.0-SNAPSHOT.jar hdfs://master:9000/wordcount.txt hdfs://master:9000/output

5 18/02/23 01:45:46 INFO SparkContext: Running Spark version 1.6.1

6 18/02/23 01:45:47 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes whereapplicable7 18/02/23 01:45:47INFO SecurityManager: Changing view acls to: root8 18/02/23 01:45:47INFO SecurityManager: Changing modify acls to: root9 18/02/23 01:45:47INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)10 18/02/23 01:45:48 INFO Utils: Successfully started service 'sparkDriver' on port 60097.11 18/02/23 01:45:49INFO Slf4jLogger: Slf4jLogger started12 18/02/23 01:45:49INFO Remoting: Starting remoting13 18/02/23 01:45:49 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.3.129:55353]

14 18/02/23 01:45:49 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 55353.15 18/02/23 01:45:49INFO SparkEnv: Registering MapOutputTracker16 18/02/23 01:45:49INFO SparkEnv: Registering BlockManagerMaster17 18/02/23 01:45:49 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-17e41a67-b880-4c06-95eb-a0f64928f66818 18/02/23 01:45:49 INFO MemoryStore: MemoryStore started with capacity 517.4MB19 18/02/23 01:45:49INFO SparkEnv: Registering OutputCommitCoordinator20 18/02/23 01:45:50 INFO Utils: Successfully started service 'SparkUI' on port 4040.21 18/02/23 01:45:50 INFO SparkUI: Started SparkUI at http://192.168.3.129:4040

22 18/02/23 01:45:50 INFO HttpFileServer: HTTP File server directory is /tmp/spark-99c897ab-ea17-4797-8695-3a5df89ed490/httpd-f346e1dd-642d-437d-8947-6190f2e8306523 18/02/23 01:45:50INFO HttpServer: Starting HTTP Server24 18/02/23 01:45:50 INFO Utils: Successfully started service 'HTTP file server' on port 35900.25 18/02/23 01:45:51 INFO SparkContext: Added JAR file:/home/hadoop/data_hadoop/sparkWordCount-1.0-SNAPSHOT.jar at http://192.168.3.129:35900/jars/sparkWordCount-1.0-SNAPSHOT.jar with timestamp 1519379151547

26 18/02/23 01:45:52 INFO AppClient$ClientEndpoint: Connecting to master spark://master:7077...

27 18/02/23 01:4