spark：Idea编译程序打jar包部署到spark运行方法--19

感觉没个老师带着学spark真是举步维艰啊···一个问题能卡了2天才解决···写一下这俩天的收获吧···

//打包
1.Project Structure
2.Artifacts --- +jar --- From module with··· --- main class --- ok
3.class paths 要填/usr/local/scala-2.10.4/lib/scala-swing.jar 
                      /usr/local/scala-2.10.4/lib/scala-library.jar  
                      /usr/local/scala-2.10.4/lib/scala-actors.jar
                      /usr/local/spark-1.2.0-bin-hadoop2.4/lib/spark-assembly-1.2.0-hadoop2.4.0.jar
4.build
5.rebuild

///
java -jar lfspark--.jar lfspark--.jar hdfs://localhost:9000/datatnt/text1.txt hdfs://localhost:9000/outputtnt/
/
jar包里设置sc.txtfile(hdfs://localhost:9000/datatnt/text1.txt)
           saveAsTextFile("hdfs://localhost:9000/outputtnt/")
///
Hadoop hdfs文件系统文件已存在 解决办法:
bin/hadoop fs -rmr /output
///
 java -jar lfspark--.jar lfspark--.jar hdfs://localhost:9000/datatnt/text1.txt hdfs://localhost:9000/outputtnt/
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/03/30 23:27:32 WARN Utils: Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; 

using 192.168.30.129 instead (on interface ens33)
15/03/30 23:27:32 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
15/03/30 23:27:38 INFO SecurityManager: Changing view acls to: root
15/03/30 23:27:38 INFO SecurityManager: Changing modify acls to: root
15/03/30 23:27:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with 

view permissions: Set(root); users with modify permissions: Set(root)
15/03/30 23:27:40 INFO Slf4jLogger: Slf4jLogger started
15/03/30 23:27:41 INFO Remoting: Starting remoting
15/03/30 23:27:42 INFO Remoting: Remoting started; listening on addresses :

[akka.tcp://[email protected]:51013]
15/03/30 23:27:42 INFO Utils: Successfully started service 'sparkDriver' on port 51013.
15/03/30 23:27:42 INFO SparkEnv: Registering MapOutputTracker
15/03/30 23:27:42 INFO SparkEnv: Registering BlockManagerMaster
15/03/30 23:27:42 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20150330232742-e238
15/03/30 23:27:42 INFO MemoryStore: MemoryStore started with capacity 129.5 MB
15/03/30 23:27:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using 

builtin-java classes where applicable
15/03/30 23:27:45 INFO HttpFileServer: HTTP File server directory is /tmp/spark-e1e41ea5-ffa7-42a0-937f-

8edb39cc7cf0
15/03/30 23:27:45 IN