sprak单机安装，从hdfs读取文件进行wordcount并写入hdfs

1，在spark-env.sh中设置相关配置信息如下：

export SCALA_HOME=/home/lin/Desktop/SparkLin/scala-2.11.7
export SPARK_WORKER_MEMORY=2g
export SPARK_MASTER_IP=192.168.1.126
export MASTER=spark://192.168.1.126:7077
export JAVA_HOME=/usr/java/jdk1.7.0_79

但是启动spark的时候发现master启动成功，而worker守护进程启动不成功，查看worker启动9日志错误信息如下所示：

16/06/10 15:22:58 WARN Utils: Service 'sparkWorker' could not bind on port 0. Attempting port 1.
16/06/10 15:22:58 WARN Utils: Service 'sparkWorker' could not bind on port 0. Attempting port 1.
16/06/10 15:22:58 WARN Utils: Service 'sparkWorker' could not bind on port 0. Attempting port 1.
Exception in thread "main" java.net.BindException: Cannot assign requested address: Service 'sparkWorker' failed after 16 retries!
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1089)
    at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:430)
    at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:415)
    at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903)
    at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198)
    at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
    at java.lang.Thread.run(Thread.java:745)

分析发现是worker的默认端口被电脑其他的运用程序占用，于是修改worker的安装端口，如下：

export SPARK_WORKER_PORT=8011

重新启动spark，错误解决。

2，操作hdfs进行数据的读取，并进行单词统计，发生以下错误:

Exception in thread "main" java.net.ConnectException: Call From ubuntu.slave5/192.168.1.126 to ubuntu.slave5:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
    at org.apache.hadoop.ipc.Client.call(Client.java:1472)
    at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy19.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)

这一般是由于hdfs文件路径写错导致，在hadoop配置的时候，我将fs.defaultFS设置为hdfs://localhost:9000，而在写wordcount中我将其写成hdfs://192.168.1.12:9000，将其修改后，问题得到解决。