The --master
option specifies the master URL for a distributed cluster, or local
to run locally with one thread, or local[N]
to run locally with N threads.
查看hdfs
$ ./bin/hadoop fs -ls hdfs:/user/xunw/
spark 读取hdfs
spark-submit --master spark://192.168.1.24:7077 examples/src/main/python/wordcount.py hdfs://zj:9000/user/xunw/input2/hello.txt
注意192.168.1.24 不能写成zj
WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@192.168.1.24:7077: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@192.168.1.24:7077]
集群没有启动 start-all.sh
800M.txt 16min:
Final centers: [array([-4.94759899, -5.10103885]), array([ 4.86596504, 5.03759974])]
Internally, a DStream is represented as a sequence ofRDDs.
作用于RDD上的Operation分为transformantion和action。 经Transformation处理之后,数据集中的内容会发生更改,由数据集A转换成为数据集B;而经Action处理之后,数据集中的内容会被归约为一个具体的数值。
只有当RDD上有action时,该RDD及其父RDD上的所有operation才会被提交到cluster中真正的被执行。