pyspark使用 jupyter ，matplotlib， ipython

export PYSPARK_DRIVER_PYTHON=jupyter export IPYTHON=1 export PYSPARK_DRIVER_PYTHON_OPTS="jupyter notebook --port=8888 --ip=raini"

--(正确用法)----  
raini@biyuzhe:~/spark1$ sudo IPYTHON=1 PYSPARK_DRIVER_PYTHON=jupyter IPYTHON_OPTS=`jupyter notebook` bin/pyspark
或者
raini@biyuzhe:~/spark1$ sudo IPYTHON=1 PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS=`jupyter notebook --ip=10.155.240.153 --port=8888 --no-browser --pylab inline` bin/pyspark

sudo IPYTHON=1 PYSPARK_DRIVER_PYTHON=jupyter bin/pyspark



（--no-browser网站）：
   translate.baiducontent.com/transpage?cb=translateCallback&ie=utf8&source=url&query=http%3A%2F%2Fstackoverflow.com%2Fquestions%2F34517293%2Frunning-jupyter-notebook-from-ubuntu-server-im-sshed-into&from=en&to=zh&token=&monLang=zh
----------------
raini@biyuzhe:~/spark1$ sudo PYSPARK_DRIVER_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS="jupyter notebook -pylab inline" bin/pyspark


--（安装jupyter以前的notebook）--raini@biyuzhe:~$ sudo pip install jupyter

--(安装matplotlib )--raini@biyuzhe:~/spark1$ pip install matplotlib


raini@biyuzhe:~/app/spark-1.6.0-bin-hadoop2.6$ sudo TPYTHON=1 IPYTHON_OPTS="jupyter notebook --pylab" ./bin/pyspark


raini@biyuzhe:~/spark1$ bin/spark-submit  --master spark://biyuzhe:7077 --class org.apache.spark.examples.SparkPi --name Spark-Pi ./lib/spark-examples-1.6.0-hadoop2.6.0.jar 

spark1/bin/spark-submit --jars /home/raini/IdeaProjects/lib/spark-streaming-kafka_2.10-1.6.0.jar /home/raini/IdeaProjects/lib/spark-streaming-kafka-assembly_2.10-1.6.0.jar /home/raini/IdeaProjects/lib/kafka_2.10-0.9.0.0.jar /home/raini/IdeaProjects/lib/kafka-clients-0.9.0.0.jar --class k_stream.src.k_staem.WebPagePopularityValueCalculator --master spark://biyuzhe:7077 --num-executors 2 --driver-memory 2g --executor-memory 1g --executor-cores 2 /home/raini/IdeaProjects/kafka_streaming/sparkexercise.jar biyuzhe:2181 2

报错 Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000bff80000, 716177408, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 716177408 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/raini/hs_err_pid17061.log


spark1/bin/spark-submit --class k_steam.WebPagePopularityValueCalculator --master spark://biyuzhe:7077 --num-executors 2 --driver-memory 4g --executor-memory 2g --executor-cores 2 /home/raini/IdeaProjects/k_steam/k_steam.jar biyuzhe:2181 2


	spark.eventLog.enabled  true
	spark.serializer  org.apache.spark.serializer.KryoSerializer



spark1/bin/spark-submit --class SparkStream.KafkaWordCount --master spark://biyuzhe:7077 --num-executors 2 --driver-memory 2g --executor-memory 512g --executor-cores 2 /home/raini/IdeaProjects/SparkStream/SparkStream.jar biyuzhe:2181 KafkaWordCount-group my-topic1 2




bin/spark-shell --driver-memory 2g



动态加载Spark属性：在一些情况下你可能想避免在SparkConf上硬编码. 举例来说， 如果你想在不同的master上或者不同的内存上运行同样的应用， Spark允许你简单创建一个空的conf:
   val sc = new SparkContext(new SparkConf())

你可以在运行时提供这些配置值:

    ./bin/spark-submit --name "My app" --master local[4] --conf spark.shuffle.spill=false
      --conf "spark.executor.extraJavaOptions=-XX:+PrintGCDetails -XX:+PrintGCTimeStamps" myApp.jar