spark安装技巧

1.spark 安装

下载spark地址为：

http://spark.apache.org/下载

2.运行spark运行示例和Shell

2.1运行java的例子

./bin/run-example SparkPi 10

20/04/01 22:00:35 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
20/04/01 22:00:35 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
20/04/01 22:00:35 INFO Executor: Running task 3.0 in stage 0.0 (TID 3)
20/04/01 22:00:35 INFO Executor: Running task 2.0 in stage 0.0 (TID 2)
20/04/01 22:00:35 INFO Executor: Fetching spark://192.168.31.36:56979/jars/spark-examples_2.11-2.4.5.jar with timestamp 1585749634196
20/04/01 22:00:35 INFO TransportClientFactory: Successfully created connection to /192.168.31.36:56979 after 48 ms (0 ms spent in bootstraps)
20/04/01 22:00:35 INFO Utils: Fetching spark://192.168.31.36:56979/jars/spark-examples_2.11-2.4.5.jar to /private/var/folders/j_/xj53k3kn3mxb_98p4681ypjh0000gn/T/spark-ef735d2c-eef3-4bdf-b699-5c996b48ab0d/userFiles-27277911-7acf-4fe0-8597-c1469bdf3110/fetchFileTemp2914441567963747275.tmp
20/04/01 22:00:35 INFO Executor: Adding file:/private/var/folders/j_/xj53k3kn3mxb_98p4681ypjh0000gn/T/spark-ef735d2c-eef3-4bdf-b699-5c996b48ab0d/userFiles-27277911-7acf-4fe0-8597-c1469bdf3110/spark-examples_2.11-2.4.5.jar to class loader
20/04/01 22:00:35 INFO Executor: Fetching spark://192.168.31.36:56979/jars/scopt_2.11-3.7.0.jar with timestamp 1585749634198
20/04/01 22:00:35 INFO Utils: Fetching spark://192.168.31.36:56979/jars/scopt_2.11-3.7.0.jar to 
20/04/01 22:00:35 INFO Executor: Adding file:/private/var/folders/j_/xj53k3kn3mxb_98p4681ypjh0000gn/T/spark-ef735d2c-eef3-4bdf-b699-5c996b48ab0d/userFiles-27277911-7acf-4fe0-8597-c1469bdf3110/scopt_2.11-3.7.0.jar to class loader
20/04/01 22:00:35 INFO Executor: Finished task 3.0 in stage 0.0 (TID 3). 867 bytes result sent to driver
20/04/01 22:00:35 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 867 bytes result sent to driver
20/04/01 22:00:35 INFO Executor: Finished task 2.0 in stage 0.0 (TID 2). 867 bytes result sent to driver
20/04/01 22:00:35 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 867 bytes result sent to driver
20/04/01 22:00:35 INFO TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4, localhost, executor driver, partition 4, PROCESS_LOCAL, 7866 bytes)
20/04/01 22:00:35 INFO TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5, localhost, executor driver, partition 5, PROCESS_LOCAL, 7866 bytes)
20/04/01 22:00:35 INFO TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6, localhost, executor driver, partition 6, PROCESS_LOCAL, 7866 bytes)
20/04/01 22:00:35 INFO Executor: Running task 5.0 in stage 0.0 (TID 5)
20/04/01 22:00:35 INFO Executor: Running task 6.0 in stage 0.0 (TID 6)
20/04/01 22:00:35 INFO Executor: Running task 4.0 in stage 0.0 (TID 4)
20/04/01 22:00:35 INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7, localhost, executor driver, partition 7, PROCESS_LOCAL, 7866 bytes)
20/04/01 22:00:35 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 447 ms on localhost (executor driver) (1/10)
20/04/01 22:00:35 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 451 ms on localhost (executor driver) (2/10)
20/04/01 22:00:35 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 477 ms on localhost (executor driver) (3/10)
20/04/01 22:00:35 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 454 ms on localhost (executor driver) (4/10)
20/04/01 22:00:35 INFO Executor: Running task 7.0 in stage 0.0 (TID 7)
20/04/01 22:00:35 INFO Executor: Finished task 5.0 in stage 0.0 (TID 5). 824 bytes result sent to driver
20/04/01 22:00:35 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, localhost, executor driver, partition 8, PROCESS_LOCAL, 7866 bytes)
20/04/01 22:00:35 INFO TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 30 ms on localhost (executor driver) (5/10)
20/04/01 22:00:35 INFO Executor: Finished task 6.0 in stage 0.0 (TID 6). 824 bytes result sent to driver
20/04/01 22:00:35 INFO Executor: Running task 8.0 in stage 0.0 (TID 8)
20/04/01 22:00:35 INFO TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9, localhost, executor driver, partition 9, PROCESS_LOCAL, 7866 bytes)
20/04/01 22:00:35 INFO TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 36 ms on localhost (executor driver) (6/10)
20/04/01 22:00:35 INFO Executor: Running task 9.0 in stage 0.0 (TID 9)
20/04/01 22:00:35 INFO Executor: Finished task 7.0 in stage 0.0 (TID 7). 824 bytes result sent to driver
20/04/01 22:00:35 INFO TaskSetManager: Finished task 7.0 in stage 0.0 (TID 7) in 52 ms on localhost (executor driver) (7/10)
20/04/01 22:00:35 INFO Executor: Finished task 9.0 in stage 0.0 (TID 9). 824 bytes result sent to driver
20/04/01 22:00:35 INFO TaskSetManager: Finished task 9.0 in stage 0.0 (TID 9) in 45 ms on localhost (executor driver) (8/10)
20/04/01 22:00:35 INFO Executor: Finished task 8.0 in stage 0.0 (TID 8). 824 bytes result sent to driver
20/04/01 22:00:35 INFO TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 54 ms on localhost (executor driver) (9/10)
20/04/01 22:00:35 INFO Executor: Finished task 4.0 in stage 0.0 (TID 4). 824 bytes result sent to driver
20/04/01 22:00:35 INFO TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 88 ms on localhost (executor driver) (10/10)
20/04/01 22:00:35 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 0.735 s
20/04/01 22:00:35 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
20/04/01 22:00:35 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 0.817976 s
Pi is roughly 3.1403551403551404
20/04/01 22:00:35 INFO SparkUI: Stopped Spark web UI at http://192.168.31.36:4040
 

2.2运行Python例子

./bin/spark-shell --master local[2]

Spark context Web UI available at http://192.168.31.36:4040

启动后出现spark的界面