jupyter sparkmagic on hdp

1、安装jupyter

2、安装sparkmagic

3、设置超时

"livy_server_heartbeat_timeout_seconds": 0,

4、设置集群模式

"session_configs": {
 	"driverMemory": "2G", "executorCores": 4, "proxyUser": "bernhard", "conf": { "spark.master": "yarn-cluster", "spark.jars.packages": "com.databricks:spark-csv_2.10:1.5.0" } }
5、配置hdp

livy.server.csrf_protection.enabled ==> false
6、启动notebook，配饰spark session

%load_ext sparkmagic.magics

%manage_spark

参考文档
https://github.com/jupyter-incubator/sparkmagic
https://github.com/bernhard-42/Sparkmagic-on-HDP

转载于:https://www.cnblogs.com/cenglinjinran/p/7878630.html