张某 提交的第三方jar /home/zhang/r_igraph.zip ,
config[["spark.yarn.dist.archives"]] <- "/home/zhang/miniconda3/envs/r_igraph.zip"
config[["spark.r.command"]] <- "./r_igraph.zip/bin/Rscript"
config$sparklyr.apply.env.R_HOME <- "./r_igraph.zip/lib/R"
config$sparklyr.apply.env.RHOME <- "./r_igraph.zip/"
config$sparklyr.apply.env.R_SHARE_DIR <- "./r_igraph.zip/lib/R/share"
config$sparklyr.apply.env.R_INCLUDE_DIR <- "./r_igraph.zip/lib/R/include"
代码使用zip中的文件,发现找不到
经过排查
1、查找yarn的executor 的container
登录d129的机器的找到container_1536303536795_778181_01_000024进程
ps -ef|grep container_1536303536795_778181_01_000024
3、找到提交的job缓存路径
r_igraph.zip 解压后会多一集r_igraph目录
config[["spark.yarn.dist.archives"]] <- "/home/zhang/miniconda3/envs/r_igraph.zip"
config[["spark.r.command"]] <- "./r_igraph.zip/r_igraph/bin/Rscript"
config$sparklyr.apply.env.R_HOME <- "./r_igraph.zip/r_igraph/lib/R"
config$sparklyr.apply.env.RHOME <- "./r_igraph.zip/r_igraph"
config$sparklyr.apply.env.R_SHARE_DIR <- "./r_igraph.zip/r_igraph/lib/R/share"
config$sparklyr.apply.env.R_INCLUDE_DIR <- "./r_igraph.zip/r_igraph/lib/R/include"