yarn分布式缓存策略

 

张某 提交的第三方jar /home/zhang/r_igraph.zip ，

config[["spark.yarn.dist.archives"]] <- "/home/zhang/miniconda3/envs/r_igraph.zip"

config[["spark.r.command"]] <- "./r_igraph.zip/bin/Rscript"

config$sparklyr.apply.env.R_HOME <- "./r_igraph.zip/lib/R"

config$sparklyr.apply.env.RHOME <- "./r_igraph.zip/"

config$sparklyr.apply.env.R_SHARE_DIR <- "./r_igraph.zip/lib/R/share"

config$sparklyr.apply.env.R_INCLUDE_DIR <- "./r_igraph.zip/lib/R/include"

代码使用zip中的文件，发现找不到

经过排查

1、查找yarn的executor 的container

登录d129的机器的找到container_1536303536795_778181_01_000024进程

ps -ef|grep container_1536303536795_778181_01_000024

3、找到提交的job缓存路径

r_igraph.zip 解压后会多一集r_igraph目录

config[["spark.yarn.dist.archives"]] <- "/home/zhang/miniconda3/envs/r_igraph.zip"

config[["spark.r.command"]] <- "./r_igraph.zip/r_igraph/bin/Rscript"

config$sparklyr.apply.env.R_HOME <- "./r_igraph.zip/r_igraph/lib/R"

config$sparklyr.apply.env.RHOME <- "./r_igraph.zip/r_igraph"

config$sparklyr.apply.env.R_SHARE_DIR <- "./r_igraph.zip/r_igraph/lib/R/share"

config$sparklyr.apply.env.R_INCLUDE_DIR <- "./r_igraph.zip/r_igraph/lib/R/include"