Spark StandAlone 集群下运行 application

cluster模式

运行spark submit应用程序

$SPARK_HOME//bin/spark-submit --master spark://vsr219:7077 --deploy-mode cluster  --class org.apache.spark.sql.SparkTPCDSQueries  ./tpcds-generate-perf_2.11-1.0.jar   10000   60000  parquet200

driver 日志

$SPARK_HOME/work/driver-20200724133824-0002

# ll
total 116
-rw-r--r-- 1 root root 82618 Jul 24 13:39 stderr
-rw-r--r-- 1 root root  8843 Jul 24 13:39 stdout
-rwxr-xr-x 1 root root 20334 Jul 24 13:38 tpcds-generate-perf_2.11-1.0.jar

print 信息在 stdout 里面

executor 日志

$SPARK_HOME/work/app-20200724133826-0006/0
“0” 为Executor 进程 序列号

# ll
total 11760
-rwxr-xr-x 1 root root 11498266 Jul 24 13:38 oap-cache-0.8.1-with-spark-2.4.4.jar
-rwxr-xr-x 1 root root    21927 Jul 24 13:38 oap-common-0.8.1-with-spark-2.4.4.jar
-rw-r--r-- 1 root root   493213 Jul 24 13:55 stderr
-rw-r--r-- 1 root root        0 Jul 24 13:38 stdout
-rwxr-xr-x 1 root root    20334 Jul 24 13:38 tpcds-generate-perf_2.11-1.0.jar

cat stderr

3731 20/07/24 13:39:02 WARN filecache.CacheGuardian: Fibers pending on removal use too much memory, current: 1173517080, max: 1073741824
3732 20/07/24 13:39:02 WARN filecache.CacheGuardian: Fibers pending on removal use too much memory, current: 1185866570, max: 1073741824
3733 20/07/24 13:39:02 WARN filecache.CacheGuardian: Fibers pending on removal use too much memory, current: 1197549631, max: 1073741824
3734 20/07/24 13:39:02 WARN filecache.CacheGuardian: Fibers pending on removal use too much memory, current: 1092219194, max: 1073741824