Exception in thread “main“ java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession

本地 IDEA 跑spark代码，出现这个报错，

import org.apache.spark.{SparkConf, SparkContext}

object testjson {
    def main(args: Array[String]): Unit = {
        val conf = new SparkConf()
            .setAppName("first spark app(scala)")
            .setMaster("local[1]");

        new SparkContext(conf)
            .parallelize(List(1, 2, 3, 4, 5, 6))
            .map(x => x * x)
            .filter(_ > 10)
            .collect()
            .foreach(println);
    }
}

 解决方案，pom.xml需要用 compile，把scope注释掉，改成,provided表示不把spark环境打包，用运行集群的环境。

<scope>compile</scope>

/data/home/spark-sh1-tq-2.4/bin/spark-submit 
--conf spark.default.parallelism=160
--class 111
--master yarn-cluster 
--name 
--deploy-mode cluster 
--driver-memory 6G 
--num-executors 50 
--executor-memory 6G 
--executor-cores 5 
--conf spark.yarn.queue=root.
--conf ml.alg.ext.job_name=S_t_ed
--conf ml.alg.ext.job_id=1111 
--conf ml.alg.ext.cycle=22222 
--conf ml.alg.ext.cluster_id=sh1-tq 
--conf ml.alg.ext.app_group_id=1805 
--conf spark.ml.alg.ext.job_id=2222 
--conf spark.ml.alg.ext.app_group_id=1805 
--conf spark.executorEnv.excustom_job_info=csdcdscs
--conf spark.yarn.appMasterEnv.custom_job_info=cdscscdsds 
--conf spark.yarn.priority=1 
--conf spark.hadoop.job.platform=csdcsdcsd 
--conf spark.hadoop.usp.param=dcsdvdscsd 
--conf spark.hadoop.tdw.ugi.groupname=dscscwec
--conf spark.hadoop.tdw.username=your name
--conf spark.yarn.tags=8161cdscwc
--conf spark.yarn.stagingDir=hdfs:/cdstaging 
--conf spark.tdw.privilege.api.url=http:/lService 
--conf spark.security.credentials.hadoopfs.enabled=false 
--conf spark.dynamicAllocation.maxExecutors=50 --properties-file /data/log/jobtemps/sparkconfsS_
/data/script/spark/yybsoftware/s_t_ed_yyb.jar 
--ds  20220829 20220829