本地调试spark，访问kerberos鉴权的hdfs、hive

准备连接hive的配置

core-site.xml、hdfs-site.xml、yarn-site.xml、hive-site.xml复制到本地；
不知道为什么yarn-site.xml里的配置也必须存在，即使不使用yarn运行spark

申请kerberos tgt

在/etc/krb5.conf记录账户域名对应的授权服务地址，注意配置default_ccache_name（Ticket Granting Ticket存放路径）的格式，让java程序能够自动读取到tgt

[libdefaults]
    dns_lookup_realm = false
    ticket_lifetime = 24h
    renew_lifetime = 7d
    forwardable = true
    rdns = false
    pkinit_anchors = FILE:/etc/pki/tls/certs/ca-bundle.crt
    spake_preauth_groups = edwards25519
    default_realm = COMPANY.UAT.COM
    # default_ccache_name = KEYRING:persistent:%{uid}
    default_ccache_name = FILE:/tmp/krb5cc_%{uid}
[realms]
COMPANY.UAT.COM = {
  kdc = 10.0.0.1
  admin_server = 10.0.0.1
}

申请当前操作系统用户的进程都可使用的tgt(Ticket Granting Ticket)

kinit -kt /etc/kerberos/hive.keytab hive@COMPANY.UAT.COM

在scala项目启动本地spark

连接配置文件全放到项目的src/test/resources后，可执行单元测试

import org.scalatest.funsuite.AnyFunSuite
import org.apache.spark.sql.SparkSession

class SparkTest extends AnyFunSuite with BeforeAndAfterAll {

  val catalogImpl: String = "hive"

  lazy val spark: SparkSession = SparkSession
    .builder().appName("scala test")
    .master("local[1]")
    .enableHiveSupport()
    // 如果不使用kinit命令预先申请tgt，也可让spark自行申请
    // .config("spark.kerberos.keytab", "/etc/kerberos/hive.keytab")
    // .config("spark.kerberos.principal", "hive@COMPANY.UAT.COM")    
    .getOrCreate()

  test("select hive") {
    val df = spark.sql("select * from default.temp_test1 limit 10")
    df.printSchema()
    df.show()
  }

  test("sleep") {
    // 挂起测试线程，可在localhost:4040访问spark控制台，确认是否加载了hdfs、spark相关配置
    Thread.sleep(1000000)
  }
}

本地Jupyter Notebook启动pyspark

安装pyspark

pip3 install pyspark==3.2.4

连接配置文件全放到本地/apps/conf/SPARK目录内

from pyspark.sql import SparkSession
import os
# spark-submit能够读取环境变量SPARK_CONF_DIR指定的配置目录
os.environ["SPARK_CONF_DIR"] = "/apps/conf/SPARK"
# pyspark调用spark-submit，启动driver
# 查看driver的java进程: ps -ef | grep pyspark-shell
spark = (
    SparkSession.builder.appName("pyspark test")
    .master("local[1]")
    .enableHiveSupport()
    # 下载hive相关依赖包
    .config("spark.sql.hive.metastore.version", "3.1.3")
    .config("spark.sql.hive.metastore.jars", "maven")
    .config(
        "spark.jars.repositories", "https://maven.aliyun.com/repository/public"
    )
    .getOrCreate()
)

在后续notebook代码单元格里随意编辑、执行spark代码

df1 = spark.sql("SHOW TABLES IN default")
df1.show()