pyspark 执行 lcs

最新推荐文章于 2024-11-02 22:17:47 发布

Logan_addoil

最新推荐文章于 2024-11-02 22:17:47 发布

阅读量330

点赞数 10

分类专栏：大数据学习之旅 python 文章标签： python spark

本文链接：https://blog.csdn.net/Logan_addoil/article/details/136755971

版权

大数据学习之旅同时被 2 个专栏收录

23 篇文章 0 订阅

订阅专栏

python

6 篇文章 0 订阅

订阅专栏

a.建表 hive中
b.导入数据
c.验证查询

from pyspark.sql.functions import udf
from pyspark.sql.types import Integer
from pyspark.sql import SparkSerssion

# 提前执行 conda install pyspark
import sys
reload(sys)
sys.setdefaultencoding('utf-8')

spark = SparkSerssion.builder
    .appName("LCS")
    .enableHiveSupport()
    .getOrCreate()

df = spark.sql("select * from badou.lcs_data")
df.show()

def lcs(a,b):
    n = len(a)
    m = len(b)
    l = [[0] * (m + 1) for x in range(n + 1)]
    for i in range(1,n+1):
        for j in range(1,m+1):
            if a[i-1]== b[j-1]:
                l[i][j] = l[i-1][j-1]+1
            else:
                l[i][j] = max(l[i-1][j], l[i][j-1])
    return l[-1][-1]

lcs_udf = udf(lcs, IntegerType())
df_score = df.withColumn("score", lcs_udf(df.a, df.b))
df_score.show()

PYSPARK_PYTHON=./ANACONDA/dev/bin/python
cd $SPARK_HOME
./bin/spark-submit \
        --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./ANACONDA/dev/bin/python \
        --master yarn-cluster \
        --files $HIVE_HOME/conf/hive-site.xml \
        --archives /usr/local/src/pyspark/dev.zip#ANACONDA \
        /usr/local/src/pyspark/lcs.py