bigdataf的博客

1.定义udf基于F.lit(data) 实现传参F.lit 函数实际上是新增一例,可以理解是自定义的函数把参数当列一样来操作from pyspark.sql import SparkSession,functions as Ffrom pyspark.sql.types import *from pyspark.sql import Windowdef filter(col,filters=["haha","hehe"]) col_map={} try: fo

1.初始化from __future__ import print_function, divisionfrom pyspark import SparkConf, SparkContextfrom pyspark.sql import SparkSession,functions as Ffrom pyspark.sql.types import StringType,MapType2.读取数据def parse_log_schema(txt): try: fileds=txt.st

1.传入多个参数 val result = spark.read.text("hdfs://hdfs-name/user/aa.txt","hdfs://hdfs-name/test/bb.txt")2.正则val result = spark.read.text("hdfs://hdfs-name/user/*")3.文件列表val path = "hdfs://hdfs-name/user/*.txt" val path2 = "hdfs://hdfs-name/test/*.txt"

1.hive mr 太慢了，基于次 适配 spark-sql的功能 ，于是乎，想通过spar启动thriftserver来解决问题1：17/12/29 16:28:59 WARN thrift.ThriftCLIService: Error opening session:java.lang.RuntimeException: Could not load shims in class org.a

spark hive hbase 结合业务需求，需要整合需要读取hive数据导入hbase中，一下是环境配置流程以及中间遇到的问题1.spark读hive需要copy hive-site和hdfs-site 等配置文件到项目资源包下object hivesql { case class Record(key: Int, value: String) def ma