Q:NameError: name 'self' is not defined
最近开发Python包,遇到一个“NameError: name ‘self’ is not defined”问题。
在执行
class Tasdfa:
def __init__(self,prompt='asdfa',newline=False):
self.newline=newline
self.prompt=prompt
def __getattr__(self,prompt=self.prompt,newline=self.newline):
if self.newline:
print('')
print(self.prompt)
12345678910
的时候,总在“def __getattr__(self,prompt=self.prompt,newline=self.newline):”处报错:
NameError: name 'self' is not defined
A:误原因是不能在类成员函数的参数里带self.******,任何self.的变量都不行。
改成
class Tasdfa:
def __init__(self,prompt='asdfa',newline=False):
self.newline=newline
self.prompt=prompt
def __getattr__(self):
if self.newline:
print('')
print(self.prompt)
12345678910
后,正确运行。
Q:类成员函数间的变量引用
A:将变量设置成了类成员变量,如self.var
Q:类内函数调用
A:加前缀self,ru self.def
Q:在使用pyspark时,调用自定义的模块(.zip、egg),可能会遇到以下问题:
File "/usr/install/anaconda2/lib/python2.7/site-packages/pyspark/serializers.py", line 454, in loads
return pickle.loads(obj)
ImportError: No module named *
A:这是由于在pyspark中调用第三方包时,在executor节点上执行,executor节点上未加载自定义的模块所致。
利用sparkSession.sparkContext.addPyFile即可解决,该方法可以将自定义的模块分发到各个executor节点上
如spark.sparkContext.addPyFile("hdfs:///tmp/rd/lp/sparkxgb.zip")
Q:org.apache.spark.sql.AnalysisException: Table or view not found spark操作时报错
A:
1. 通过在项目的类路径中添加hive-site.xml配置文件
2. 以编程的方式在代码中设置Hive MateStore参数:
2.1 spark1.x设置:
val conf = new SparkConf();
val sc = new SparkContext(conf);
val hiveContext = new HiveContext(sc);
hiveContext.setConf("hive.metastore.uris", "thrift://METASTORE:9083");
2.2 spark2.x设置:
package com.aldb.bigdata.basic
import com.aldb.bigdata.utils.{CommonUtils, DateUtils}
import com.alibaba.fastjson.JSON
import org.apache.spark.sql.types.{StringType, StructField, StructType}
import org.apache.spark.sql.{Row, SparkSession}
import org.slf4j.LoggerFactory
/**
* 地理位置埋点数据解析写入Hive
* Created by fc.w on 2018/07/10
*/
object DeviceLocHdfsToHiveLauncher {
val logger = LoggerFactory.getLogger(DeviceLocHdfsToHiveLauncher.getClass)
def main(args: Array[String]): Unit = {
val ss = SparkSession.builder()
.appName("DeviceLoc_Hdfs_To_Hive_Job")
.config("hive.metastore.uris", "thrift://METASTORE:9083")
.config(CommonUtils.SPARK_SQL_DIR, CommonUtils.HIVE_WAREHOUSE)
.enableHiveSupport()
.getOrCreate()
import spark.implicits._
import spark.sql
// create an arbitrary frame
val frame = Seq(("one", 1), ("two", 2), ("three", 3)).toDF("word", "count")
// see the frame created
frame.show()
/**
* +-----+-----+
* | word|count|
* +-----+-----+
* | one| 1|
* | two| 2|
* |three| 3|
* +-----+-----+
*/
// write the frame
frame.write.mode("overwrite").saveAsTable("t4")
}
参考:
https://www.jb51.net/article/166981.htm
sparkContext.addPyFile:https://www.jianshu.com/p/6e13fa556859
spark链接hive数据库:https://www.jianshu.com/p/10c1afbe609a