随笔记录

一、堡垒机上 Mysql数据导出 ：
mysql -xx.xx.com -Pxx -uxx -pxx -D xx -e “SELECT * FROM xx” >test.csv

二、spark hive 小文件(可能无效)和动态分区
spark.sqlContext.setConf(“mapred.compress.map.output”,“false”)
spark.sqlContext.setConf(“hive.merge.mapfiles”,“true”)
spark.sqlContext.setConf(“hive.merge.mapredfiles”,“true”)
spark.sqlContext.setConf(“hive.merge.size.per.task”,“256000000”)
spark.sqlContext.setConf(“hive.merge.smallfiles.avgsize”,“200000000”)
spark.sqlContext.setConf(“hive.exec.dynamic.partition”,“true”)
spark.sqlContext.setConf(“hive.exec.dynamic.partition.mode”,“nonstrict”)