hive/spark-SQL常用积累

1. 空值处理

1.1 过滤：

Spark SQL : select * from raw_data where isNaN(var1) = false
Hive SQL :select * from raw_data where var1 is not null and cast(var1 as String) <> 'NaN'
参考：https://stackoverflow.com/questions/43526225/how-to-filter-out-rows-with-nan-values-in-hive/43576400#43576400