spark开发笔记-scala 读写lzo文件两种写法
方法一:
val files = sc.newAPIHadoopFile("s3n://<YOUR_BUCKET>/<YOUR_PATH_TO_LZO_FILES/*.lzo", classOf[com.hadoop.mapreduce.LzoTextInputFormat],
classOf[org.apache.hadoop.io.LongWritable],classOf[org.apache.hadoop.io.Text]).map(_._2.toString)
方法二:
val files = sc.newAPIHadoopFile[LongWritable, Text, LzoTextInputFormat]("s3n://<YOUR_BUCKET>/<YOUR_PATH_TO_LZO_FILES/*.lzo").map(_._2.toString)