CSV Parser

Most of our data files are in CSV format. Although the String.split('\t') approach can handle a lot cases, there are CSV files which has quotes. In that case if a delimiter character is in between of quotes, split method will fail. In other words, we do need a good CSV parser which can work well with Spark.

Since Scala so far works fine with me, the first try is to use some RegexParser based approach within Scala. There is a small project called scala-csv which is doing exactly this. However, with some simple performance test, this parser is just too SLOW. The problem is actually from some performance issues with RegexParser on the new JVM. Then I found this java library opencsv. Some benchmarks showed really promising performance results. Here is what the code for using opencsv with Spark

1 2 3 4 5 6 7 8

import au.com.bytecode.opencsv.CSVParser ...      def mLine(line : String) = {        val parser = new CSVParser( '\t' )        parser.parseLine(line)      }      val hist = myRDD.map(mLine( _ ).size).mapPartitions(histFeed( _ )).reduce( _::_ )      hist.toSeq.sortBy( _ . _ 1 ).foreach{ case (k,v) = >println(k+ " " +v)}

Here I used the histogram helper to calculate the histogram on the field counts of an RDD.

The performance of opencsv parser is just AMAZING! On my single machine case, it’s sometimes even faster than the split method. Most of the times, it is almost as fast as split.