scala spark 数据对比_使用Scala的Spark 2.0时间戳差异以毫秒为单位

I am using Spark 2.0 and looking for a way to achieve the following in Scala:

Need the time-stamp difference in milliseconds between two Data-frame column values.

Value_1 = 06/13/2017 16:44:20.044

Value_2 = 06/13/2017 16:44:21.067

Data-types for both is timestamp.

Note:Applying the function unix_timestamp(Column s) on both values and subtracting works but not upto the milliseconds value which is the requirement.

Final query would look like this:

Select **timestamp_diff**(Value_2,Value_1) from table1

this should return the following output:

1023 milliseconds

where timestamp_diff is the function that would calculate the difference in milliseconds.

解决方案

One way would be to use Unix epoch time, the number of milliseconds since 1 January 1970. Below is an example using an UDF, it takes two timestamps and returns the difference between them in milliseconds.

val timestamp_diff = udf((startTime: Timestamp, endTime: Timestamp) => {

(startTime.getTime() - endTime.getTime())

})

val df = // dataframe with two timestamp columns (col1 and col2)

.withColumn("diff", timestamp_diff(col("col2"), col("col1")))

Alternatively, you can register the function to use with SQL commands:

val timestamp_diff = (startTime: Timestamp, endTime: Timestamp) => {

(startTime.getTime() - endTime.getTime())

}

spark.sqlContext.udf.register("timestamp_diff", timestamp_diff)

df.createOrReplaceTempView("table1")

val df2 = spark.sqlContext.sql("SELECT *, timestamp_diff(col2, col1) as diff from table1")