spark dataframe helloworld

最新推荐文章于 2022-07-25 15:23:32 发布

dataastron

最新推荐文章于 2022-07-25 15:23:32 发布

阅读量172

点赞数

分类专栏： Spark 文章标签： spark

本文链接：https://blog.csdn.net/dataastron/article/details/80180371

版权

Spark 专栏收录该内容

14 篇文章 0 订阅

订阅专栏

package com.astron.spark2

import org.apache.spark
import org.apache.spark.sql.SparkSession

object BasicOperation {
  def main(args: Array[String]): Unit = {
    println("astron")
    val spark = SparkSession
      .builder()
      .master("local")
      .appName("star")
      .getOrCreate()

    // 导入spark的隐式转换
    import spark.implicits._
    val emp = spark.read.json("d://employee.json")
    val dep = spark.read.json("d://department.json")
    println("-"*10 + "完整数据")
    emp.show()
    //数值型过滤
    println("-"*10 + "数值型过滤emp.filter(\"age>40\").filter(\"salary>1000\").show()")
    emp.filter("age>40").filter("salary>1000").show()
    //字符型过滤
    println("-"*10 + "字符型过滤:等于emp.filter(emp(\"name\")===\"Jen\").show()")
    emp.filter(emp("name")==="Jen").show()
    println("-"*10 + "字符型过滤:不等于，已经过期用法emp.filter(emp(\"name\")!==\"Jen\").show()")
    emp.filter(emp("name")!=="Jen").show()
    println("-"*10 + "字符型过滤:不等于，目前用法emp.filter(emp(\"name\")=!=\"Jen\").show()")
    emp.filter(emp("name")=!="Jen").show()

    //$访问
    emp.filter($"name"=!="Jen"&&($"age")===25).show()
  }

}

astron
----------完整数据
+---+-----+------+------+------+
|age|depId|gender|  name|salary|
+---+-----+------+------+------+
| 25|    1|  male|   Leo| 20000|
| 30|    2|female| Marry| 25000|
| 35|    1|  male|  Jack| 15000|
| 42|    3|  male|   Tom| 18000|
| 21|    3|female|Kattie| 21000|
| 30|    2|female|   Jen| 28000|
| 19|    2|female|   Jen|  8000|
+---+-----+------+------+------+

----------数值型过滤emp.filter("age>40").filter("salary>1000").show()
+---+-----+------+----+------+
|age|depId|gender|name|salary|
+---+-----+------+----+------+
| 42|    3|  male| Tom| 18000|
+---+-----+------+----+------+

----------字符型过滤:等于emp.filter(emp("name")==="Jen").show()
+---+-----+------+----+------+
|age|depId|gender|name|salary|
+---+-----+------+----+------+
| 30|    2|female| Jen| 28000|
| 19|    2|female| Jen|  8000|
+---+-----+------+----+------+

----------字符型过滤:不等于，已经过期用法emp.filter(emp("name")!=="Jen").show()
+---+-----+------+------+------+
|age|depId|gender|  name|salary|
+---+-----+------+------+------+
| 25|    1|  male|   Leo| 20000|
| 30|    2|female| Marry| 25000|
| 35|    1|  male|  Jack| 15000|
| 42|    3|  male|   Tom| 18000|
| 21|    3|female|Kattie| 21000|
+---+-----+------+------+------+

----------字符型过滤:不等于，目前用法emp.filter(emp("name")=!="Jen").show()
+---+-----+------+------+------+
|age|depId|gender|  name|salary|
+---+-----+------+------+------+
| 25|    1|  male|   Leo| 20000|
| 30|    2|female| Marry| 25000|
| 35|    1|  male|  Jack| 15000|
| 42|    3|  male|   Tom| 18000|
| 21|    3|female|Kattie| 21000|
+---+-----+------+------+------+

+---+-----+------+----+------+
|age|depId|gender|name|salary|
+---+-----+------+----+------+
| 25|    1|  male| Leo| 20000|
+---+-----+------+----+------+