Spark读取文件并清洗流程

import org.apache.spark.{SparkConf, SparkContext}

object MyHome {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setMaster("local[2]").setAppName("cust")
    val sc = new SparkContext(conf)
    val rdd = sc.textFile("file:///f:/customers.csv").cache()
//    rdd.foreach(println)

    rdd.map(_.split(",").apply(2).replaceAll("\"","")).map((_,1)).reduceByKey(_+_)
      .sortBy(x=> -x._2).take(5).foreach(println)

    sc.stop()
  }
}

文档格式如下:第0列序号,第1列名,第二列姓,题目要求找出排名前五的姓

1 Richard Hernandez XXXXXXXXX XXXXXXXXX 6303 Heather Plaza Brownsville TX 78521
2 Mary Barrett XXXXXXXXX XXXXXXXXX 9526 Noble Embers Ridge Littleton CO 80126
3 Ann Smith XXXXXXXXX XXXXXXXXX 3422 Blue Pioneer Bend Caguas PR 725
4 Mary Jones XXXXXXXXX XXXXXXXXX 8324 Little Common San Marcos CA 92069
5 Robert Hudson XXXXXXXXX XXXXXXXXX 10 Crystal River Mall Caguas PR 725
6 Mary Smith XXXXXXXXX XXXXXXXXX 3151 Sleepy Quail Promenade Passaic NJ 7055
7 Melissa Wilcox XXXXXXXXX XXXXXXXXX 9453 High Concession Caguas PR 725
8 Megan Smith XXXXXXXXX XXXXXXXXX 3047 Foggy Forest Plaza Lawrence MA 1841
9 Mary Perez XXXXXXXXX XXXXXXXXX 3616 Quaking Street Caguas PR 725
10 Melissa Smith XXXXXXXXX XXXXXXXXX 8598 Harvest Beacon Plaza Stafford VA 22554
11 Mary Huffman XXXXXXXXX XXXXXXXXX 3169 Stony Woods Caguas PR 725
12 Christopher Smith XXXXXXXXX XXXXXXXXX 5594 Jagged Embers By-pass San Antonio TX 78227
13 Mary Baldwin XXXXXXXXX XXXXXXXXX 7922 Iron Oak Gardens Caguas PR 725
14 Katherine Smith XXXXXXXXX XXXXXXXXX 5666 Hazy Pony Square Pico Rivera CA 90660
15 Jane Luna XXXXXXXXX XXXXXXXXX 673 Burning Glen Fontana CA 92336
16 Tiffany Smith XXXXXXXXX XXXXXXXXX 6651 Iron Port Caguas PR 725
17 Mary Robinson XXXXXXXXX XXXXXXXXX 1325 Noble Pike Taylor MI 48180
18 Robert Smith XXXXXXXXX XXXXXXXXX 2734 Hazy Butterfly Circle Martinez CA 94553
19 Stephanie Mitchell XXXXXXXXX XXXXXXXXX 3543 Red Treasure Bay Caguas PR 725
20 Mary Ellis XXXXXXXXX XXXXXXXXX 4703 Old Route West New York NJ 7093
21 William Zimmerman XXXXXXXXX XXXXXXXXX 3323 Old Willow Mall Caguas PR 725
22 Joseph Smith XXXXXXXXX XXXXXXXXX 7740 Broad Fox Villas North Bergen NJ 7047
23 Benjamin Duarte XXXXXXXXX XXXXXXXXX 8811 High Horse Impasse San Juan PR 921
24 Mary Smith XXXXXXXXX XXXXXXXXX 9417 Emerald Towers Caguas PR 725
25 Paul Richardson XXXXXXXXX XXXXXXXXX 318 Merry Creek Mountain Peoria AZ 85345
26 Johnny Hood XXXXXXXXX XXXXXXXXX 9576 Middle Hills Round Glenview IL 60025
27 Mary Vincent XXXXXXXXX XXXXXXXXX 1768 Sleepy Zephyr Place Caguas PR 725
28 Timothy Smith XXXXXXXXX XXXXXXXXX 9366 Colonial Horse Close Longview WA 98632
29 Mary Humphrey XXXXXXXXX XXXXXXXXX 2469 Blue Brook Crossing Fort Worth TX 76133
30 Barbara Smith XXXXXXXXX XXXXXXXXX 2455 Merry Hollow Caguas PR 725
31 Mary Byrd XXXXXXXXX XXXXXXXXX 8259 Foggy Edge Albuquerque NM 87111
32 Alice Smith XXXXXXXXX XXXXXXXXX 2082 Hidden Green Caguas PR 725
33 Adam Petersen XXXXXXXXX XXXXXXXXX 9768 Noble Horse Heights Norfolk VA 23503
34 Mary Smith XXXXXXXXX XXXXXXXXX 3330 Easy Berry Range Caguas PR 725
35 Margaret Wright XXXXXXXXX XXXXXXXXX 9456 Sleepy Jetty Oceanside CA 92056
36 Michelle Carey XXXXXXXXX XXXXXXXXX 6336 Fallen Village Caguas PR 725
37 Virginia Conrad XXXXXXXXX XXXXXXXXX 6115 Indian Beacon Close Norfolk VA 23503
38 Mary Smith XXXXXXXXX XXXXXXXXX 2805 Crystal Moor Opelousas LA 70570
39 Juan Mckinney XXXXXXXXX XXXXXXXXX 7274 Blue Wagon Villas Caguas PR 725
40 Mary Smith XXXXXXXXX XXXXXXXXX 7358 Rocky Villas Long Beach CA 90805
41 Victoria Mason XXXXXXXXX XXXXXXXXX 7869 Crystal View Villas Brooklyn NY 11230
42 Ethan Smith XXXXXXXXX XXXXXXXXX 7533 Thunder Wagon Beach Duluth GA 30096
43 Mary Herring XXXXXXXXX XXXXXXXXX 4575 Thunder Dale Caguas PR 725
44 Howard Smith XXXXXXXXX XXXXXXXXX 1356 Easy Plaza Napa CA 94558
45 Mary Nunez XXXXXXXXX XXXXXXXXX 1229 Bright Robin Corners Denver CO 80231
46 Jennifer Smith XXXXXXXXX XXXXXXXXX 5463 Rocky Autoroute Freeport NY 11520
47 Lori Fuller XXXXXXXXX XXXXXXXXX 357 Noble Lane Caguas PR 725
48 Mary Cowan XXXXXXXXX XXXXXXXXX 1040 Hazy Timber Lookout Strongsville OH 44136
49 Martha Smith XXXXXXXXX XXXXXXXXX 7449 Merry Chase Caguas PR 725
50 Mary Kim XXXXXXXXX XXXXXXXXX 938 Rustic Pine Ridge San Bernardino CA 92410
51 Jessica Smith XXXXXXXXX XXXXXXXXX 8344 Dewy Fawn Farms Caguas PR 725
52 Mary Lawson XXXXXXXXX XXXXXXXXX 65 Silent Inlet Chicago IL 60622
53 Katherine Smith XXXXXXXXX XXXXXXXXX 975 Honey Loop Roswell GA 30075
54 Mary Olson XXXXXXXXX XXXXXXXXX 7333 Cinder Way Caguas PR 725
55 Janet Farley XXXXXXXXX XXXXXXXXX 6512 Middle Parade Phoenix AZ 85029
56 Mary Kaiser XXXXXXXXX XXXXXXXXX 7554 Stony Goose Crossing Caguas PR 725
57 Mary Smith XXXXXXXXX XXXXXXXXX 57 High Villas Baltimore MD 21229
58 Michael Edwards XXXXXXXXX XXXXXXXXX 4770 Blue Pines Caguas PR 725
59 Douglas James XXXXXXXXX XXXXXXXXX 2306 Green Lane Sunnyvale CA 94086
60 Mary Gutierrez XXXXXXXXX XXXXXXXXX 8632 Bright Route Webster NY 14580
61 Margaret Summers XXXXXXXXX XXXXXXXXX 8963 Hidden Panda Meadow Caguas PR 725
62 Mary Oneal XXXXXXXXX XXXXXXXXX 2659 Jagged Rabbit View Brooklyn NY 11223
63 Aaron Smith XXXXXXXXX XXXXXXXXX 2265 Foggy Lagoon Bend Caguas PR 725
64 Mary Reynolds XXXXXXXXX XXXXXXXXX 2407 Gentle Woods Chicago IL 60647
65 Jennifer Smith XXXXXXXXX XXXXXXXXX 7126 Shady Bluff Promenade Princeton NJ 8540
66 Mary Booth XXXXXXXXX XXXXXXXXX 8824 Noble View Valley Caguas PR 725
67 Samantha Smith XXXXXXXXX XXXXXXXXX 7869 Wishing Anchor Court Chicago IL 60619

©️2020 CSDN 皮肤主题: 游动-白 设计师:上身试试 返回首页