前段时间在研究 pySpark 数据处理,深刻地感受到spark的极大魅力。自己是一个初学者,这篇博客也只是简单的完成了几个简单的工作任务,哈哈@@不说了上代码:
from pyspark import SparkConf,SparkContext
import csv
from _operator import add
import re
#!/usr/bin/python
# -*- coding: UTF-8 -*-
conf = SparkConf().setAppName("Simple").setMaster("local")
sc = SparkContext(conf = conf)
file1 = open("D:\\lbossdata.CSV","r")
f1 = csv.reader(file1) //打开csv文件将文件的第一行的第一列读入到row1
row1 = [row[0] for row in f1]
file1.close()
with open("D:\\lbossdata.CSV","r") as file2:
f2 = csv.reader(file2)
row2 = [row[1] for row in f2]
with open("D:\\lbossdata.CSV","r") as file3:
f3 = csv.reader(file3)
row3 = [row[2] for row in f3]
通第一个方法一样,将其余几行也分别读入到row2\row3,row4,ro