一个简单的数据处理任务
任务说明
样例如下：
18 Jogging 102271561469000 -13.53 16.89 -6.4
18 Jogging 102271641608000 -5.75 16.89 -0.46
18 Jogging 102271681617000 -2.18 16.32 11.07
18 Jogging 3.36
18 Downstairs 103260201636000 -4.44 7.06 1.95
18 Downstairs 103260241614000 -3.87 7.55 3.3
18 Downstairs 103260321693000 -4.06 8.08 4.79
18 Downstairs 103260365577000 -6.32 8.66 4.94
18 Downstairs 103260403083000 -5.37 11.22 3.06
18 Downstairs 103260443305000 -5.79 9.92 2.53
6 Walking 0 0 0 3.214402
Step 1
将数据集中所有信息异常的行删除。
比如上面的样例中第4行数据只有3个元素，而其他行都有6个元素，所以第4行是信息异常的行，将其删除。再如第12行数据的第3个元素明显也是有问题的，所以它也是信息异常的行，将其删除。
数据集中可能还会存在一些其他异常。
将全部信息处理之后，每行的元素以逗号为分隔符，写入文件test1。
文件test1共100471行，样例如下：
6,Walking,23445542281000,-0.72,9.62,0.14982383
6,Walking,23445592299000,-4.02,11.03,3.445948
6,Walking,23470662276000,0.95,14.71,3.636633
...
Step 2
统计文件test1的数据中所有动作的数目并打印到屏幕，然后将动作数目对100取整后写入test2文件，多余的信息行抛弃。比如统计出Jogging的数量为3021次，则在屏幕上打印Movement: Jogging Amount: 3021，然后将前3000行信息写入test2文件。
文件test2共100200行。
Step 3
读取文件test2的数据，取每行的后3列元素，以空格为分隔符写入文件test3。
文件test3共100200行，样例如下：
-0.72 9.62 0.14982383
-4.02 11.03 3.445948
0.95 14.71 3.636633
...
Step 4
读取文件test3的数据，每行数据为一组，每组组内的元素以空格为分隔符，组与组之间的数据以逗号为分隔符，每20组元素为一行，写入文件finally。
文件finally共5010行，样例如下：
-0.72 9.62 0.14982383,-4.02 11.03 3.445948,0.95 14.71 3.636633,-3.57 5.75 -5.407278,-5.28 8.85 -9.615966,-1.14 15.02 -3.8681788,7.86 11.22 -1.879608,6.28 4.9 -2.3018389,0.95 7.06 -3.445948,-1.61 9.7 0.23154591,6.44 12.18 -0.7627395,5.83 12.07 -0.53119355,7.21 12.41 0.3405087,6.17 12.53 -6.701211,-1.08 17.54 -6.701211,-1.69 16.78 3.214402,-2.3 8.12 -3.486809,-2.91 0 -4.7535014,-2.91 0 -4.7535014,-4.44 1.84 -2.8330324
一、将文件类型转化为csv类型
由于OriginalData文本类型并非是csv，因此第一步要将其用python转化为csv文件类型
转变前
代码
fp=open('C:\\Users\\Anan\\Desktop\\OriginalData','r')
fp_new=open('C:\\Users\\Anan\\Desktop\\OriginalData.csv','w')
for row in fp:
row=row.replace(' ',',')
fp_new.write(row)
fp.close()
fp_new.close()
转变后
将此文件保存到pycharm中进行后续操作
二、删除异常数据写入text1
1.思路
将每行长度不为6，并且第三列数据为0的异常数据删除
2.代码
import csv
with open('OriginalData.csv','r',newline='') as csv_in_file:
with open('OriginData_text1.csv','w',newline='') as csv_out_file:
filereader = csv.reader(csv_in_file)
filewriter = csv.writer(csv_out_file)
#删除掉行长度不为6的并且第三个数据为0的数据
for row in filereader:
if len(row) == 6 and float(row[2]) != 0 :
filewriter.writerow(row)
3.text1中数据
共计100471行
三、统计text1中动作数据，并对100取整写入text2
1.思路
先统计出每种动作出现的总次数，并打印出来次数，再对其取整，之后再打开一次csv文件，将符合行数的数据存入到text2中
2.代码
import csv
with open('OriginData_text1.csv','r',newline='') as csv_in_file:
filereader = csv.reader(csv_in_file)
Walking_count = 0
Jogging_count = 0
Upstairs_count = 0
Downstairs_count = 0
Sitting_count = 0
Standing_count = 0
for row in filereader:
if row[1] == 'Walking':
Walking_count += 1
if row[1] == 'Jogging':
Jogging_count += 1
if row[1] == 'Upstairs':
Upstairs_count += 1
if row[1] == 'Downstairs':
Downstairs_count += 1
if row[1] == 'Sitting':
Sitting_count += 1
if row[1] == 'Standing':
Standing_count += 1
print('Movement: Walking Amount: %d' % Walking_count)
print('Movement: Jogging Amount: %d' % Jogging_count)
print('Movement: Upstairs Amount: %d' % Upstairs_count)
print('Movement: Downstairs Amount: %d' % Downstairs_count)
print('Movement: Sitting Amount: %d' % Sitting_count)
print('Movement: Standing Amount: %d' % Standing_count)
Walking_count = Walking_count // 100 * 100
Jogging_count = Jogging_count // 100 * 100
Upstairs_count = Upstairs_count // 100 * 100
Downstairs_count = Downstairs_count // 100 * 100
Sitting_count = Sitting_count // 100 * 100
Standing_count = Standing_count // 100 * 100
csv_in_file.seek(0,0)
with open('OriginalData_text2.csv','w',newline='') as csv_out_file:
filewriter = csv.writer(csv_out_file)
for row_list in filereader:
if row_list[1] == 'Walking' and Walking_count != 0:
filewriter.writerow(row_list)
Walking_count -= 1
if row_list[1] == 'Jogging' and Jogging_count != 0:
filewriter.writerow(row_list)
Jogging_count -= 1
if row_list[1] == 'Upstairs' and Upstairs_count != 0:
filewriter.writerow(row_list)
Upstairs_count -= 1
if row_list[1] == 'Downstairs' and Downstairs_count != 0:
filewriter.writerow(row_list)
Downstairs_count -= 1
if row_list[1] == 'Sitting' and Sitting_count != 0:
filewriter.writerow(row_list)
Sitting_count -= 1
if row_list[1] == 'Standing' and Standing_count != 0:
filewriter.writerow(row_list)
Standing_count -= 1
3.打印结果
4.text2中数据
共计100200行
四、取text2中后三行数据，写入text3
1思路
运用选取特定的列的方法进行操作，之后按文件打开将间隔符由逗号变为空格
2.代码
import csv
value = [3,4,5]
with open('OriginalData_text2.csv','r',newline='') as csv_in_file:
with open('OriginalData_text3.csv','w',newline='') as csv_out_file:
filewriter = csv.writer(csv_out_file)
filereader = csv.reader(csv_in_file)
for row in filereader:
row_output = []
for index in value:
row_output.append(row[index])
filewriter.writerow(row_output)
fp = open('OriginalData_text3.csv','r')
fp_new = open('OriginalData_text3','w')
for row in fp:
row = row.replace(',',' ')
fp_new.write(row)
fp.close()
fp_new.close()
3.text3中数据
五、按要求读取text3数据，写入text4
1.思路
以文件的方式读入，用计数器进行计算，如果可以不能被20整除，则将换行符改为逗号，如果能整除则不变，最后存入到finally文件中
2.代码1
fp = open('OriginalData_text3','r')
fp_new = open('finally','w')
count = 0
for row in fp:
count += 1
if count % 20 != 0:
row = row.replace('\n', ',')
fp_new.write(row)
fp.close()
fp_new.close()
3.代码2
import csv
with open('OriginalData_text3','r',newline='') as csv_in_file:
filereader = csv.reader(csv_in_file)
count = 0
file = open('finally', 'w')
for row in filereader:
count += 1
row = str(row).replace("[","").replace("]",",")
row = row.replace("'","")
if count % 20 == 0:
row = row.replace(",","\n")
file.write(row)
4…finally文件中数据
总计5010行
此简单的数据处理任务就完成了
扫一扫
9514
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
