MapReduce综合应用案例 — 电信数据清洗

第1关：数据清洗

首先离开头歌，在电脑桌面创建文件1.sh，在文件里面输入

#!/bin/bash
echo '刘能宗,杜泽文,18620192711,15733218050,2017-09-28 19:49:34,2017-09-28 19:51:05,91,新疆维吾尔自治区,香港特别行政区
郭美彤,杜泽文,18641241020,15733218050,2017-11-04 01:01:16,2017-11-04 01:04:24,188,浙江省,甘肃省
梁鹏,任宗阳,15778423030,15614201525,2017-05-20 14:27:31,2017-05-20 14:35:23,472,山东省,湖北省
段光星,张倩,13341109505,15151889601,2017-04-20 04:16:02,2017-04-20 04:23:20,438,浙江省,海南省
段光星,邓二,13341109505,13666666666,2016-08-02 04:10:26,2016-08-02 04:16:36,370,江西省,黑龙江省
张涛,菜中路,15032293356,13799999999,2017-05-28 02:06:21,2017-05-28 02:09:20,179,重庆市,青海省
杜泽文,段光星,15733218050,13341109505,2016-01-12 12:32:56,2016-01-12 12:40:01,425,甘肃省,云南省
王世昌,王世昌,13269361119,13269361119,2017-02-21 01:31:30,2017-02-21 01:37:03,333,广西壮族自治区,湖南省
菜中路,李进全,13799999999,15338595369,2017-11-29 04:13:34,2017-11-29 04:18:31,297,西藏自治区,黑龙江省
杜泽文,梁鹏,15733218050,15778423030,2018-11-17 12:27:13,2018-11-17 12:27:58,45,广西壮族自治区,云南省
段光星,张阳,13341109505,17731088562,2017-01-14 03:34:04,2017-01-14 03:42:22,498,海南省,江西省
朱尚宽,梁鹏,18332562075,15778423030,2018-03-30 16:11:15,2018-03-30 16:14:33,198,山西省,天津市
唐会华,杨力谋,13560190665,18301589432,2017-01-29 00:09:56,2017-01-29 00:14:19,263,甘肃省,澳门特别行政区
杜泽文,温海英,15733218050,13520404983,2018-10-08 09:55:31,2018-10-08 09:56:45,74,河北省,内蒙古自治区
梁鹏,刘柳,15778423030,13566666666,2017-01-10 00:38:41,2017-01-10 00:46:50,489,香港特别行政区,浙江省
刘柳,张阳,13566666666,17731088562,2018-11-10 01:21:53,2018-11-10 01:26:54,301,吉林省,江西省
梁鹏,杜泽文,15778423030,15733218050,2016-05-25 17:50:21,2016-05-25 17:53:23,182,青海省,安徽省
张倩,段光星,15151889601,13341109505,2016-07-02 06:30:52,2016-07-02 06:38:58,486,宁夏回族自治区,广东省
刘能宗,邓二,18620192711,13666666666,2017-11-18 09:37:56,2017-11-18 09:44:13,377,广西壮族自治区,陕西省
段光星,郭美彤,13341109505,18641241020,2017-10-27 03:29:06,2017-10-27 03:33:21,255,台湾省,上海市
张阳,段光星,17731088562,13341109505,2016-08-19 01:47:50,2016-08-19 01:55:06,436,湖南省,青海省
温海英,唐会华,13520404983,13560190665,2016-10-16 13:56:34,2016-10-16 14:04:43,489,重庆市,广东省
李进全,段光星,15338595369,13341109505,2018-04-17 20:13:51,2018-04-17 20:14:19,28,湖北省,海南省
张倩,段光星,15151889601,13341109505,2017-03-16 09:56:39,2017-03-16 09:59:54,195,浙江省,重庆市
唐会华,李进全,13560190665,15338595369,2017-11-17 03:51:21,2017-11-17 03:58:49,448,河南省,贵州省
杜泽文,刘柳,15733218050,13566666666,2017-08-23 14:29:00,2017-08-23 14:32:06,186,湖北省,上海市
张阳,唐会华,17731088562,13560190665,2016-08-07 12:00:55,2016-08-07 12:08:28,453,西藏自治区,浙江省
李进全,任宗阳,15338595369,15614201525,2017-06-06 16:51:19,2017-06-06 16:59:24,485,贵州省,重庆市
张阳,梁鹏,17731088562,15778423030,2017-05-12 15:22:47,2017-05-12 15:26:24,217,重庆市,湖北省
梁鹏,郭美彤,15778423030,18641241020,2018-02-01 00:30:07,2018-02-01 00:35:58,351,广西壮族自治区,云南省
刘柳,张阳,13566666666,17731088562,2016-05-31 12:29:25,2016-05-31 12:31:34,129,江西省,甘肃省
梁鹏,菜中路,15778423030,13799999999,2018-05-05 18:00:18,2018-05-05 18:04:53,275,重庆市,澳门特别行政区
段光星,温海英,13341109505,13520404983,2018-03-24 03:13:58,2018-03-24 03:17:01,183,重庆市,河北省
刘柳,唐会华,13566666666,13560190665,2017-12-22 04:49:20,2017-12-22 04:55:38,378,安徽省,辽宁省
张涛,刘能宗,15032293356,18620192711,2017-01-29 11:13:08,2017-01-29 11:15:37,149,西藏自治区,云南省
菜中路,段光星,13799999999,13341109505,2018-07-10 04:19:23,2018-07-10 04:20:38,75,黑龙江省,江苏省
李进全,张倩,15338595369,15151889601,2017-12-01 10:51:54,2017-12-01 10:59:38,464,西藏自治区,香港特别行政区
朱尚宽,唐会华,18332562075,13560190665,2018-04-09 22:12:31,2018-04-09 22:17:19,288,新疆维吾尔自治区,河南省
梁鹏,张涛,15778423030,15032293356,2016-07-08 04:56:22,2016-07-08 05:00:54,272,香港特别行政区,西藏自治区
张倩,杜泽文,15151889601,15733218050,2018-07-06 03:35:47,2018-07-06 03:37:11,84,上海市,内蒙古自治区
王世昌,杨力谋,13269361119,18301589432,2018-11-03 19:04:34,2018-11-03 19:11:13,399,上海市,上海市
张涛,李进全,15032293356,15338595369,2018-03-12 05:51:55,2018-03-12 05:56:41,286,广西壮族自治区,江西省
梁鹏,王世昌,15778423030,13269361119,2016-01-15 00:23:11,2016-01-15 00:26:36,205,澳门特别行政区,河南省
温海英,朱尚宽,13520404983,18332562075,2016-09-22 16:55:16,2016-09-22 16:59:53,277,台湾省,西藏自治区
杨力谋,梁鹏,18301589432,15778423030,2016-09-11 12:18:04,2016-09-11 12:22:08,244,甘肃省,上海市
刘飞飞,张倩,15732648446,15151889601,2018-08-29 23:17:25,2018-08-29 23:25:17,472,云南省,上海市
杨力谋,王世昌,18301589432,13269361119,2017-11-28 23:05:16,2017-11-28 23:05:41,25,安徽省,江苏省
唐会华,郭美彤,13560190665,18641241020,2018-08-04 10:47:39,2018-08-04 10:48:37,58,天津市,台湾省
李进全,朱尚宽,15338595369,18332562075,2016-09-17 22:54:07,2016-09-17 22:58:12,245,浙江省,重庆市
温海英,张阳,13520404983,17731088562,2017-09-08 21:50:56,2017-09-08 21:53:37,161,澳门特别行政区,四川省
刘飞飞,杨力谋,15732648446,18301589432,2018-03-22 04:27:16,2018-03-22 04:29:37,141,吉林省,山东省
张涛,任宗阳,15032293356,15614201525,2016-08-17 14:48:13,2016-08-17 14:55:56,463,江西省,云南省
郭美彤,梁鹏,18641241020,15778423030,2018-01-29 02:25:28,2018-01-29 02:30:50,322,辽宁省,陕西省
张阳,杜泽文,17731088562,15733218050,2017-04-28 22:57:29,2017-04-28 23:02:35,306,山东省,澳门特别行政区
刘能宗,菜中路,18620192711,13799999999,2016-10-31 22:25:09,2016-10-31 22:31:28,379,上海市,山西省
邓二,菜中路,13666666666,13799999999,2018-11-01 09:54:36,2018-11-01 10:02:21,465,黑龙江省,宁夏回族自治区
王世昌,张阳,13269361119,17731088562,2018-10-20 18:29:01,2018-10-20 18:35:11,370,内蒙古自治区,西藏自治区
朱尚宽,菜中路,18332562075,13799999999,2017-03-17 17:39:50,2017-03-17 17:46:57,427,西藏自治区,台湾省
菜中路,刘飞飞,13799999999,15732648446,2017-08-28 01:00:21,2017-08-28 01:05:32,311,云南省,贵州省
刘柳,任宗阳,13566666666,15614201525,2017-09-09 18:51:24,2017-09-09 18:54:01,157,澳门特别行政区,山西省
郭美彤,张涛,18641241020,15032293356,2016-05-17 01:03:50,2016-05-17 01:04:40,50,浙江省,宁夏回族自治区
杨力谋,刘柳,18301589432,13566666666,2017-05-01 13:47:31,2017-05-01 13:53:16,345,上海市,四川省
刘飞飞,张涛,15732648446,15032293356,2018-09-17 11:52:05,2018-09-17 12:00:19,494,湖南省,香港特别行政区
李进全,段光星,15338595369,13341109505,2017-04-28 20:37:52,2017-04-28 20:38:11,19,山东省,内蒙古自治区
梁鹏,张阳,15778423030,17731088562,2018-10-27 09:17:27,2018-10-27 09:24:31,424,江苏省,重庆市
邓二,任宗阳,13666666666,15614201525,2018-12-19 06:25:34,2018-12-19 06:29:19,225,江西省,宁夏回族自治区
张涛,菜中路,15032293356,13799999999,2016-02-09 06:56:10,2016-02-09 06:58:04,114,海南省,新疆维吾尔自治区
郭美彤,刘能宗,18641241020,18620192711,2018-06-25 23:14:58,2018-06-25 23:17:06,128,河南省,四川省
张阳,刘飞飞,17731088562,15732648446,2016-02-13 11:02:58,2016-02-13 11:05:05,127,广东省,新疆维吾尔自治区
杨力谋,邓二,18301589432,13666666666,2018-02-13 23:23:52,2018-02-13 23:27:01,189,辽宁省,宁夏回族自治区
杜泽文,刘能宗,15733218050,18620192711,2018-01-11 12:13:14,2018-01-11 12:19:09,355,江西省,江西省
温海英,刘能宗,13520404983,18620192711,2018-03-21 08:22:26,2018-03-21 08:28:33,367,澳门特别行政区,山东省
朱尚宽,郭美彤,18332562075,18641241020,2017-06-22 11:32:39,2017-06-22 11:35:00,141,澳门特别行政区,黑龙江省
邓二,杨力谋,13666666666,18301589432,2017-04-05 01:02:22,2017-04-05 01:09:04,402,吉林省,台湾省
杨力谋,任宗阳,18301589432,15614201525,2017-11-26 21:26:00,2017-11-26 21:33:35,455,香港特别行政区,甘肃省
王世昌,邓二,13269361119,13666666666,2018-10-09 06:03:51,2018-10-09 06:04:56,65,香港特别行政区,香港特别行政区
梁鹏,郭美彤,15778423030,18641241020,2018-02-11 15:55:28,2018-02-11 16:03:15,467,河北省,陕西省
杜泽文,张涛,15733218050,15032293356,2017-04-12 05:28:18,2017-04-12 05:35:16,418,安徽省,香港特别行政区
王世昌,杜泽文,13269361119,15733218050,2018-11-29 18:07:30,2018-11-29 18:09:06,96,河南省,海南省
段光星,邓二,13341109505,13666666666,2016-12-20 08:38:20,2016-12-20 08:46:17,477,吉林省,河南省
李进全,段光星,15338595369,13341109505,2017-05-28 08:09:52,2017-05-28 08:14:52,300,浙江省,湖北省
段光星,郭美彤,13341109505,18641241020,2017-11-18 13:00:03,2017-11-18 13:04:52,289,西藏自治区,甘肃省
刘能宗,王世昌,18620192711,13269361119,2016-05-05 13:01:38,2016-05-05 13:05:59,261,江苏省,江西省
邓二,菜中路,13666666666,13799999999,2018-02-07 23:52:07,2018-02-07 23:59:27,440,宁夏回族自治区,湖北省
段光星,邓二,13341109505,13666666666,2016-09-26 06:54:46,2016-09-26 06:55:07,21,江西省,四川省
邓二,郭美彤,13666666666,18641241020,2016-09-11 06:31:33,2016-09-11 06:34:23,170,内蒙古自治区,贵州省
张倩,刘飞飞,15151889601,15732648446,2017-10-31 02:54:43,2017-10-31 03:01:31,408,四川省,西藏自治区
唐会华,温海英,13560190665,13520404983,2016-07-05 05:35:46,2016-07-05 05:38:23,157,内蒙古自治区,海南省
温海英,任宗阳,13520404983,15614201525,2017-11-17 22:44:46,2017-11-17 22:51:04,378,江苏省,陕西省
梁鹏,任宗阳,15778423030,15614201525,2016-08-01 00:47:37,2016-08-01 00:51:00,203,辽宁省,辽宁省
梁鹏,张阳,15778423030,17731088562,2018-11-20 02:13:49,2018-11-20 02:19:42,353,青海省,贵州省
朱尚宽,李进全,18332562075,15338595369,2016-01-27 12:00:30,2016-01-27 12:08:42,492,宁夏回族自治区,山东省
张涛,刘能宗,15032293356,18620192711,2017-02-28 13:01:38,2017-02-28 13:07:28,350,云南省,内蒙古自治区
郭美彤,杜泽文,18641241020,15733218050,2017-03-18 02:28:53,2017-03-18 02:29:45,52,内蒙古自治区,青海省
杜泽文,邓二,15733218050,13666666666,2017-09-30 14:45:51,2017-09-30 14:47:34,103,吉林省,重庆市
温海英,张阳,13520404983,17731088562,2017-02-18 12:40:22,2017-02-18 12:43:04,162,黑龙江省,浙江省
张倩,王世昌,15151889601,13269361119,2018-09-28 05:51:02,2018-09-28 05:51:42,40,山东省,青海省
梁鹏,张阳,15778423030,17731088562,2016-06-23 14:11:58,2016-06-23 14:16:52,294,西藏自治区,云南省
张涛,温海英,15032293356,13520404983,2018-03-15 22:05:09,2018-03-15 22:08:21,192,贵州省,湖南省
任宗阳,邓二,15614201525,13666666666,2016-05-24 09:46:06,2016-05-24 09:49:12,186,浙江省,新疆维吾尔自治区'

接着在头歌实验界面上传

检查上传的文件名

ls /data/workspace/userfiles/

如果为文件名不为1.sh,而是输出 1.sh.txt,则输入命令

mv /data/workspace/userfiles/1.sh.txt /data/workspace/userfiles/1.sh

最后输入命令

rm /data/workspace/myshixun/step1/1.sh
cp /data/workspace/userfiles/1.sh /data/workspace/myshixun/step1/

测评即可