序列化Writable接口

悠然予夏

于 2022-10-02 10:30:51 发布

阅读量907

点赞数

分类专栏：大数据文章标签： java 开发语言 hadoop mapreduce 大数据

本文链接：https://blog.csdn.net/weixin_52851967/article/details/127140402

版权

大数据专栏收录该内容

28 篇文章 0 订阅

订阅专栏

基本序列化类型往往不能满足所有需求，比如在Hadoop框架内部传递一个自定义bean对象，那么该对象就需要实现Writable序列化接口。

实现Writable序列化步骤如下

（1）必须实现Writable接口

（2）反序列化时，需要反射调用空参构造函数，所以必须有空参构造

public CustomBean() {
    super();
}

（3）重写序列化方法

@Override
public void write(DataOutput out) throws IOException {
    ....
}

（4）重写反序列化方法

@Override
public void readFields(DataInput in) throws IOException {
    ....
}

（5）反序列化的字段顺序和序列化字段的顺序必须完全一致

（6）方便展示结果数据，需要重写bean对象的toString()方法，可以自定义分隔符

（7）如果自定义Bean对象需要放在Mapper输出KV中的K,则该对象还需实现Comparable接口，因为因为MapReduce框中的Shuffle过程要求对key必须能排序！！

@Override
public int compareTo(CustomBean o) {
    // 自定义排序规则
    return this.num > o.getNum() ? -1 : 1;
}

Writable接口案例

1、需求

统计每台智能音箱设备内容播放时长

原始日志格式

001     001577c3     kar_890809             120.196.100.99     1116         954                       200
日志id 设备id         appkey(合作硬件厂商)   网络ip         自有内容时长(秒) 第三方内容时长(秒)     网络状态码

输出结果

001577c3     11160         9540                 20700
设备id     自有内容时长(秒) 第三方内容时长(秒)     总时长

mr编程总结

map()方法输出的kv以及具体类型如何确定？mr中map()方法输出的key如果是相同key则会去往同个reduce调用reduce方法。
reduce()方法中输入参数kv，key:map()方法输出某个key，而value:value是一个集合，这个集合中的数据都来自于map输出的kv，而且是k相同的所有kv对的value集合在一起。

整体思路分析

Map阶段：

读取一行文本数据，按照制表符切分
抽取出自由内容时长，第三方内容时长，设备id
输出：key-->设备id，value:封装一个bean对象，bean对象携带自由时长，第三方内容时长，设备id
自定义bean对象作为value输出，需要实现writable序列化接口

Reduce阶段：

在reduce方法中直接遍历迭代器，累加时长即可

原始数据：

01	a00df6s	kar	120.196.100.99	384	33	200
11	0sfs01	kar	120.196.100.99	198	86	200
21	adfd00fd5	pandora	120.196.100.99	513	261	200
31	0ad0s7	pandora	120.196.100.99	840	413	200
41	0sfs01	kar	120.196.100.99	190	401	200
51	00fdaf3	kar	120.196.100.99	273	527	200
61	00fdaf3	pandora	120.196.100.99	950	526	200
71	0a0fe2	kar	120.196.100.99	730	496	200
81	0a0fe2	pandora	120.196.100.99	367	759	200
91	0sfs01	kar	120.196.100.99	529	484	200
101	adfd00fd5	kar	120.196.100.99	516	18	200
111	a00df6s	pandora	120.196.100.99	300	745	200
121	0a0fe2	kar	120.196.100.99	416	847	200
131	00fdaf3	kar	120.196.100.99	268	489	200
141	00wersa4	kar	120.196.100.99	74	815	200
151	00wersa4	pandora	120.196.100.99	445	293	200
161	00fdaf3	kar	120.196.100.99	673	77	200
171	a00df6s	kar	120.196.100.99	378	138	200
181	0sfs01	pandora	120.196.100.99	55	80	200
191	00fdaf3	kar	120.196.100.99	699	7	200
201	00fdaf3	pandora	120.196.100.99	752	687	200
211	00wersa4	kar	120.196.100.99	448	117	200
221	a00df6s	pandora	120.196.100.99	17	771	200
231	00fdaf3	kar	120.196.100.99	732	701	200
241	0ad0s7	pandora	120.196.100.99	923	24	200
251	0sfs01	pandora	120.196.100.99	476	781	200
261	adfd00fd5	pandora	120.196.100.99	655	163	200
271	0a0fe2	pandora	120.196.100.99	128	251	200
281	00fdaf3	kar	120.196.100.99	704	149	200
291	a00df6s	pandora	120.196.100.99	839	383	200
301	0sfs01	pandora	120.196.100.99	230	38	200
311	adfd00fd5	kar	120.196.100.99	100	661	200
321	adfd00fd5	kar	120.196.100.99	839	442	200
331	0sfs01	pandora	120.196.100.99	165	256	200
341	a00df6s	pandora	120.196.100.99	950	247	200
351	0ad0s7	pandora	120.196.100.99	791	760	200
361	0a0fe2	pandora	120.196.100.99	118	180	200
371	adfd00fd5	kar	120.196.100.99	749	581	200
381	0a0fe2	pandora	120.196.100.99	69	324	200
391	0ad0s7	pandora	120.196.100.99	337	144	200
401	0a0fe2	kar	120.196.100.99	43	120	200
411	adfd00fd5	pandora	120.196.100.99	311	608	200
421	00wersa4	pandora	120.196.100.99	10	680	200
431	00wersa4	pandora	120.196.100.99	206	843	200
441	00fdaf3	kar	120.196.100.99	733	58	200
451	00wersa4	kar	120.196.100.99	247	528	200
461	00fdaf3	pandora	120.196.100.99	403	121	200
471	adfd00fd5	pandora	120.196.100.99	494	771	200
481	00fdaf3	kar	120.196.100.99	770	511	200
491	0ad0s7	kar	120.196.100.99	686	313	200
501	adfd00fd5	pandora	120.196.100.99	566	914	200
511	00fdaf3	kar	120.196.100.99	895	687	200
521	00wersa4	kar	120.196.100.99	217	322	200
531	0sfs01	kar	120.196.100.99	585	766	200
541	adfd00fd5	kar	120.196.100.99	450	209	200
551	00fdaf3	kar	120.196.100.99	392	658	200
561	a00df6s	kar	120.196.100.99	33	689	200
571	0a0fe2	kar	120.196.100.99	71	550	200
581	0sfs01	pandora	120.196.100.99	742	345	200
591	a00df6s	pandora	120.196.100.99	335	640	200
601	00fdaf3	kar	120.196.100.99	807	752	200
611	00fdaf3	kar	120.196.100.99	447	811	200
621	00wersa4	kar	120.196.100.99	32	120	200
631	a00df6s	pandora	120.196.100.99	170	472	200
641	0sfs01	kar	120.196.100.99	538	915	200
651	0sfs01	pandora	120.196.100.99	870	659	200
661	0a0fe2	pandora	120.196.100.99	158	97	200
671	0ad0s7	pandora	120.196.100.99	97	276	200
681	adfd00fd5	pandora	120.196.100.99	713	80	200
691	0a0fe2	pandora	120.196.100.99	324	818	200
701	00fdaf3	pandora	120.196.100.99	808	386	200
711	00wersa4	kar	120.196.100.99	418	129	200
721	0sfs01	kar	120.196.100.99	994	842	200
731	0ad0s7	pandora	120.196.100.99	639	525	200
741	a00df6s	kar	120.196.100.99	827	134	200
751	00wersa4	kar	120.196.100.99	594	44	200
761	0ad0s7	kar	120.196.100.99	197	558	200
771	00wersa4	pandora	120.196.100.99	477	832	200
781	0a0fe2	pandora	120.196.100.99	870	283	200
791	00fdaf3	pandora	120.196.100.99	50	50	200
801	a00df6s	pandora	120.196.100.99	158	488	200
811	00wersa4	kar	120.196.100.99	701	736	200
821	0sfs01	pandora	120.196.100.99	127	316	200
831	00wersa4	pandora	120.196.100.99	670	719	200
841	adfd00fd5	kar	120.196.100.99	405	630	200
851	00wersa4	kar	120.196.100.99	78	976	200
861	00wersa4	kar	120.196.100.99	510	527	200
871	adfd00fd5	kar	120.196.100.99	62	407	200
881	00fdaf3	kar	120.196.100.99	873	403	200
891	0a0fe2	kar	120.196.100.99	157	431	200
901	a00df6s	pandora	120.196.100.99	112	319	200
911	adfd00fd5	kar	120.196.100.99	144	14	200
921	00fdaf3	kar	120.196.100.99	879	444	200
931	a00df6s	pandora	120.196.100.99	882	47	200
941	adfd00fd5	pandora	120.196.100.99	620	114	200
951	00wersa4	kar	120.196.100.99	35	59	200
961	00wersa4	kar	120.196.100.99	79	181	200
971	00wersa4	kar	120.196.100.99	34	758	200
981	a00df6s	pandora	120.196.100.99	649	907	200
991	0ad0s7	kar	120.196.100.99	206	408	200
1001	0a0fe2	pandora	120.196.100.99	710	300	200
1011	0sfs01	pandora	120.196.100.99	326	232	200
1021	0ad0s7	pandora	120.196.100.99	749	648	200
1031	00fdaf3	kar	120.196.100.99	749	522	200
1041	0ad0s7	kar	120.196.100.99	516	692	200
1051	adfd00fd5	kar	120.196.100.99	242	706	200
1061	a00df6s	kar	120.196.100.99	6	477	200
1071	00wersa4	pandora	120.196.100.99	401	79	200
1081	00wersa4	kar	120.196.100.99	238	178	200
1091	0ad0s7	pandora	120.196.100.99	613	640	200
1101	00fdaf3	pandora	120.196.100.99	83	925	200
1111	0ad0s7	pandora	120.196.100.99	424	937	200
1121	00wersa4	pandora	120.196.100.99	468	194	200
1131	0sfs01	pandora	120.196.100.99	790	503	200
1141	00fdaf3	pandora	120.196.100.99	51	839	200
1151	0sfs01	kar	120.196.100.99	480	399	200
1161	00fdaf3	pandora	120.196.100.99	884	130	200
1171	a00df6s	kar	120.196.100.99	580	955	200
1181	0ad0s7	pandora	120.196.100.99	316	358	200
1191	a00df6s	pandora	120.196.100.99	255	884	200
1201	adfd00fd5	pandora	120.196.100.99	244	813	200
1211	0sfs01	pandora	120.196.100.99	693	504	200
1221	adfd00fd5	pandora	120.196.100.99	708	222	200
1231	0a0fe2	pandora	120.196.100.99	125	845	200
1241	0sfs01	kar	120.196.100.99	274	528	200
1251	0ad0s7	pandora	120.196.100.99	679	196	200
1261	00wersa4	pandora	120.196.100.99	859	541	200
1271	0sfs01	kar	120.196.100.99	279	606	200
1281	00wersa4	pandora	120.196.100.99	62	720	200
1291	adfd00fd5	pandora	120.196.100.99	833	466	200
1301	0ad0s7	kar	120.196.100.99	622	438	200
1311	00wersa4	pandora	120.196.100.99	356	916	200
1321	adfd00fd5	pandora	120.196.100.99	214	640	200
1331	00wersa4	kar	120.196.100.99	369	15	200
1341	0a0fe2	kar	120.196.100.99	404	367	200
1351	0sfs01	kar	120.196.100.99	827	916	200
1361	00fdaf3	pandora	120.196.100.99	88	798	200
1371	0sfs01	kar	120.196.100.99	175	136	200
1381	00wersa4	pandora	120.196.100.99	132	988	200
1391	0sfs01	kar	120.196.100.99	307	369	200
1401	a00df6s	pandora	120.196.100.99	483	951	200
1411	adfd00fd5	pandora	120.196.100.99	760	689	200
1421	0sfs01	pandora	120.196.100.99	75	449	200
1431	00fdaf3	kar	120.196.100.99	17	969	200
1441	a00df6s	pandora	120.196.100.99	685	541	200
1451	00fdaf3	pandora	120.196.100.99	690	935	200
1461	0a0fe2	kar	120.196.100.99	161	794	200
1471	0ad0s7	kar	120.196.100.99	772	400	200
1481	0sfs01	kar	120.196.100.99	885	567	200
1491	0a0fe2	kar	120.196.100.99	597	646	200
1501	0sfs01	pandora	120.196.100.99	93	157	200
1511	0a0fe2	pandora	120.196.100.99	290	18	200
1521	adfd00fd5	pandora	120.196.100.99	241	368	200
1531	a00df6s	kar	120.196.100.99	47	658	200
1541	0a0fe2	pandora	120.196.100.99	899	878	200
1551	00wersa4	pandora	120.196.100.99	787	956	200
1561	a00df6s	pandora	120.196.100.99	888	21	200
1571	adfd00fd5	kar	120.196.100.99	439	41	200
1581	0a0fe2	kar	120.196.100.99	564	792	200
1591	adfd00fd5	pandora	120.196.100.99	661	682	200
1601	0a0fe2	kar	120.196.100.99	680	11	200
1611	a00df6s	kar	120.196.100.99	18	566	200
1621	a00df6s	kar	120.196.100.99	613	658	200
1631	00fdaf3	kar	120.196.100.99	574	242	200
1641	0ad0s7	pandora	120.196.100.99	992	453	200
1651	0a0fe2	kar	120.196.100.99	592	334	200
1661	0sfs01	kar	120.196.100.99	30	719	200
1671	00wersa4	pandora	120.196.100.99	988	494	200
1681	0a0fe2	kar	120.196.100.99	654	306	200
1691	0sfs01	kar	120.196.100.99	519	198	200
1701	adfd00fd5	pandora	120.196.100.99	796	83	200
1711	0a0fe2	pandora	120.196.100.99	318	701	200
1721	0a0fe2	pandora	120.196.100.99	98	244	200
1731	adfd00fd5	pandora	120.196.100.99	293	634	200
1741	00wersa4	pandora	120.196.100.99	62	812	200
1751	0sfs01	kar	120.196.100.99	548	130	200
1761	0a0fe2	pandora	120.196.100.99	496	120	200
1771	0sfs01	kar	120.196.100.99	737	223	200
1781	0a0fe2	kar	120.196.100.99	498	262	200
1791	adfd00fd5	kar	120.196.100.99	218	193	200
1801	00fdaf3	kar	120.196.100.99	120	564	200
1811	adfd00fd5	pandora	120.196.100.99	177	209	200
1821	00fdaf3	kar	120.196.100.99	392	397	200
1831	0a0fe2	pandora	120.196.100.99	824	336	200
1841	a00df6s	pandora	120.196.100.99	721	395	200
1851	0sfs01	kar	120.196.100.99	764	660	200
1861	0ad0s7	pandora	120.196.100.99	589	101	200
1871	00wersa4	pandora	120.196.100.99	356	356	200
1881	0sfs01	pandora	120.196.100.99	216	307	200
1891	00fdaf3	kar	120.196.100.99	1	582	200
1901	00wersa4	kar	120.196.100.99	912	436	200
1911	0sfs01	pandora	120.196.100.99	405	719	200
1921	a00df6s	kar	120.196.100.99	941	542	200
1931	00wersa4	kar	120.196.100.99	966	490	200
1941	00fdaf3	pandora	120.196.100.99	960	454	200
1951	0ad0s7	kar	120.196.100.99	545	39	200
1961	00fdaf3	pandora	120.196.100.99	293	299	200
1971	0a0fe2	kar	120.196.100.99	825	462	200
1981	adfd00fd5	kar	120.196.100.99	34	751	200
1991	a00df6s	kar	120.196.100.99	807	569	200
2001	0a0fe2	pandora	120.196.100.99	160	973	200
2011	00wersa4	pandora	120.196.100.99	497	637	200
2021	00fdaf3	kar	120.196.100.99	302	27	200
2031	0sfs01	pandora	120.196.100.99	653	104	200
2041	0a0fe2	kar	120.196.100.99	963	65	200
2051	0ad0s7	kar	120.196.100.99	204	828	200
2061	00wersa4	kar	120.196.100.99	91	656	200
2071	0a0fe2	pandora	120.196.100.99	937	34	200
2081	adfd00fd5	pandora	120.196.100.99	350	748	200
2091	0a0fe2	pandora	120.196.100.99	97	688	200
2101	0ad0s7	kar	120.196.100.99	430	338	200
2111	0sfs01	pandora	120.196.100.99	595	827	200
2121	0a0fe2	kar	120.196.100.99	726	578	200
2131	adfd00fd5	pandora	120.196.100.99	31	501	200
2141	0ad0s7	kar	120.196.100.99	389	925	200
2151	adfd00fd5	pandora	120.196.100.99	136	559	200
2161	00wersa4	pandora	120.196.100.99	3	927	200
2171	adfd00fd5	pandora	120.196.100.99	908	583	200
2181	00wersa4	pandora	120.196.100.99	921	598	200
2191	adfd00fd5	pandora	120.196.100.99	112	956	200
2201	0sfs01	kar	120.196.100.99	115	32	200
2211	a00df6s	pandora	120.196.100.99	86	996	200
2221	0ad0s7	pandora	120.196.100.99	790	705	200
2231	0a0fe2	kar	120.196.100.99	24	548	200
2241	a00df6s	kar	120.196.100.99	611	58	200
2251	00fdaf3	pandora	120.196.100.99	500	800	200
2261	adfd00fd5	kar	120.196.100.99	852	530	200
2271	00wersa4	pandora	120.196.100.99	208	957	200
2281	0a0fe2	kar	120.196.100.99	900	137	200
2291	adfd00fd5	kar	120.196.100.99	292	550	200
2301	0a0fe2	kar	120.196.100.99	977	983	200
2311	00fdaf3	pandora	120.196.100.99	517	244	200
2321	0a0fe2	pandora	120.196.100.99	419	481	200
2331	00fdaf3	kar	120.196.100.99	117	500	200
2341	adfd00fd5	pandora	120.196.100.99	76	68	200
2351	00wersa4	pandora	120.196.100.99	318	285	200
2361	00wersa4	pandora	120.196.100.99	263	72	200
2371	00fdaf3	kar	120.196.100.99	222	150	200
2381	00wersa4	pandora	120.196.100.99	210	493	200
2391	0sfs01	pandora	120.196.100.99	538	4	200
2401	0ad0s7	kar	120.196.100.99	214	880	200
2411	adfd00fd5	pandora	120.196.100.99	709	369	200
2421	0a0fe2	kar	120.196.100.99	411	383	200
2431	0ad0s7	pandora	120.196.100.99	15	350	200
2441	0a0fe2	kar	120.196.100.99	135	440	200
2451	0sfs01	pandora	120.196.100.99	120	737	200
2461	0a0fe2	kar	120.196.100.99	941	510	200
2471	adfd00fd5	pandora	120.196.100.99	363	263	200
2481	0sfs01	kar	120.196.100.99	928	427	200
2491	0ad0s7	pandora	120.196.100.99	590	635	200
2501	a00df6s	pandora	120.196.100.99	335	967	200
2511	0sfs01	pandora	120.196.100.99	214	756	200
2521	0a0fe2	kar	120.196.100.99	208	119	200
2531	adfd00fd5	kar	120.196.100.99	821	579	200
2541	00wersa4	pandora	120.196.100.99	319	262	200
2551	adfd00fd5	pandora	120.196.100.99	310	10	200
2561	00fdaf3	pandora	120.196.100.99	389	103	200
2571	00wersa4	kar	120.196.100.99	772	957	200
2581	0a0fe2	pandora	120.196.100.99	43	359	200
2591	00wersa4	pandora	120.196.100.99	920	4	200
2601	0a0fe2	kar	120.196.100.99	542	391	200
2611	adfd00fd5	kar	120.196.100.99	730	407	200
2621	0a0fe2	pandora	120.196.100.99	682	712	200
2631	0a0fe2	pandora	120.196.100.99	349	304	200
2641	00fdaf3	pandora	120.196.100.99	529	678	200
2651	0sfs01	kar	120.196.100.99	109	100	200
2661	0ad0s7	pandora	120.196.100.99	715	501	200
2671	0sfs01	kar	120.196.100.99	401	41	200
2681	a00df6s	kar	120.196.100.99	5	110	200
2691	00wersa4	pandora	120.196.100.99	155	377	200
2701	00fdaf3	kar	120.196.100.99	172	954	200
2711	0a0fe2	kar	120.196.100.99	772	838	200
2721	00wersa4	kar	120.196.100.99	984	397	200
2731	00wersa4	kar	120.196.100.99	803	451	200
2741	0ad0s7	kar	120.196.100.99	636	380	200
2751	0ad0s7	pandora	120.196.100.99	565	288	200
2761	adfd00fd5	kar	120.196.100.99	74	423	200
2771	adfd00fd5	pandora	120.196.100.99	643	734	200
2781	0sfs01	pandora	120.196.100.99	808	847	200
2791	adfd00fd5	kar	120.196.100.99	377	415	200
2801	0a0fe2	pandora	120.196.100.99	263	968	200
2811	0a0fe2	kar	120.196.100.99	258	287	200
2821	0ad0s7	kar	120.196.100.99	32	248	200
2831	0ad0s7	kar	120.196.100.99	343	213	200
2841	00fdaf3	pandora	120.196.100.99	239	62	200
2851	0sfs01	kar	120.196.100.99	327	58	200
2861	a00df6s	kar	120.196.100.99	146	691	200
2871	adfd00fd5	pandora	120.196.100.99	158	896	200
2881	00fdaf3	kar	120.196.100.99	897	788	200
2891	0sfs01	pandora	120.196.100.99	904	755	200
2901	0sfs01	pandora	120.196.100.99	661	781	200
2911	00wersa4	kar	120.196.100.99	516	379	200
2921	0a0fe2	kar	120.196.100.99	946	791	200
2931	adfd00fd5	pandora	120.196.100.99	83	272	200
2941	a00df6s	pandora	120.196.100.99	350	906	200
2951	0ad0s7	kar	120.196.100.99	711	580	200
2961	0sfs01	pandora	120.196.100.99	162	939	200
2971	00fdaf3	pandora	120.196.100.99	38	956	200
2981	00fdaf3	kar	120.196.100.99	406	567	200
2991	00fdaf3	pandora	120.196.100.99	191	47	200
3001	adfd00fd5	kar	120.196.100.99	78	229	200
3011	0ad0s7	kar	120.196.100.99	122	12	200
3021	adfd00fd5	pandora	120.196.100.99	137	856	200
3031	a00df6s	pandora	120.196.100.99	610	0	200
3041	0a0fe2	kar	120.196.100.99	598	722	200
3051	adfd00fd5	kar	120.196.100.99	195	629	200
3061	0a0fe2	kar	120.196.100.99	550	482	200
3071	00fdaf3	kar	120.196.100.99	781	76	200
3081	a00df6s	pandora	120.196.100.99	623	78	200
3091	00wersa4	pandora	120.196.100.99	513	628	200
3101	00fdaf3	kar	120.196.100.99	893	809	200
3111	00fdaf3	pandora	120.196.100.99	720	447	200
3121	0a0fe2	kar	120.196.100.99	521	662	200
3131	00wersa4	kar	120.196.100.99	169	291	200
3141	0a0fe2	kar	120.196.100.99	676	930	200
3151	a00df6s	kar	120.196.100.99	522	775	200
3161	0ad0s7	kar	120.196.100.99	889	421	200
3171	a00df6s	kar	120.196.100.99	299	833	200
3181	0a0fe2	pandora	120.196.100.99	912	209	200
3191	0sfs01	kar	120.196.100.99	856	670	200
3201	0ad0s7	pandora	120.196.100.99	151	929	200
3211	0a0fe2	pandora	120.196.100.99	213	442	200
3221	0ad0s7	pandora	120.196.100.99	857	11	200
3231	0sfs01	pandora	120.196.100.99	361	659	200
3241	0a0fe2	kar	120.196.100.99	953	374	200
3251	adfd00fd5	kar	120.196.100.99	917	594	200
3261	a00df6s	pandora	120.196.100.99	909	268	200
3271	0ad0s7	kar	120.196.100.99	828	785	200
3281	a00df6s	kar	120.196.100.99	85	953	200
3291	a00df6s	pandora	120.196.100.99	715	757	200
3301	00fdaf3	kar	120.196.100.99	82	105	200
3311	00wersa4	kar	120.196.100.99	575	972	200
3321	a00df6s	pandora	120.196.100.99	305	573	200
3331	0ad0s7	kar	120.196.100.99	675	64	200
3341	0sfs01	pandora	120.196.100.99	488	38	200
3351	adfd00fd5	pandora	120.196.100.99	553	283	200
3361	adfd00fd5	pandora	120.196.100.99	973	608	200
3371	0a0fe2	pandora	120.196.100.99	166	296	200
3381	00fdaf3	kar	120.196.100.99	156	608	200
3391	00wersa4	kar	120.196.100.99	640	880	200
3401	00wersa4	kar	120.196.100.99	679	598	200
3411	a00df6s	kar	120.196.100.99	345	650	200
3421	0a0fe2	kar	120.196.100.99	61	99	200
3431	00fdaf3	pandora	120.196.100.99	139	384	200
3441	0ad0s7	kar	120.196.100.99	901	19	200
3451	00fdaf3	kar	120.196.100.99	379	920	200
3461	0ad0s7	kar	120.196.100.99	153	252	200
3471	a00df6s	pandora	120.196.100.99	816	897	200
3481	adfd00fd5	kar	120.196.100.99	126	237	200
3491	a00df6s	pandora	120.196.100.99	890	993	200
3501	0ad0s7	kar	120.196.100.99	656	855	200
3511	a00df6s	kar	120.196.100.99	21	541	200
3521	a00df6s	pandora	120.196.100.99	264	407	200
3531	00fdaf3	pandora	120.196.100.99	749	849	200
3541	0sfs01	kar	120.196.100.99	975	481	200
3551	adfd00fd5	pandora	120.196.100.99	762	107	200
3561	a00df6s	kar	120.196.100.99	514	446	200
3571	a00df6s	kar	120.196.100.99	123	212	200
3581	00wersa4	kar	120.196.100.99	422	23	200
3591	a00df6s	pandora	120.196.100.99	564	638	200
3601	00fdaf3	kar	120.196.100.99	866	543	200
3611	00fdaf3	pandora	120.196.100.99	601	542	200
3621	adfd00fd5	kar	120.196.100.99	321	748	200
3631	a00df6s	kar	120.196.100.99	458	323	200
3641	0sfs01	kar	120.196.100.99	503	685	200
3651	0ad0s7	kar	120.196.100.99	551	173	200
3661	00wersa4	pandora	120.196.100.99	833	21	200
3671	0a0fe2	kar	120.196.100.99	171	694	200
3681	0sfs01	pandora	120.196.100.99	943	846	200
3691	a00df6s	pandora	120.196.100.99	339	316	200
3701	0a0fe2	kar	120.196.100.99	524	511	200
3711	0ad0s7	pandora	120.196.100.99	438	324	200
3721	adfd00fd5	pandora	120.196.100.99	9	6	200
3731	0ad0s7	kar	120.196.100.99	284	960	200
3741	00fdaf3	pandora	120.196.100.99	191	446	200
3751	0ad0s7	pandora	120.196.100.99	216	618	200
3761	adfd00fd5	pandora	120.196.100.99	812	906	200
3771	a00df6s	kar	120.196.100.99	975	630	200
3781	0a0fe2	kar	120.196.100.99	578	47	200
3791	a00df6s	pandora	120.196.100.99	448	227	200
3801	0a0fe2	kar	120.196.100.99	965	221	200
3811	a00df6s	pandora	120.196.100.99	643	442	200
3821	0a0fe2	pandora	120.196.100.99	819	206	200
3831	a00df6s	pandora	120.196.100.99	420	293	200
3841	00wersa4	kar	120.196.100.99	555	765	200
3851	00wersa4	pandora	120.196.100.99	576	817	200
3861	0a0fe2	kar	120.196.100.99	200	552	200
3871	0ad0s7	kar	120.196.100.99	372	212	200
3881	0sfs01	pandora	120.196.100.99	862	471	200
3891	a00df6s	pandora	120.196.100.99	139	22	200
3901	0ad0s7	kar	120.196.100.99	849	682	200
3911	0ad0s7	pandora	120.196.100.99	62	884	200
3921	00fdaf3	pandora	120.196.100.99	578	453	200
3931	00wersa4	kar	120.196.100.99	867	536	200
3941	0sfs01	pandora	120.196.100.99	915	631	200
3951	0ad0s7	kar	120.196.100.99	456	449	200
3961	0sfs01	pandora	120.196.100.99	887	462	200
3971	00fdaf3	pandora	120.196.100.99	438	329	200
3981	adfd00fd5	pandora	120.196.100.99	117	956	200
3991	00fdaf3	pandora	120.196.100.99	32	178	200
4001	0a0fe2	pandora	120.196.100.99	140	573	200
4011	a00df6s	kar	120.196.100.99	323	800	200
4021	a00df6s	pandora	120.196.100.99	94	262	200
4031	0sfs01	kar	120.196.100.99	22	617	200
4041	00wersa4	kar	120.196.100.99	958	223	200
4051	0sfs01	kar	120.196.100.99	993	367	200
4061	0a0fe2	pandora	120.196.100.99	292	665	200
4071	0a0fe2	kar	120.196.100.99	551	684	200
4081	00wersa4	pandora	120.196.100.99	869	137	200
4091	a00df6s	kar	120.196.100.99	309	596	200
4101	adfd00fd5	kar	120.196.100.99	157	450	200
4111	0ad0s7	pandora	120.196.100.99	90	86	200
4121	0sfs01	kar	120.196.100.99	231	118	200
4131	a00df6s	pandora	120.196.100.99	727	435	200
4141	00wersa4	pandora	120.196.100.99	112	918	200
4151	adfd00fd5	pandora	120.196.100.99	262	61	200
4161	00fdaf3	kar	120.196.100.99	553	164	200
4171	0a0fe2	kar	120.196.100.99	597	321	200
4181	00fdaf3	kar	120.196.100.99	20	464	200
4191	adfd00fd5	kar	120.196.100.99	602	109	200
4201	adfd00fd5	kar	120.196.100.99	508	80	200
4211	0ad0s7	pandora	120.196.100.99	485	513	200
4221	0ad0s7	kar	120.196.100.99	159	237	200
4231	00fdaf3	pandora	120.196.100.99	979	530	200
4241	0sfs01	pandora	120.196.100.99	557	356	200
4251	a00df6s	kar	120.196.100.99	502	810	200
4261	0a0fe2	pandora	120.196.100.99	223	642	200
4271	adfd00fd5	kar	120.196.100.99	720	89	200
4281	a00df6s	kar	120.196.100.99	538	103	200
4291	0a0fe2	pandora	120.196.100.99	615	415	200
4301	0sfs01	kar	120.196.100.99	676	228	200
4311	0a0fe2	kar	120.196.100.99	231	94	200
4321	0sfs01	kar	120.196.100.99	515	756	200
4331	a00df6s	kar	120.196.100.99	885	884	200
4341	adfd00fd5	pandora	120.196.100.99	843	262	200
4351	0a0fe2	kar	120.196.100.99	24	326	200
4361	a00df6s	pandora	120.196.100.99	558	904	200
4371	00fdaf3	pandora	120.196.100.99	128	454	200
4381	adfd00fd5	pandora	120.196.100.99	499	501	200
4391	00wersa4	kar	120.196.100.99	49	38	200
4401	adfd00fd5	pandora	120.196.100.99	9	432	200
4411	00wersa4	kar	120.196.100.99	174	138	200
4421	adfd00fd5	kar	120.196.100.99	138	150	200
4431	0a0fe2	pandora	120.196.100.99	151	130	200
4441	a00df6s	kar	120.196.100.99	907	896	200
4451	00fdaf3	pandora	120.196.100.99	165	892	200
4461	0sfs01	kar	120.196.100.99	816	326	200
4471	adfd00fd5	kar	120.196.100.99	483	461	200
4481	adfd00fd5	kar	120.196.100.99	810	276	200
4491	00fdaf3	kar	120.196.100.99	253	601	200
4501	0ad0s7	pandora	120.196.100.99	634	176	200
4511	0a0fe2	pandora	120.196.100.99	377	710	200
4521	0a0fe2	kar	120.196.100.99	5	673	200
4531	0a0fe2	pandora	120.196.100.99	290	837	200
4541	0a0fe2	pandora	120.196.100.99	623	465	200
4551	0a0fe2	pandora	120.196.100.99	362	568	200
4561	0a0fe2	kar	120.196.100.99	866	508	200
4571	00fdaf3	kar	120.196.100.99	965	173	200
4581	00wersa4	pandora	120.196.100.99	174	670	200
4591	a00df6s	pandora	120.196.100.99	578	377	200
4601	0a0fe2	pandora	120.196.100.99	752	65	200
4611	adfd00fd5	pandora	120.196.100.99	131	564	200
4621	00wersa4	kar	120.196.100.99	349	756	200
4631	00wersa4	pandora	120.196.100.99	33	291	200
4641	0a0fe2	pandora	120.196.100.99	749	316	200
4651	0a0fe2	pandora	120.196.100.99	242	505	200
4661	a00df6s	pandora	120.196.100.99	930	486	200
4671	0a0fe2	pandora	120.196.100.99	445	916	200
4681	00wersa4	kar	120.196.100.99	584	154	200
4691	a00df6s	pandora	120.196.100.99	35	62	200
4701	a00df6s	pandora	120.196.100.99	892	633	200
4711	00wersa4	pandora	120.196.100.99	445	889	200
4721	adfd00fd5	pandora	120.196.100.99	543	664	200
4731	0a0fe2	pandora	120.196.100.99	850	497	200
4741	0sfs01	kar	120.196.100.99	626	691	200
4751	0ad0s7	kar	120.196.100.99	895	886	200
4761	0a0fe2	pandora	120.196.100.99	601	106	200
4771	0a0fe2	pandora	120.196.100.99	718	638	200
4781	a00df6s	pandora	120.196.100.99	538	158	200
4791	0ad0s7	kar	120.196.100.99	514	663	200
4801	0ad0s7	pandora	120.196.100.99	605	996	200
4811	0ad0s7	kar	120.196.100.99	695	234	200
4821	0ad0s7	pandora	120.196.100.99	307	784	200
4831	00fdaf3	kar	120.196.100.99	844	128	200
4841	0a0fe2	kar	120.196.100.99	352	963	200
4851	a00df6s	pandora	120.196.100.99	690	641	200
4861	0a0fe2	kar	120.196.100.99	690	878	200
4871	00fdaf3	kar	120.196.100.99	179	400	200
4881	0ad0s7	kar	120.196.100.99	593	588	200
4891	0ad0s7	pandora	120.196.100.99	405	888	200
4901	00wersa4	pandora	120.196.100.99	783	611	200
4911	00fdaf3	pandora	120.196.100.99	711	793	200
4921	a00df6s	kar	120.196.100.99	95	603	200
4931	0a0fe2	kar	120.196.100.99	145	707	200
4941	0a0fe2	pandora	120.196.100.99	480	597	200
4951	0ad0s7	kar	120.196.100.99	693	288	200
4961	00fdaf3	pandora	120.196.100.99	249	556	200
4971	00wersa4	pandora	120.196.100.99	234	822	200
4981	0a0fe2	kar	120.196.100.99	868	847	200
4991	00wersa4	kar	120.196.100.99	885	707	200

2、编写MapReduce程序

（1）创建SpeakBean对象

package com.lagou.speak;

import org.apache.hadoop.io.Writable;

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;

// 这个类型是map输出kv中value的类型，需要实现writable序列化接口
public class SpeakBean implements Writable {
    // 定义属性
    private Long selfDuration; // 自由内容时长
    private Long thirdPartDuration; // 第三方内容时长
    private String deviceId; // 设备ID
    private Long sumDuration; // 总时长

    // 空参构造
    public SpeakBean() {
    }

    // 有参构造
    public SpeakBean(Long selfDuration, Long thirdPartDuration, String deviceId) {
        this.selfDuration = selfDuration;
        this.thirdPartDuration = thirdPartDuration;
        this.deviceId = deviceId;
        this.sumDuration = this.selfDuration + this.thirdPartDuration;
    }

    // 序列化方法：就是把内容输出到网络或者文本中
    @Override
    public void write(DataOutput dataOutput) throws IOException {
        dataOutput.writeLong(selfDuration);// 自由内容时长
        dataOutput.writeLong(thirdPartDuration);// 第三方内容时长
        dataOutput.writeUTF(deviceId);// 设备ID
        dataOutput.writeLong(sumDuration);// 总时长
    }

    // 反序列化方法
    @Override
    public void readFields(DataInput dataInput) throws IOException {
        this.selfDuration = dataInput.readLong();
        this.thirdPartDuration = dataInput.readLong();
        this.deviceId = dataInput.readUTF();
        this.sumDuration = dataInput.readLong();
    }

    public Long getSelfDuration() {
        return selfDuration;
    }

    public void setSelfDuration(Long selfDuration) {
        this.selfDuration = selfDuration;
    }

    public Long getThirdPartDuration() {
        return thirdPartDuration;
    }

    public void setThirdPartDuration(Long thirdPartDuration) {
        this.thirdPartDuration = thirdPartDuration;
    }

    public String getDeviceId() {
        return deviceId;
    }

    public void setDeviceId(String deviceId) {
        this.deviceId = deviceId;
    }

    public Long getSumDuration() {
        return sumDuration;
    }

    public void setSumDuration(Long sumDuration) {
        this.sumDuration = sumDuration;
    }

    // 方便观察数据
    @Override
    public String toString() {
        return selfDuration + "\t" + thirdPartDuration + "\t" + deviceId + "\t" + sumDuration;
    }
}

（2）编写Mapper类

package com.lagou.speak;

import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

// 四个参数：分为两队KV

/**
 * 第一对kv：map输入参数的kv类型; k-->一行文本偏移量，v-->一行文本内容
 * 第二队kv：map输出参数kv类型; k-->map输出的key类型，value:map输出的value类型
 */
public class SpeakMapper extends Mapper<LongWritable, Text, Text, SpeakBean> {
    /**
     * 转换接收到的text数据为String
     * 按照制表符进行切分：得到自有内容时长，第三方内容时长，设备id，封装为SpeakBean
     * 直接输出：k-->设备id，value-->speakBane
     */
    Text device_id = new Text();

    @Override
    protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, SpeakBean>.Context context) throws IOException, InterruptedException {
        // 转换接收到的text数据为String
        String line = value.toString();
        String[] fields = line.split("\t");
        // 自有内容时长
        String selfDuration = fields[fields.length - 3];
        // 第三方内容时长
        String thirdPartDuration = fields[fields.length - 2];
        // 设备id
        String deviceId = fields[1];
        SpeakBean bean = new SpeakBean(Long.parseLong(selfDuration), Long.parseLong(thirdPartDuration), deviceId);

        // 直接输出: k-->设备id, value -->speakBean
        device_id.set(deviceId);
        context.write(device_id, bean);
    }
}

（3）编写Reducer

package com.lagou.speak;

import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

import java.io.IOException;

public class SpeakReducer extends Reducer<Text, SpeakBean, Text, SpeakBean> {
    @Override
    protected void reduce(Text key, Iterable<SpeakBean> values, Reducer<Text, SpeakBean, Text, SpeakBean>.Context context) throws IOException, InterruptedException {
        // 定义时长累加的初始值（这是局部变量，只对当前该方法起作用，用完就销毁了）
        Long self_duration = 0L;
        Long third_part_duration = 0L;


        // reduce 方法的key：map输出的是某一个key
        // reduce方法的value：map输出的kv对中相同key的value组成的一个集合
        // reduce逻辑：遍历迭代器累加时长即可
        for (SpeakBean bean : values) {
            Long selfDuration = bean.getSelfDuration();
            Long thirdPartDuration = bean.getThirdPartDuration();
            self_duration += selfDuration;
            third_part_duration += thirdPartDuration;
        }

        // 输出，封装成一个bean对象输出
        SpeakBean bean = new SpeakBean(self_duration, third_part_duration, key.toString());
        context.write(key, bean);
    }
}

（4）编写驱动

package com.lagou.speak;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import java.io.IOException;

public class SpeakDriver {
    public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException {
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf, "speakDriver");
        // 设置jar包本地路径
        job.setJarByClass(SpeakDriver.class);
        // 使用mapper和reducer
        job.setMapperClass(SpeakMapper.class);
        job.setReducerClass(SpeakReducer.class);

        // 设置map输出kv类型
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(SpeakBean.class);

        // 设置reduce输出
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(SpeakBean.class);


        // 读取数据路径
        FileInputFormat.setInputPaths(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));

        // 提交任务
        boolean flag = job.waitForCompletion(true);

        System.exit(flag ? 0 : 1);
    }
}

注意：本次所用的依赖与上篇博文依赖相同，故省略。

MapReduce编程规范及示例编写_悠然予夏的博客-CSDN博客