ETL-AWK应用

在日常的etl中，我们经常是将不同的DB的数据合并，但是也有很多的情况下是将一份数据拆分，这个例子就是这样的，拿到一份数据，样例如下

000000000000001 953301488c75b7a8|c1ea053aec866a22       11:84:95:27:63:74|00:1b:96:de:6d:56     北京市,本溪市,温州市            8079|酷派|手
机|5.0-4.6英寸|双卡双待|500-999 Android+4.0.3   480*854*213     01010302;0.24|010103;0.24|010305;0.24|01010301;0.24|010106;1.11|01010602;1.1
1       02011304;0.71|02011207;0.71|020106;1.19|02010703;5.0|02011308;1.67|020103;2.22|02011303;2.05|02010305;0.24|020113;35.55|020107;5.0|0
2011110;0.56|02010501;0.56|02010702;0.63|02011310;0.4|02010602;0.24|02010505;1.94|02010307;2.22|020108;0.24|02010701;0.24|02010601;1.19|0201
1306;1.35|020111;3.33|020105;1.94|02011301;33.89|02011101;0.24|02011112;2.78|02011305;0.24|02010801;0.24|020112;0.71|02011302;0.63      0304
02;8.86|030502;8.86|030102;100.0|030208;100.0                   070503;100.0|070402;100.0

需求如下：
1.将1-8列另存文件
2.9-14列用|拆分为多条数据，用；分割为多列，将文件的第一列作为首列，最后一列是拆分的；前面类别编码的级别，公式为长度/2,输出类似如下的文件
000000000000001 02011304 0.71 4

想了很久，想用程序实现，但是性能很差，最后尝试了很多，最终使用awk实现

#!/bin/sh

awk -F'\t' '{for(i=1;i<=8;i++) printf $i"\t";print ""}' blue.dat  > imei_usertagmain.txt
awk -F"\t" '{_l=split($9,a,"|");for(i=1;i<=_l;i++)print $1"\t",gensub(";","\t","g",a[i])"\t",(index(a[i],";")-1)/2}' blue.dat > imei_usertag1.txt
awk -F"\t" '{_l=split($10,a,"|");for(i=1;i<=_l;i++)print $1"\t",gensub(";","\t","g",a[i])"\t",(index(a[i],";")-1)/2}' blue.dat > imei_usertag2.txt
awk -F"\t" '{_l=split($11,a,"|");for(i=1;i<=_l;i++)print $1"\t",gensub(";","\t","g",a[i])"\t",(index(a[i],";")-1)/2}' blue.dat > imei_usertag3.txt
awk -F"\t" '{_l=split($12,a,"|");for(i=1;i<=_l;i++)print $1"\t",gensub(";","\t","g",a[i])"\t",(index(a[i],";")-1)/2}' blue.dat > imei_usertag4.txt
awk -F"\t" '{_l=split($13,a,"|");for(i=1;i<=_l;i++)print $1"\t",gensub(";","\t","g",a[i])"\t",(index(a[i],";")-1)/2}' blue.dat > imei_usertag6.txt
awk -F"\t" '{_l=split($14,a,"|");for(i=1;i<=_l;i++)print $1"\t",gensub(";","\t","g",a[i])"\t",(index(a[i],";")-1)/2}' blue.dat > imei_usertag7.txt

生成20G的输出文件，用时40分钟，性能很好。