服务器文件编码格式发生变化,使用BufferedReader和BufferedWriter转换文件编码格式遇到的坑~...

最近一个省的导出功能遇到了一个奇葩的BUG，自个折腾了很久很无助，先记录吧~

启初还以为是部署的代码版本过久导致的，于是又重新去部署一份最新代码还是无法解决。因为导出功能是先从gp数据库上copy出 UTF-8的csv文件,基本上很快生成。可是到了需要到数据转换成GBK编码格式时候就出问题了。

BUG是这样的，开始导出并且转换小数据量呢基本上莫得问题，可到了50W条数据以后就变成假死状态。

起初我看代码都发现不了问题所在，觉得代码在几个比较大的省份都能跑起来~数据量基本上都在百万基本上。

进而我分了这这几步排除：

1.检查跳转tomcat的jvm参数/或者server.xml是否有存在限制---正常

2.检查是否是服务器的缓冲区设置问题

cat /proc/sys/vm/drop_caches

1

2

cat/proc/sys/vm/drop_caches

发现/proc/sys/vm/drop_caches 状态为3，且crontab 定时也是1/3 状态每十分钟执行一遍，禁止后还是会无法生成 转换后的文件--假死状态

3.既然环境都修改了，还是无法成功那么继续从代码定位。

debug定位到假死代码块：

while ((line = bin.readLine()) != null) {

content.append(line);

content.append(System.getProperty("line.separator"));

if(content.length()>9999999) {

log.info("content.length:{}",content.length());

writer.write(content.toString());

writer.flush();

content = new StringBuffer();

}

}

1

2

3

4

5

6

7

8

9

10

11

while((line=bin.readLine())!=null){

content.append(line);

content.append(System.getProperty("line.separator"));

if(content.length()>9999999){

log.info("content.length:{}",content.length());

writer.write(content.toString());

writer.flush();

content=newStringBuffer();

}

}

通过日志发现，每次当 content.length()满足条件之后并且进行写入后，就卡住了.且日志的content.length 只会提示一遍。

因此判断if(content.length()>9999999)条件存在一定问题。可能就是因为content太大，缓冲区无法一次性写入造成的。于是换一个条件进行写入：

int count = 0;

while ((line = bin.readLine()) != null) {

count++;

content.append(line);

content.append(System.getProperty("line.separator"));

if (count%100000==0) {

log.info("count.length:{}",count);

writer.write(content.toString());

writer.flush();

content = new StringBuffer();

}

}

1

2

3

4

5

6

7

8

9

10

11

12

13

intcount=0;

while((line=bin.readLine())!=null){

count++;

content.append(line);

content.append(System.getProperty("line.separator"));

if(count%100000==0){

log.info("count.length:{}",count);

writer.write(content.toString());

writer.flush();

content=newStringBuffer();

}

}

即每当遍历条数到达一定程度即可写入，防止缓冲区过大被回收(后期遍历条数设置多少可在参数管理设置)

测试部署后，转换生成文件耗时是原来的1/2.那么解决...

因此，使用BufferedWriter写入时候也需要考虑系统性能设置的问题

完整的代码：

/**

* 将一指定编码的文件转换为另一编码的文件

*

* @param oldFullFileName

* @param oldCharsetName

* @param newFullFileName

* @param newCharsetName

*/

public static void convert(String oldFullFileName, String oldCharsetName, String newFullFileName, String newCharsetName) {

log.info("the old file name is : {}, The oldCharsetName is : {}", oldFullFileName, oldCharsetName);

log.info("the new file name is : {}, The newCharsetName is : {}", newFullFileName, newCharsetName);

try {

semaphore.acquire();// 获取信号量,占用

} catch (InterruptedException e1) {

log.error("文件转码并发占用异常",e1);

return;

}

try (BufferedReader bin = new BufferedReader(new InputStreamReader(new FileInputStream(oldFullFileName), oldCharsetName));

BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(newFullFileName), newCharsetName))){

StringBuffer content = new StringBuffer();

String line;

newFullFileName = newFullFileName.replace("\\", "/");

File dir = new File(newFullFileName.substring(0, newFullFileName.lastIndexOf("/")));

if (!dir.exists()) {

dir.mkdirs();

}

int count = 0;

while ((line = bin.readLine()) != null) {

count++;

content.append(line);

content.append(System.getProperty("line.separator"));

if (count%100000==0) {

log.info("count.length:{}",count);

writer.write(content.toString());

writer.flush();

content = new StringBuffer();

}

//if(content.length()>9999999) {

//log.info("content.length:{}",content.length());

//writer.write(content.toString());

//writer.flush();

//content = new StringBuffer();

//Thread.sleep(1000);

//log.info("Thread sleep end");

//}

}

log.info("count:{}",count);

//write(newFullFileName,content.toString(),newCharsetName);

writer.write(content.toString());

writer.flush();

log.info("writer---end{}");

semaphore.release();

} catch (Exception e) {

log.error("文件转码异常",e);

semaphore.release();

}

}

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

/**

* 将一指定编码的文件转换为另一编码的文件

*

* @param oldFullFileName

* @param oldCharsetName

* @param newFullFileName

* @param newCharsetName

*/

publicstaticvoidconvert(StringoldFullFileName,StringoldCharsetName,StringnewFullFileName,StringnewCharsetName){

log.info("the old file name is : {}, The oldCharsetName is : {}",oldFullFileName,oldCharsetName);

log.info("the new file name is : {}, The newCharsetName is : {}",newFullFileName,newCharsetName);

try{

semaphore.acquire();// 获取信号量,占用

}catch(InterruptedExceptione1){

log.error("文件转码并发占用异常",e1);

return;

}

try(BufferedReaderbin=newBufferedReader(newInputStreamReader(newFileInputStream(oldFullFileName),oldCharsetName));

BufferedWriterwriter=newBufferedWriter(newOutputStreamWriter(newFileOutputStream(newFullFileName),newCharsetName))){

StringBuffercontent=newStringBuffer();

Stringline;

newFullFileName=newFullFileName.replace("\\","/");

Filedir=newFile(newFullFileName.substring(0,newFullFileName.lastIndexOf("/")));

if(!dir.exists()){

dir.mkdirs();

}

intcount=0;

while((line=bin.readLine())!=null){

count++;

content.append(line);

content.append(System.getProperty("line.separator"));

if(count%100000==0){

log.info("count.length:{}",count);

writer.write(content.toString());

writer.flush();

content=newStringBuffer();

}

//if(content.length()>9999999) {

//log.info("content.length:{}",content.length());

//writer.write(content.toString());

//writer.flush();

//content = new StringBuffer();

//Thread.sleep(1000);

//log.info("Thread sleep end");

//}

}

log.info("count:{}",count);

//write(newFullFileName,content.toString(),newCharsetName);

writer.write(content.toString());

writer.flush();

log.info("writer---end{}");

semaphore.release();

}catch(Exceptione){

log.error("文件转码异常",e);

semaphore.release();

}

}