java.io.IOException: Filesystem closed

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/bitcarmanlee/article/details/68488616

1.问题描述

往集群提交任务的时候,需要在hdfs上面读取一个资源文件。在读取该资源文件的时候,代码爆出如下异常:

Error: java.io.IOException: Filesystem closed
    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:823)
    at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:846)
    at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:907)
    at java.io.DataInputStream.readFully(DataInputStream.java:195)
    at java.io.DataInputStream.readFully(DataInputStream.java:169)
    at org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:756)
    at org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:494)
    at org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:127)
    at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:208)
    at org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:201)
    at org.apache.hadoop.mapreduce.lib.input.DelegatingRecordReader.nextKeyValue(DelegatingRecordReader.java:89)
...

2.问题的原因

一般读取hdfs上文件的api是这样写的:

public void initPackageNameTypeMap {
    Configuration conf = new Configuration();
    FileSystem fs = FileSystem.get(conf);
    Path file = new Path("filePath");
    ...
}

当任务提交到集群上面以后,多个datanode在getFileSystem过程中,由于Configuration一样,会得到同一个FileSystem。如果有一个datanode在使用完关闭连接,其它的datanode在访问就会出现上述异常。

3.解决方案

解决的方法其实很简单,在代码中加入如下配置就好:

conf.setBoolean("fs.hdfs.impl.disable.cache", true);

FileSytem类内部有一个static CACHE,用来保存每种文件系统的实例集合,FileSystem类中可以通过”fs.%s.impl.disable.cache”来指定是否缓存FileSystem实例(其中%s替换为相应的scheme,比如hdfs、local、s3、s3n等),即一旦创建了相应的FileSystem实例,这个实例将会保存在缓存中,此后每次get都会获取同一个实例。所以设为true以后,就能解决上面的异常。

参考链接:
1.http://stackoverflow.com/questions/23779186/ioexception-filesystem-closed-exception-when-running-oozie-workflow
2.http://stackoverflow.com/questions/20057881/hadoop-filesystem-closed-exception-when-doing-bufferedreader-close
3.http://shift-alt-ctrl.iteye.com/blog/2108760

阅读更多
换一批

java.io.IOException: Stream closed

03-04

HTTP Status 500 - Stream closedrnrnrntype Exception reportrnrnmessage Stream closedrnrndescription The server encountered an internal error that prevented it from fulfilling this request.rnrnexception rnjava.io.IOException: Stream closedrn org.apache.jasper.runtime.JspWriterImpl.ensureOpen(JspWriterImpl.java:210)rn org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:115)rn org.apache.jasper.runtime.JspWriterImpl.flush(JspWriterImpl.java:177)rn org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:942)rn org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:656)rn org.apache.struts2.dispatcher.ServletDispatcherResult.doExecute(ServletDispatcherResult.java:123)rn org.apache.struts2.dispatcher.StrutsResultSupport.execute(StrutsResultSupport.java:186)rn com.opensymphony.xwork2.DefaultActionInvocation.executeResult(DefaultActionInvocation.java:362)rn com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:266)rn cn.labsoft.labos.framework.common.interceptor.IsLoginInterceptorNew.intercept(Unknown Source)rn cn.labsoft.labos.framework.common.interceptor.IsLoginInterceptorNew$$FastClassByCGLIB$$d5f81fd5.invoke()rn net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)rn org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)rn org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)rn org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:50)rn org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)rn org.springframework.aop.framework.adapter.ThrowsAdviceInterceptor.invoke(ThrowsAdviceInterceptor.java:126)rn org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)rn org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)rn org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)rn org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:635)rn cn.labsoft.labos.framework.common.interceptor.IsLoginInterceptorNew$$EnhancerByCGLIB$$bb81b672.intercept()rn com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237)rn org.apache.struts2.impl.StrutsActionProxy.execute(StrutsActionProxy.java:52)rn org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:488)rn org.apache.struts2.dispatcher.ng.ExecuteOperations.executeAction(ExecuteOperations.java:77)rn org.apache.struts2.dispatcher.ng.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepareAndExecuteFilter.java:91)rn cn.labsoft.labos.framework.common.filter.HttpSessionContextDataFilter.doFilter(Unknown Source)rn org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:198)rn org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)rnrnrnrnnote The full stack trace of the root cause is available in the Apache Tomcat/7.0.59 logs.rn

没有更多推荐了,返回首页