问题描述
某个工程在启动时报如下错误
Worker is dead on 10.237.65.30:6804,20181213154018, at 2018-12-13 15:40
org.springframework.beans.factory.BeanDefinitionStoreException: IOException parsing XML document from class path resource [applicationContext-act.xml]; nested exception is java.io.IOException: Stream closed , at 2018-12-13 15:40
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:410)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:181)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:217)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:188)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:252)
at org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:127)
at org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:93)
at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:129)
at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:613)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:514)
at org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:139)
at org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:93)
at com.xxx.SpringContext.init(SpringContext.java:14)(业务类)
或者是这种
org.springframework.beans.factory.BeanDefinitionStoreException: Unexpected exception parsing XML document from class path resource [applicationContext-act.xml]; nested exception is java.lang.IllegalStateException: zip file closed , at 2018-12-13 16:30
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:414)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:181)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:217)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:188)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:252)
at org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:127)
at org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:93)
at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:129)
at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:613)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:514)
at org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:139)
at org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:93)
问题以一定概率出现(最头痛的情况),但是能够正常启动。报的文件也不固定。因为xml里又import了多个xml。
其实也许这是一个无伤大雅的问题,但是我们不能这样的无视。谁知道哪天故障了系统崩溃这就成为你背锅的理由呢。作为程序员,我们确实不能交付一个用户有感知的BUG的系统并对这个BUG一无所知。
业务所提交的代码是这个样子的。
public class SpringContext {
public static ApplicationContext context;
public static synchronized void init() {
if (context == null) {
context = new ClassPathXmlApplicationContext(new String[]{
"applicationContext-act.xml"
});
}
}
在每一个bolt的启动前初始化context。利用JAVA的static模式保证单例模式。并加了锁。看样子真是一点问题都没有。
问题排查
上述代码严格来说并非没有问题。因为指令重排的原因,context可能在初始化完成前就已经发布。可能会有线程读到别的线程没有初始化完成的context。不过这无法解释为什么会出现流错误。
Storm会在nimbus端初始化bolt对象,然后序列化到各个节点。在执行bolt的pepare。同时把jar报作为一个文件传递给不同的节点。不同的worker上,其所谓的流,都是指自己的文件系统的该XML文件(Spring配置)。
我们也并非不熟悉StreamClosed这个异常。多个线程索引同一个inputstream时,当某一个thread在执行完之后,把这个inputstream关闭了;而此时正在从这个input stream流中读取信息的线程就会抛出 java.io.IOException: Stream closed 异常。
public
class InflaterInputStream extends FilterInputStream {
private boolean closed = false;
/**
* Check to make sure that this stream has not been closed
*/
private void ensureOpen() throws IOException {
if (closed) {
throw new IOException("Stream closed");
}
}
public void close() throws IOException {
if (!closed) {
if (usesDefaultInflater)
inf.end();
in.close();
closed = true;
}
}
很遗憾,我们没办法本地复现这个错误。我们也缺乏重写JDK加入日志的勇气。依然没有头绪。