【WebLogic】记录一次WebLogic进程无法启动的故障排查过程

故障现象：

WebLogic进程在启动的时候出现报错，导致进程异常SHUTDOWN。

报错信息：

####<Jan 4, 2022 2:51:13 PM CST> <Critical> <WebLogicServer> <aMDSJK02> <AdminServer> <main> <<WLS Kernel>> <> <> <1641279073888> <BEA-000386> <Server subsystem failed. Reason: java.lang.AssertionError: java.lang.reflect.InvocationTargetException
java.lang.AssertionError: java.lang.reflect.InvocationTargetException
	at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy._invokeServiceMethod(DescriptorManager.java:175)
	at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy.decrypt(DescriptorManager.java:192)
	at weblogic.descriptor.DescriptorManager$SecurityServiceImpl.decrypt(DescriptorManager.java:114)
	at weblogic.descriptor.internal.AbstractDescriptorBean._decrypt(AbstractDescriptorBean.java:1191)
	at weblogic.management.configuration.SecurityConfigurationMBeanImpl.getCredential(SecurityConfigurationMBeanImpl.java:737)
	at weblogic.security.internal.ServerPrincipalValidatorImpl.getSecret(ServerPrincipalValidatorImpl.java:88)
	at weblogic.security.internal.ServerPrincipalValidatorImpl.sign(ServerPrincipalValidatorImpl.java:67)
	at weblogic.security.service.PrivilegedActions$SignPrincipalAction.run(PrivilegedActions.java:62)
	at java.security.AccessController.doPrivileged(Native Method)
	at weblogic.security.service.SecurityServiceManager.createServerID(SecurityServiceManager.java:1111)
	at weblogic.security.service.SecurityServiceManager.getServerID(SecurityServiceManager.java:1124)
	at weblogic.security.service.SecurityServiceManager.sendASToWire(SecurityServiceManager.java:600)
	at weblogic.server.channels.ChannelService.resetQOS(ChannelService.java:298)
	at weblogic.server.channels.ChannelService.start(ChannelService.java:256)
	at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
	at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy._invokeServiceMethod(DescriptorManager.java:173)
	at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy.decrypt(DescriptorManager.java:192)
	at weblogic.descriptor.DescriptorManager$SecurityServiceImpl.decrypt(DescriptorManager.java:114)
	at weblogic.descriptor.internal.AbstractDescriptorBean._decrypt(AbstractDescriptorBean.java:1191)
	at weblogic.management.configuration.SecurityConfigurationMBeanImpl.getCredential(SecurityConfigurationMBeanImpl.java:737)
	at weblogic.security.internal.ServerPrincipalValidatorImpl.getSecret(ServerPrincipalValidatorImpl.java:88)
	at weblogic.security.internal.ServerPrincipalValidatorImpl.sign(ServerPrincipalValidatorImpl.java:67)
	at weblogic.security.service.PrivilegedActions$SignPrincipalAction.run(PrivilegedActions.java:62)
	at java.security.AccessController.doPrivileged(Native Method)
	at weblogic.security.service.SecurityServiceManager.createServerID(SecurityServiceManager.java:1111)
	at weblogic.security.service.SecurityServiceManager.getServerID(SecurityServiceManager.java:1124)
	at weblogic.security.service.SecurityServiceManager.sendASToWire(SecurityServiceManager.java:600)
	at weblogic.server.channels.ChannelService.resetQOS(ChannelService.java:298)
	at weblogic.server.channels.ChannelService.start(ChannelService.java:256)
	at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
	at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: weblogic.security.internal.encryption.EncryptionServiceException
	at weblogic.security.internal.encryption.JSafeEncryptionServiceImpl.decryptBytes(JSafeEncryptionServiceImpl.java:139)
	at weblogic.security.internal.encryption.JSafeEncryptionServiceImpl.decryptString(JSafeEncryptionServiceImpl.java:187)
	at weblogic.security.internal.encryption.ClearOrEncryptedService.decrypt(ClearOrEncryptedService.java:96)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy._invokeServiceMethod(DescriptorManager.java:173)
	at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy.decrypt(DescriptorManager.java:192)
	at weblogic.descriptor.DescriptorManager$SecurityServiceImpl.decrypt(DescriptorManager.java:114)
	at weblogic.descriptor.internal.AbstractDescriptorBean._decrypt(AbstractDescriptorBean.java:1191)
	at weblogic.management.configuration.SecurityConfigurationMBeanImpl.getCredential(SecurityConfigurationMBeanImpl.java:737)
	at weblogic.security.internal.ServerPrincipalValidatorImpl.getSecret(ServerPrincipalValidatorImpl.java:88)
	at weblogic.security.internal.ServerPrincipalValidatorImpl.sign(ServerPrincipalValidatorImpl.java:67)
	at weblogic.security.service.PrivilegedActions$SignPrincipalAction.run(PrivilegedActions.java:62)
	at java.security.AccessController.doPrivileged(Native Method)
	at weblogic.security.service.SecurityServiceManager.createServerID(SecurityServiceManager.java:1111)
	at weblogic.security.service.SecurityServiceManager.getServerID(SecurityServiceManager.java:1124)
	at weblogic.security.service.SecurityServiceManager.sendASToWire(SecurityServiceManager.java:600)
	at weblogic.server.channels.ChannelService.resetQOS(ChannelService.java:298)
	at weblogic.server.channels.ChannelService.start(ChannelService.java:256)
	at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
	at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
> 
####<Jan 4, 2022 2:51:13 PM CST> <Notice> <WebLogicServer> <aMDSJK02> <AdminServer> <main> <<WLS Kernel>> <> <> <1641279073976> <BEA-000365> <Server state changed to FAILED> 
####<Jan 4, 2022 2:51:13 PM CST> <Error> <WebLogicServer> <aMDSJK02> <AdminServer> <main> <<WLS Kernel>> <> <> <1641279073976> <BEA-000383> <A critical service failed. The server will shut itself down> 
####<Jan 4, 2022 2:51:13 PM CST> <Notice> <WebLogicServer> <aMDSJK02> <AdminServer> <main> <<WLS Kernel>> <> <> <1641279073978> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN> 
####<Jan 4, 2022 2:51:13 PM CST> <Info> <WebLogicServer> <aMDSJK02> <AdminServer> <main> <<WLS Kernel>> <> <> <1641279073978> <BEA-000236> <Stopping execute threads.> 

故障排查：

根据报错信息分析和故障现象模拟，Oracle原厂工程师判断故障的原因为 SerializedSystemIni.dat （位于$DOMAIN_HOME/security目录下）文件损坏，该文件中包含了具有解密config.xml信息凭证的hash值。另外，Oracle原厂工程师根据官方的文档，试图通过技术手段恢复该文件，但是恢复后的文件并不能使得进程成功启动。

相关文档：

Diagnosing BEA-000386 Critical Message ( Doc ID 1493826.1 )
How to Recover SerializedSystemIni.dat File? ( Doc ID 1602857.1 )

处理方法：

方法-1：重新创建应用域；

方法-2：使用之前应用域的完整备份，进行应用域的恢复。