主机系统:EXSI 6.7
故障场景:主机断电后,停置N天后再开机,VCenter的 web ui 服务启动失败,访问不了。
排查过程:
1.SSH 登陆VCenter所在的服务器,用service-control检查服务状态,可以发现一堆服务没启动。
root@photon-machine [ ~ ]# cd /bin
root@photon-machine [ ~ ]# service-control --status
2020-07-06T16:59:59.906Z Error: Service name "mbcs" is invalid.
2020-07-06T16:59:59.929Z Error: Service name "sps" is invalid.
2020-07-06T16:59:59.978Z Error: Service name "rbd" is invalid.
2020-07-06T17:00:00.055Z Error: Service name "perfcharts" is invalid.
2020-07-06T17:00:00.078Z Error: Service name "vsan-dps" is invalid.
2020-07-06T17:00:00.102Z Error: Service name "cm" is invalid.
2020-07-06T17:00:00.126Z Error: Service name "vpxd" is invalid.
2020-07-06T17:00:00.149Z Error: Service name "statsmonitor" is invalid.
2020-07-06T17:00:00.173Z Error: Service name "vapi-endpoint" is invalid.
2020-07-06T17:00:00.197Z Error: Service name "vmware-postgres-archiver" is invalid.
2020-07-06T17:00:00.243Z Error: Service name "content-library" is invalid.
2020-07-06T17:00:00.267Z Error: Service name "vcha" is invalid.
2020-07-06T17:00:00.291Z Error: Service name "vpxd-svcs" is invalid.
2020-07-06T17:00:00.314Z Error: Service name "vmonapi" is invalid.
2020-07-06T17:00:00.338Z Error: Service name "cis-license" is invalid.
2020-07-06T17:00:00.362Z Error: Service name "updatemgr" is invalid.
2020-07-06T17:00:00.408Z Error: Service name "sca" is invalid.
2020-07-06T17:00:00.456Z Error: Service name "vsm" is invalid.
2020-07-06T17:00:00.481Z Error: Service name "vmware-vpostgres" is invalid.
2020-07-06T17:00:00.505Z Error: Service name "applmgmt" is invalid.
2020-07-06T17:00:00.529Z Error: Service name "imagebuilder" is invalid.
2020-07-06T17:00:00.554Z Error: Service name "rhttpproxy" is invalid.
2020-07-06T17:00:00.578Z Error: Service name "vsan-health" is invalid.
2020-07-06T17:00:00.602Z Error: Service name "vmcam" is invalid.
2020-07-06T17:00:00.626Z Error: Service name "eam" is invalid.
2020-07-06T17:00:00.649Z Error: Service name "vsphere-client" is invalid.
2020-07-06T17:00:00.729Z Error: Service name "vsphere-ui" is invalid.
2020-07-06T17:00:00.779Z Error: Service name "analytics" is invalid.
2020-07-06T17:00:00.804Z Error: Service name "pschealth" is invalid.
2020-07-06T17:00:00.829Z Error: Service name "netdumper" is invalid.
Running:
lwsmd vmafdd vmcad vmdird vmdnsd vmware-pod vmware-sts-idmd vmware-stsd vmware-vmon
解决方法:
参见了以下文章:https://communities.vmware.com/message/2816068?tstart=0
想到自己前两天也改过去主机的时间,进虚拟机看了下时间,果然和主机不一致,进VC所在的虚拟机的BIOS,改完时间后,再删除备份并删除/storage/vmware-vmon/.svcStats下的所有文件,然后重启所有服务,问题解决。
root@photon-machine [ ~ ]# cd /storage/vmware-vmon/
root@photon-machine [ /storage/vmware-vmon ]# tar cvf svcstats.back .svcStats
root@photon-machine [ /storage/vmware-vmon ]# cd .svcStats
root@photon-machine [ /storage/vmware-vmon/.svcStats ]# rm -rf *
root@photon-machine [ /bin ]# service-control --stop --all
root@photon-machine [ /bin ]# service-control --start --all
root@photon-machine [ /bin ]# service-control --status
Stopped:
vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-vcha vsan-dps
Running:
applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-analytics vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-pod vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-updatemgr vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui