第一、环境
1、软件版本
CloudStack 3.0.5+XenServer 6.0.2
2、硬件设备
管理节点:DELL R420 10.30.18.2
计算节点:DELL R720 10.30.18.9
存储节点:DELL IBM520 10.30.18.4
3、网络环境
网段:10.30.18.0/24
网关:10.30.18.1
DNS:202.194.104.35
4、CloudStack规划
物理服务器网段: 10.30.18.2--10.30.18.15
管理网段: 10.30.18.16--10.30.18.24
来宾网段: 10.30.18.25--10.30.18.40
DNS: 8.8.8.8/8.8.4.4
存储类型: NFS
主存储: /export/primary1
辅助存储: /export/secondary
http服务器: 10.30.18.2
www目录: /var/www/html
5、故障原因及现象
1、硬件故障:计算节点R720内存问题导致非正常死机,并且不能正常启动
2、平台故障:排出了计算节点R720的硬件问题后,计算节点的状态是“error”
第二、处理流程
步骤一、首先备份cloud数据库
1.1登录管理节点10.30.18.2
[root@master ~]# hostname
master
[root@master ~]#
1.2查看当前的数据库列表
[root@master ~]# mysql -u root -p
Enter password:
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| cloud |
| cloud_usage |
| cloudbridge |
| glance |
| mysql |
| nova |
+--------------------+
7 rows in set (0.11 sec)
1.3备份cloud数据库
[root@master ~]# mysqldump -u root -p -A --event > ./db_all.20140426.sql
Enter password:
备注:-A 表示备份MySql中所有的数据库,恢复是需要先在MySql中新建数据库名。
步骤二、其次备份存储
2.1ssh登录存储:
[root@store ~]# hostname
store
[root@store ~]#
2.2备份主存储
[root@store ~]# cp -rpv /export/primary1/ /export/primary2
2.3备份服务存储
[root@store ~]# cp -rpv /export/secondary1/ /export/secondary2
步骤三、重建CloudStack环境
3.1先停止并删除在host下运行的实例VM
3.2禁用区域,并删除系统虚拟机(v,s,r)
3.4启用主存储的维护模式然后删除主存储 、二级存储
3.5启用主机host的维护模式然后删除host
3.6删除群集
3.7删除pod
3.8网络|来宾网络|删除
3.9区域|物理网络|删除
3.10删除区域
3.11清空NFS共享存储目录
#rm -rf /export/primary1/*
备注:原有的primary1目录数据已经备份到了primary2目录中
#rm -rf /export/secondary1/*
备注:原有的secondary1目录数据已经备份到了secondary2目录中
3.12停止CloudStack服务
3.13重新安装计算节点xenserver
3.14、删除CloudStack数据库
#drop database cloud; drop database cloud_usage;
3.15重新部署CS数据库
#cloud-setup-databases cloud:cloud@localhost --deploy-as=root:password
#cloud-setup-management
3.16安装系统模版(不同的hypervisor请安装不同的系统模版)
[root@management CloudStack-3.0.5-1-rhel6.2]# mount -t nfs 192.168.1.250:/storage/secondary /opt
[root@management CloudStack-3.0.5-1-rhel6.2]# /usr/lib64/cloud/agent/scripts/storage/secondary/cloud-install-sys-tmplt -m /opt -f /root/acton-systemvm-02062012.vhd.bz2 -h xenserver -F
[root@management ~]# umount /opt
3.17、启动cloud服务
#service cloud-management start
3.18、设置全局变量
Secstorage.allowed.internal.sites :设置辅助存储的允许范围
Expunge.delay :设置删除的延时
Expunge.interval :设置删除的时间间隔
Cpu.overprovisioning.factor :设置CPU超分
Mem.overprovisioning.factor :设置内存超分(3.0.5版本无效)
重启cloud-management服务:
#service cloud-management restart
步骤四、创建基本资源域环境
备注:此处不是重点,故此略过
步骤五、原有的数据结构
5.1进入msyql数据库
[root@master ~]# mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 128
Server version: 5.1.61-log Source distribution
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
5.2切换到cloud数据库
mysql> use cloud;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
5.3查看cloud数据库的表
mysql> show tables ;
+--------------------------------------+
| Tables_in_cloud |
+--------------------------------------+
| account |
| account_details |
| account_network_ref |
| account_vlan_map |
| alert |
| async_job |
| autoscale_policies |
| autoscale_policy_condition_map |
| autoscale_vmgroup_policy_map |
| autoscale_vmgroups |
| autoscale_vmprofiles |
| cluster |
| cluster_details |
| cluster_vsm_map |
| cmd_exec_log |
| conditions |
| configuration |
| console_proxy |
| counter |
| data_center |
| data_center_details |
| dc_storage_network_ip_range |
| disk_offering |
| domain |
| domain_network_ref |
| domain_router |
| elastic_lb_vm_map |
| event |
| external_firewall_devices |
| external_load_balancer_devices |
| firewall_rules |
| firewall_rules_cidrs |
| guest_os |
| guest_os_category |
| guest_os_hypervisor |
| host |
| host_details |
| host_pod_ref |
| host_tags |
| hypervisor_capabilities |
| inline_load_balancer_nic_map |
| instance_group |
| instance_group_vm_map |
| keystore |
| launch_permission |
| load_balancer_stickiness_policies |
| load_balancer_vm_map |
| load_balancing_rules |
| mshost |
| mshost_peer |
| netapp_lun |
| netapp_pool |
| netapp_volume |
| network_external_firewall_device_map |
| network_external_lb_device_map |
| network_offerings |
| network_rule_config |
| networks |
| nics |
| ntwk_offering_service_map |
| ntwk_service_map |
| op_dc_ip_address_alloc |
| op_dc_link_local_ip_address_alloc |
| op_dc_storage_network_ip_address |
| op_dc_vnet_alloc |
| op_ha_work |
| op_host |
| op_host_capacity |
| op_host_transfer |
| op_host_upgrade |
| op_it_work |
| op_lock |
| op_networks |
| op_nwgrp_work |
| op_pod_vlan_alloc |
| op_user_stats_log |
| op_vm_ruleset_log |
| ovs_tunnel_interface |
| ovs_tunnel_network |
| physical_network |
| physical_network_isolation_methods |
| physical_network_service_providers |
| physical_network_tags |
| physical_network_traffic_types |
| pod_vlan_map |
| port_forwarding_rules |
| port_profile |
| private_ip_address |
| project_account |
| project_invitations |
| projects |
| remote_access_vpn |
| resource_count |
| resource_limit |
| resource_tags |
| router_network_ref |
| s2s_customer_gateway |
| s2s_vpn_connection |
| s2s_vpn_gateway |
| secondary_storage_vm |
| security_group |
| security_group_rule |
| security_group_vm_map |
| sequence |
| service_offering |
| snapshot_policy |
| snapshot_schedule |
| snapshots |
| ssh_keypairs |
| stack_maid |
| static_routes |
| storage_pool |
| storage_pool_details |
| storage_pool_host_ref |
| storage_pool_work |
| swift |
| sync_queue |
| sync_queue_item |
| template_host_ref |
| template_spool_ref |
| template_swift_ref |
| template_zone_ref |
| upload |
| usage_event |
| user |
| user_ip_address |
| user_statistics |
| user_vm |
| user_vm_details |
| version |
| virtual_router_providers |
| virtual_supervisor_module |
| vlan |
| vm_instance |
| vm_template |
| vm_template_details |
| volume_host_ref |
| volumes |
| vpc |
| vpc_gateways |
| vpc_offering_service_map |
| vpc_offerings |
| vpn_users |
+--------------------------------------+