Openstack(两控制节点+四计算节点)系列四篇文章中openstack采用E版本,主要测试计算节点和控制节点宕机后的恢复过程,在计算节点采用iscsi模拟共享存储,在控制节点采用heartbeat+pacemaker监控控制节点的主要服务,所以控制节点是主备双机,需要设置vip,openstack的网络模式采用Flat模式。
主控制节点:10.1.6.186
备控制节点:10.1.6.188
heartbeat vip:10.1.6.100
计算节点:10.1.6.142、10.1.6.152、10.1.6.162、10.1.6.172
共享存储:10.1.6.39
先操作主control-node
1 安装网桥相关软件
apt-get install -y bridge-utils
2 修改网卡配置:/etc/network/interfaces
auto lo
iface lo inet loopback
auto br100
iface br100 inet static
address 10.1.6.186
netmask 255.255.255.0
gateway 10.1.6.254
dns-nameservers 10.1.1.2
bridge_ports em1
bridge_hello 2
bridge_maxage 12
bridge_fd 0
bridge_stp off
3 重启网卡,使网桥配置生效
/etc/init.d/networking restart
4 作为控制节点需要安装NTP服务
apt-get install -y ntp
5 修改/etc/ntp.conf,在server ntp.ubuntu.com下添加如下内容:
server 127.127.1.0
fudge 127.127.1.0 stratum 10
6 重启ntp服务,使配置生效
/etc/init.d/ntp restart
7 安装消息队列、缓存、kvm等软件
apt-get install -y rabbitmq-server memcached python-memcache kvm libvirt-bin curl
8 安装iscsi、lvm2软件,在控制节点上volume会使用到(在本系列中属于选做)
apt-get install -y tgt lvm2
9 挑选一个分区或者盘做LVM卷(/dev/sda5),卷名为nova-volumes(在本系列中属于选做)
pvcreate /dev/sda3
vgcreate nova-volumes /dev/sda3
10 安装mysql数据库,数据库管理员密码设置为:my_password
apt-get install -y mysql-server python-mysqldb
11 修改mysql配置文件:/etc/mysql/my.cnf
#监听0.0.0.0
bind-address = 0.0.0.0
#启动binlog,为以后主从做好准备
server-id = 1
log_bin = /var/log/mysql/mysql-bin.log
12 重启mysql使配置生效
/etc/init.d/mysql restart
13 创建openstack管理所需要的相关库:nova、keystone、glance
CREATE DATABASE nova;
GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'%' IDENTIFIED BY 'my_password';
CREATE DATABASE glance;
GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'%' IDENTIFIED BY 'my_password';
CREATE DATABASE keystone;
GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%'IDENTIFIED BY 'my_password';
GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'vm.my.com' IDENTIFIED BY 'my_password';
GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'vm.my.com' IDENTIFIED BY 'my_password';
GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'vm.my.com'IDENTIFIED BY 'my_password';
FLUSH PRIVILEGES;
14 安装keystone相关组件
apt-get install -y keystone python-keystone python-keystoneclient
15 修改/etc/keystone/keystone.conf配置文件
[DEFAULT]
bind_host = 0.0.0.0
public_port = 5000
admin_port = 35357
admin_token = my_cloud
[sql]
connection = mysql://keystone:my_password@vm.my.com/keystone
期中配置vm.my.com通过DNS指向主控制节点上面heartbeat的vip地址上面
16 重启keystone服务
/etc/init.d/keystone restart
17 初始化keystone库
keystone-manage db_sync
18 为了安装设置keystone组件方便快捷,提前设置好环境变量
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=password
export SERVICE_PASSWORD=password
export FIXED_RANGE=10.1.6.0/24
export OS_AUTH_URL="http://vm.my.com:5000/v2.0/"
export SERVICE_ENDPOINT="http://vm.my.com:35357/v2.0"
export SERVICE_TOKEN=my_cloud
export MASTER="vm.my.com"
export OS_NO_CACHE=1
为了避免每次ssh连接时需要设置环境变量,可以把以上设置加入.bashrc文件中
19 设置users and tenants and services,用脚本设置keystone.sh
#!/bin/bash -x
#
# Initial data for Keystone using python-keystoneclient
#
# Tenant User Roles
# ------------------------------------------------------------------
# admin admin admin
# service glance admin
# service nova admin, [ResellerAdmin (swift only)]
# service quantum admin # if enabled
# service swift admin # if enabled
# demo admin admin
# demo demo Member, anotherrole
# invisible_to_admin demo Member
#
# Variables set before calling this script:
# SERVICE_TOKEN - aka admin_token in keystone.conf
# SERVICE_ENDPOINT - local Keystone admin endpoint
# SERVICE_TENANT_NAME - name of tenant containing service accounts
# ENABLED_SERVICES - stack.sh's list of services to start
# DEVSTACK_DIR - Top-level DevStack directory
ADMIN_PASSWORD=${ADMIN_PASSWORD:-$OS_PASSWORD}
SERVICE_TENANT_NAME=${SERVICE_TENANT_NAME:-service}
ENABLED_SERVICES="swift"
function get_id () {
echo `$@ | awk '/ id / { print $4 }'`
}
# Tenants
ADMIN_TENANT=$(get_id keystone tenant-create --name=admin)
SERVICE_TENANT=$(get_id keystone tenant-create --name=$SERVICE_TENANT_NAME)
# Users
ADMIN_USER=$(get_id keystone user-create --name=admin \
--pass="$ADMIN_PASSWORD" \
--email=admin@my.com)
# Roles
ADMIN_ROLE=$(get_id keystone role-create --name=admin)
MEMBER_ROLE=$(get_id keystone role-create --name=Member)
KEYSTONEADMIN_ROLE=$(get_id keystone role-create --name=KeystoneAdmin)
KEYSTONESERVICE_ROLE=$(get_id keystone role-create --name=KeystoneServiceAdmin)
# Add Roles to Users in Tenants
keystone user-role-add --user-id $ADMIN_USER --role-id $ADMIN_ROLE --tenant_id $ADMIN_TENANT
#keystone user-role-add --user $DEMO_USER --role $MEMBER_ROLE --tenant_id $DEMO_TENANT
# Configure service users/roles
NOVA_USER=$(get_id keystone user-create --name=nova \
--pass="$SERVICE_PASSWORD" \
--tenant_id $SERVICE_TENANT \
--email=nova@my.com)
keystone user-role-add --tenant_id $SERVICE_TENANT \
--user-id $NOVA_USER \
--role-id $ADMIN_ROLE
GLANCE_USER=$(get_id keystone user-create --name=glance \
--pass="$SERVICE_PASSWORD" \
--tenant_id $SERVICE_TENANT \
--email=glance@my.com)
keystone user-role-add --tenant_id $SERVICE_TENANT \
--user-id $GLANCE_USER \
--role-id $ADMIN_ROLE
if [[ "$ENABLED_SERVICES" =~ "swift" ]]; then
SWIFT_USER=$(get_id keystone user-create --name=swift \
--pass="$SERVICE_PASSWORD" \
--tenant_id $SERVICE_TENANT \
--email=swift@my.com)
keystone user-role-add --tenant_id $SERVICE_TENANT \
--user-id $SWIFT_USER \
--role-id $ADMIN_ROLE
# Nova needs ResellerAdmin role to download images when accessing
# swift through the s3 api. The admin role in swift allows a user
# to act as an admin for their tenant, but ResellerAdmin is needed
# for a user to act as any tenant. The name of this role is also
# configurable in swift-proxy.conf
RESELLER_ROLE=$(get_id keystone role-create --name=ResellerAdmin)
keystone user-role-add --tenant_id $SERVICE_TENANT \
--user-id $NOVA_USER \
--role-id $RESELLER_ROLE
fi
if [[ "$ENABLED_SERVICES" =~ "quantum" ]]; then
QUANTUM_USER=$(get_id keystone user-create --name=quantum \
--pass="$SERVICE_PASSWORD" \
--tenant_id $SERVICE_TENANT \
--email=quantum@my.com)
keystone user-role-add --tenant_id $SERVICE_TENANT \
--user-id $QUANTUM_USER \
--role-id $ADMIN_ROLE
fi
20 设置endpoint服务,使用endpoint.sh脚本设置
#!/bin/bash -x
# Author: Martin Gerhard Loschwitz
# (c) 2012 hastexo Professional Services GmbH
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# On Debian-based systems the full text of the Apache version 2.0
# license can be found in `/usr/share/common-licenses/Apache-2.0'.
# MySQL definitions
MYSQL_USER=keystone
MYSQL_DATABASE=keystone
MYSQL_PASSWORD=my_password
MYSQL_HOST=vm.my.com
MASTER=vm.my.com
# Keystone definitions
KEYSTONE_REGION=RegionOne
SERVICE_ENDPOINT="http://vm.my.com:35357/v2.0"
# other definitions
while getopts "u:D:p:m:K:R:E:S:T:vh" opt; do
case $opt in
u)
MYSQL_USER=$OPTARG
;;
D)
MYSQL_DATABASE=$OPTARG
;;
p)
MYSQL_PASSWORD=$OPTARG
;;
m)
MYSQL_HOST=$OPTARG
;;
K)
MASTER=$OPTARG
;;
R)
KEYSTONE_REGION=$OPTARG
;;
E)
export SERVICE_ENDPOINT=$OPTARG
;;
S)
SWIFT_MASTER=$OPTARG
;;
T)
export SERVICE_TOKEN=$OPTARG
;;
v)
set -x
;;
h)
cat <<EOF
Usage: $0 [-m mysql_hostname] [-u mysql_username] [-D mysql_database] [-p mysql_password]
[-K keystone_master ] [ -R keystone_region ] [ -E keystone_endpoint_url ]
[ -S swift_master ] [ -T keystone_token ]
Add -v for verbose mode, -h to display this message.
EOF
exit 0
;;
\?)
echo "Unknown option -$OPTARG" >&2
exit 1
;;
:)
echo "Option -$OPTARG requires an argument" >&2
exit 1
;;
esac
done
if [ -z "$KEYSTONE_REGION" ]; then
echo "Keystone region not set. Please set with -R option or set KEYSTONE_REGION variable." >&2
missing_args="true"
fi
if [ -z "$SERVICE_TOKEN" ]; then
echo "Keystone service token not set. Please set with -T option or set SERVICE_TOKEN variable." >&2
missing_args="true"
fi
if [ -z "$SERVICE_ENDPOINT" ]; then
echo "Keystone service endpoint not set. Please set with -E option or set SERVICE_ENDPOINT variable." >&2
missing_args="true"
fi
if [ -z "$MYSQL_PASSWORD" ]; then
echo "MySQL password not set. Please set with -p option or set MYSQL_PASSWORD variable." >&2
missing_args="true"
fi
if [ -n "$missing_args" ]; then
exit 1
fi
keystone service-create --name nova --type compute --description 'OpenStack Compute Service'
keystone service-create --name volume --type volume --description 'OpenStack Volume Service'
keystone service-create --name glance --type image --description 'OpenStack Image Service'
keystone service-create --name swift --type object-store --description 'OpenStack Storage Service'
keystone service-create --name keystone --type identity --description 'OpenStack Identity'
keystone service-create --name ec2 --type ec2 --description 'OpenStack EC2 service'
create_endpoint () {
case $1 in
compute)
keystone endpoint-create --region $KEYSTONE_REGION --service_id $2 --publicurl 'http://'"$MASTER"':8774/v2/%(tenant_id)s' --adminurl 'http://'"$MASTER"':8774/v2/%(tenant_id)s' --internalurl 'http://'"$MASTER"':8774/v2/%(tenant_id)s'
;;
volume)
keystone endpoint-create --region $KEYSTONE_REGION --service_id $2 --publicurl 'http://'"$MASTER"':8776/v1/%(tenant_id)s' --adminurl 'http://'"$MASTER"':8776/v1/%(tenant_id)s' --internalurl 'http://'"$MASTER"':8776/v1/%(tenant_id)s'
;;
image)
keystone endpoint-create --region $KEYSTONE_REGION --service_id $2 --publicurl 'http://'"$MASTER"':9292/v1' --adminurl 'http://'"$MASTER"':9292/v1' --internalurl 'http://'"$MASTER"':9292/v1'
;;
object-store)
if [ $SWIFT_MASTER ]; then
keystone endpoint-create --region $KEYSTONE_REGION --service_id $2 --publicurl 'http://'"$SWIFT_MASTER"':8080/v1/AUTH_%(tenant_id)s' --adminurl 'http://'"$SWIFT_MASTER"':8080/v1' --internalurl 'http://'"$SWIFT_MASTER"':8080/v1/AUTH_%(tenant_id)s'
else
keystone endpoint-create --region $KEYSTONE_REGION --service_id $2 --publicurl 'http://'"$MASTER"':8080/v1/AUTH_%(tenant_id)s' --adminurl 'http://'"$MASTER"':8080/v1' --internalurl 'http://'"$MASTER"':8080/v1/AUTH_%(tenant_id)s'
fi
;;
identity)
keystone endpoint-create --region $KEYSTONE_REGION --service_id $2 --publicurl 'http://'"$MASTER"':5000/v2.0' --adminurl 'http://'"$MASTER"':35357/v2.0' --internalurl 'http://'"$MASTER"':5000/v2.0'
;;
ec2)
keystone endpoint-create --region $KEYSTONE_REGION --service_id $2 --publicurl 'http://'"$MASTER"':8773/services/Cloud' --adminurl 'http://'"$MASTER"':8773/services/Admin' --internalurl 'http://'"$MASTER"':8773/services/Cloud'
;;
esac
}
for i in compute volume image object-store identity ec2; do
id=`mysql -h "$MYSQL_HOST" -u "$MYSQL_USER" -p"$MYSQL_PASSWORD" "$MYSQL_DATABASE" -ss -e "SELECT id FROM service WHERE type='"$i"';"` || exit 1
create_endpoint $i $id
done
21 验证设置是否正确
keystone tenant-list
keystone user-list
keystone role-list
22 安装glance服务
apt-get install -y glance glance-api glance-client glance-common glance-registry python-glance
23 修改配置文件/etc/glance/glance-api.conf 和/etc/glance/glance-registry.conf
#admin_tenant_name = %SERVICE_TENANT_NAME%
#admin_user = %SERVICE_USER%
#admin_password = %SERVICE_PASSWORD%
admin_tenant_name = service
admin_user = glance
admin_password = password
24 修改/etc/glance/glance-registry.conf使用mysql连接
sql_connection = mysql://glance:my_password@vm.my.com/glance
25 修改/etc/glance/glance-registry.conf和/etc/glance/glance-api.conf
[paste_deploy]
flavor = keystone
26 重启glance相关服务
/etc/init.d/glance-api restart
/etc/init.d/glance-registry restart
27 初始化glance库
glance-manage version_control 0
glance-manage db_sync
28 再次重启glance相关服务
/etc/init.d/glance-api restart
/etc/init.d/glance-registry restart
29 测试glance,F版中会提示一个警告,E版中则无输出
glance index
WARNING! This tool is deprecated in favor of python-glanceclient (see http://github.com/openstack/python-glanceclient).
30 上传自制的Debian6镜像
glance add name="debian6 initrd" disk_format=qcow2 container_format=ovf is_public=true < initrd.img-2.6.32-5-amd64
#上面命令返回ID:9fd89cc3-c479-4544-aeae-6201bfbd504b
glance add name="debian6 vmlinuz" disk_format=qcow2 container_format=ovf is_public=true < vmlinuz-2.6.32-5-amd64
#上面命令返回ID:fd855e9b-465f-43d7-b2c1-73b8af289097
glance add name="debian6 OS" disk_format=qcow2 container_format=ovf is_public=true ramdisk_id=9fd89cc3-c479-4544-aeae-6201bfbd504b kernel_id=fd855e9b-465f-43d7-b2c1-73b8af289097 < debian6.img
31 测试glance,glance index会看到上面三个镜像
glance index
glance中存储镜像的本地存储目录用NFS代替,因为后面要模拟控制节点宕机。
32 安装nova组件
apt-get install -y nova-api nova-cert nova-common nova-objectstore nova-scheduler nova-volume nova-consoleauth novnc python-nova python-novaclient nova-compute nova-compute-kvm nova-network
33 修改/etc/nova/api-paste.ini文件
#admin_tenant_name = %SERVICE_TENANT_NAME%
#admin_user = %SERVICE_USER%
#admin_password = %SERVICE_PASSWORD%
admin_tenant_name = service
admin_user = nova
admin_password = password
34 修改/etc/nova/nova.conf配置文件
[DEFAULT]
###### LOGS/STATE
#verbose=True
verbose=False
###### AUTHENTICATION
auth_strategy=keystone
###### SCHEDULER
compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler
scheduler_driver=nova.scheduler.simple.SimpleScheduler
###### VOLUMES
volume_group=nova-volumes
volume_name_template=volume-%08x
iscsi_helper=tgtadm
###### DATABASE
sql_connection=mysql://nova:my_password@vm.my.com/nova
###### COMPUTE
libvirt_type=kvm
#libvirt_type=qemu
connection_type=libvirt
instance_name_template=instance-%08x
api_paste_config=/etc/nova/api-paste.ini
allow_resize_to_same_host=True
libvirt_use_virtio_for_bridges=true
start_guests_on_host_boot=true
resume_guests_state_on_host_boot=true
###### APIS
osapi_compute_extension=nova.api.openstack.compute.contrib.standard_extensions
allow_admin_api=true
s3_host=vm.my.com
cc_host=vm.my.com
###### RABBITMQ
rabbit_host=vm.my.com
###### GLANCE
image_service=nova.image.glance.GlanceImageService
glance_api_servers=vm.my.com:9292
###### NETWORK
network_manager=nova.network.manager.FlatManager
firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver
public_interface=em1
flat_interface=em1
flat_network_bridge=br100
fixed_range=10.1.6.0/24
multi_host=true
###### NOVNC CONSOLE
novnc_enabled=true
novncproxy_base_url= http://vm.my.com:6080/vnc_auto.html
vncserver_proxyclient_address=vm.my.com
vncserver_listen=vm.my.com
########Nova
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lock/nova
#####MISC
use_deprecated_auth=false
root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf
35 重启nova相关服务
service rabbitmq-server restart
service libvirt-bin restart
service nova-scheduler restart
service nova-network restart
service nova-cert restart
service nova-compute restart
service nova-api restart
service nova-objectstore restart
service nova-volume restart
36 初始化nova库,会有警告提示
nova-manage db sync
2012-11-10 10:28:04 WARNING nova.common.deprecated [-] Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statements writing to a table with an auto-increment column after selecting from another table are unsafe because the order in which rows are retrieved determines what (if any) rows will be written. This order cannot be predicted and may differ on master and the slave.
37 创建fix ipnova-manage network create private --fixed_range_v4=10.1.6.0/24 --num_networks=1 \
--bridge=br100 --bridge_interface=br100 --network_size=256 --multi_host=T
38 再次
重启nova相关服务
service rabbitmq-server restart
service libvirt-bin restart
service nova-scheduler restart
service nova-network restart
service nova-cert restart
service nova-compute restart
service nova-api restart
service nova-objectstore restart
service nova-volume restart
39 测试nova服务是否正常启动
nova-manage service list
40 停止不必要的服务
nova-manage service disable --host=control-1-186 --service=nova-compute
nova-manage service disable --host=control-1-186 --service=nova-volume
nova-manage service disable --host=control-1-186 --service=nova-network
40 安装horizon
apt-get install -y apache2 libapache2-mod-wsgi openstack-dashboard
41 安装vnc
apt-get install -y nova-novncproxy nova-xvpvncproxy novnc python-novnc
42 添加安全组策略
nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
43 访问dashboard(E版本中直接my.vm.com访问)
http://my.vm.com/horizon
44 在控制节点上停止nova-network和nova-compute服务,因为要让vm运行在计算节点
/etc/init.d/nova-network stop
/etc/init.d/nova-compute stop
![](http://static.oschina.net/uploads/space/2012/1110/111241_zGVP_123777.jpg)
45 跳转至第二章安装compute-1节点,并设置模拟存储配置
46 compute-1一切妥当后,转到control-node上面,查看有那些镜像和网络可以使用,在创建VM时需要提供相关ID
nova image-list
nova-manage network list
47 启动你的第一台VM
http://my.oschina.net/guol/blog/90134