OpenStack对象存储服务swift多节点安装（VirtualBox multi-host）

OpenStack对象存储服务swift多节点安装（VirtualBoxmulti-host）

 

 

 

 

LIUYulong

刘宇龙

目录

0.基本结构...3

1. 虚拟机的创建... 3

1.1 代理节点虚拟机proxy-node. 3

1.2 五个存储节点      swift-Storage-server.4

2.网络配置...5

2.1 proxy节点... 5

2.2各个存储节点...5

3.各个节点安装的公共软件和操作...6

3.1公共部分的软件安装...6

3.2各个节点的公共操作...7

4.proxy节点的安装和操作...8

4.1安装proxy服务...8

4.2生成ssh-key.8

4.3配置memcached服务...8

4.4 配置proxy服务/etc/swift/proxy-server.conf，使用keystone认证...8

4.5创建account, container 以及object环（ring）... 10

4.6添加各个存储节点到环中...10

4.7验证环...10

4.8平衡环...11

4.9复制4.8生成的account.ring.gz, container.ring.gz, and object.ring.gz到 Proxy 和各个Storage 节点中的目录/etc/swift.11

4.10确保swift用户对/etc/swift目录的权限...11

4.11启动proxy服务...12

5.storage存储节点的安装和操作...12

5.1相关软件的安装...12

5.2给swift分配新硬盘...12

5.3 创建并设置/etc/rsyncd.conf12

5.4设置/etc/default/rsync.13

5.5启动rsync.13

5.6设置account-server服务 /etc/swift/account-server.conf13

5.7设置container-server服务/etc/swift/container-server.conf14

5.8设置object-server 服务/etc/swift/object-server.conf14

6.安装keystone参考其他文档...15

7.验证安装...22

7.1查看租户状态...22

7.2上传文件到swift.22

7.3查看各个存储节点的对象分布...23

 

0.基本结构

 

1. 虚拟机的创建

1.1 代理节点虚拟机proxy-node

       （1）双网卡

       （2）高计算性能  2核心cpu  

       （3）一个硬盘  40G

1.2 五个存储节点    swift-Storage-server

       （1）单网卡

       （2）计算性能要求不高  1核心cpu

       （3）两个40G硬盘  系统盘和swift数据盘

1.2各个服务器系统安装

选择ubuntu-12.04.2-desktop-amd64.iso

2.网络配置

2.1 proxy节点

删除ununtu的网络管理器network-manager

sudo apt-get purge network-manager

配置network网络管理器

sudo vim /etc/network/interfaces

        auto lo

        iface lo inet loopback

        auto eth0

        iface eth0 inet dhcp

        auto eth1

        iface eth1 inet static

         address 172.16.0.1

         netmask 255.255.0.0

         network 172.16.0.0

         broadcast 172.16.255.255

重启网络

sudo /etc/init.d/networking restart

 

2.2各个存储节点

如图：四个存储节点配置在和代理节点172.16.xxx.xxx。

这里设置的IP分别为：

    swift-Storage-server-01:172.16.0.2

    swift-Storage-server-02:172.16.0.3

    swift-Storage-server-03:172.16.0.4

    swift-Storage-server-04:172.16.0.5

删除ununtu的网络管理器network-manager

sudo apt-get purge network-manager

配置network网络管理器

sudo vim /etc/network/interfaces

        auto lo

        iface lo inet loopback

        auto eth0

        iface eth0 inet dhcp

        auto eth1

        iface eth1 inet static

                address 172.16.0.2

                netmask 255.255.0.0

                network 172.16.0.0

                boradcast 172.16.255.255

重启网络

sudo /etc/init.d/networking restart

3.各个节点安装的公共软件和操作

3.1公共部分的软件安装

(1)最新版的SAIO - Swift AllIn One

apt-get update

apt-get install curl gcc memcached rsync sqlite3 xfsprogs git-core libffi-dev python-setuptools

apt-get install python-coverage python-dev python-nose python-simplejson python-xattr python-eventlet python-greenlet python-pastedeploy python-netifaces python-pip python-dnspython python-mock python-mysqldbsudo apt-get install -y  python-software-properties

(2)老版本的SAIO - Swift AllIn One

sudo apt-get install -y  python-software-properties

sudo add-apt-repository -y ppa:swift-core/release

sudo apt-get update

sudo apt-get install -y  curl gcc git-core memcached python-coverage python-dev python-nose python-setuptools python-simplejson python-xattr sqlite3 xfsprogs python-eventlet python-greenlet python-pastedeploy python-netifaces python-pip

sudo pip install  mock

sudo apt-get install -y  screen ssh  vim

（3）官方网站的多节点安装

apt-get install python-software-properties

add-apt-repository ppa:swift-core/release

apt-get update

apt-get install swift python-swiftclient openssh-server

3.2各个节点的公共操作

(1)给swift用户目录权限

mkdir -p /etc/swift

chown -R swift:swift /etc/swift/

(2)随机生成swift.conf

cat >/etc/swift/swift.conf <<EOF

[swift-hash]

# random unique strings that can never change (DO NOT LOSE)

swift_hash_path_prefix = `od -t x8 -N 8 -A n </dev/random`

swift_hash_path_suffix = `od -t x8 -N 8 -A n </dev/random`

EOF

(3) 拷贝(2)生成的swift.conf到所有节点的/etc/swift/目录，包括proxy-node和storage-node

scp firstnode.example.com:/etc/swift/swift.conf /etc/swift/

(4)给swift用户/var/run/swift目录权限

mkdir -p /var/run/swift

chown swift:swift /var/run/swift

4.proxy节点的安装和操作

4.1安装proxy服务

apt-get install swift-proxy memcached

4.2生成ssh-key

cd /etc/swift

openssl req -new -x509 -nodes -out cert.crt -keyout cert.key

4.3配置memcached服务

perl -pi -e "s/-l 127.0.0.1/-l $PROXY_LOCAL_NET_IP/" /etc/memcached.conf

 

service memcached restart

4.4 配置proxy服务/etc/swift/proxy-server.conf，使用keystone认证

[DEFAULT]

#cert_file = /etc/swift/cert.crt

#key_file = /etc/swift/cert.key

bind_port = 8888

bind_ip=172.16.0.1

user = swift

log_facility = LOG_LOCAL1

#[pipeline:main]

#pipeline = healthcheck cache tempauth proxy-logging proxy-server

[pipeline:main]

pipeline = healthcheck cache  authtoken keystone proxy-server

[app:proxy-server]

use = egg:swift#proxy

allow_account_management = true

account_autocreate = true

#[filter:tempauth]

#use = egg:swift#tempauth

#user_admin_admin = admin .admin .reseller_admin

#user_test_tester = testing .admin

#user_test2_tester2 = testing2 .admin

#user_test_tester3 = testing3

[filter:keystone]

paste.filter_factory = keystone.middleware.swift_auth:filter_factory

operator_roles = Member,admin, swiftoperator

[filter:authtoken]

paste.filter_factory = keystone.middleware.auth_token:filter_factory

# Delaying the auth decision is required to support token-less

# usage for anonymous referrers ('.r:*').

delay_auth_decision = 10

service_port = 5000

service_host = 172.16.0.1

auth_port = 35357

auth_host = 172.16.0.1

auth_protocol = http

auth_uri = http://172.16.0.1:5000/

auth_token =  0123456789TOKEN987654231

admin_token = 0123456789TOKEN987654231

admin_tenant_name = service

admin_user = swift

admin_password = swiftpass

[filter:healthcheck]

use = egg:swift#healthcheck

[filter:cache]

use = egg:swift#memcache

[filter:proxy-logging]

use = egg:swift#proxy_logging

4.5创建account, container 以及object环（ring）

cd /etc/swift

swift-ring-builder account.builder create 18 3 1

swift-ring-builder container.builder create 18 3 1

swift-ring-builder object.builder create 18 3 1

4.6添加各个存储节点到环中

4个节点

172.16.0.2-172.16.0.5

swift-ring-builder account.builder add z1-172.16.0.2:6002/sdb1 100

swift-ring-builder container.builder add z1-172.16.0.2:6001/sdb1 100

swift-ring-builder object.builder add z1-172.16.0.2:6000/sdb1 100

swift-ring-builder account.builder add z2-172.16.0.3:6002/sdb1 100

swift-ring-builder container.builder add z2-172.16.0.3:6001/sdb1 100

swift-ring-builder object.builder add z2-172.16.0.3:6000/sdb1 100

swift-ring-builder account.builder add z3-172.16.0.4:6002/sdb1 100

swift-ring-builder container.builder add z3-172.16.0.4:6001/sdb1 100

swift-ring-builder object.builder add z3-172.16.0.4:6000/sdb1 100

swift-ring-builder account.builder add z4-172.16.0.5:6002/sdb1 100

swift-ring-builder container.builder add z4-172.16.0.5:6001/sdb1 100

swift-ring-builder object.builder add z4-172.16.0.5:6000/sdb1 100

4.7验证环

swift-ring-builder account.builder

swift-ring-builder container.builder

swift-ring-builder object.builder

结果如下：

4.8平衡环

swift-ring-builder account.builder rebalance

swift-ring-builder container.builder rebalance

swift-ring-builder object.builder rebalance

4.9复制4.8生成的account.ring.gz,container.ring.gz, and object.ring.gz到 Proxy 和各个Storage 节点中的目录/etc/swift.

scp 172.16.0.1:/etc/swift/*.gz /etc/swift/

4.10确保swift用户对/etc/swift目录的权限

chown -R swift:swift /etc/swift

4.11启动proxy服务

swift-init proxy start

 

5.storage存储节点的安装和操作

5.1相关软件的安装

apt-get install swift-account swift-container swift-object xfsprogs

 

5.2给swift分配新硬盘

fdisk /dev/sdb 

        n 写入分区

        p 主分区

        e扩展分区

        w 写入分区表退出

mkfs.xfs -i size=512 /dev/sdb1

echo "/dev/sdb1 /srv/node/sdb1 xfs noatime,nodiratime,nobarrier,logbufs=8 0 0" >> /etc/fstab

mkdir -p /srv/node/sdb1

mount /srv/node/sdb1

chown swift:swift /srv/node/sdb1

5.3创建并设置/etc/rsyncd.conf

cat >/etc/rsyncd.conf <<EOF

uid = swift

gid = swift

log file = /var/log/rsyncd.log

pid file = /var/run/rsyncd.pid

address = 172.16.0.2-172.16.0.5

[account]

max connections = 2

path = /srv/node/

read only = false

lock file = /var/lock/account.lock

[container]

max connections = 2

path = /srv/node/

read only = false

lock file = /var/lock/container.lock

[object]

max connections = 2

path = /srv/node/

read only = false

lock file = /var/lock/object.lock

EOF

5.4设置/etc/default/rsync

perl -pi -e 's/RSYNC_ENABLE=false/RSYNC_ENABLE=true/' /etc/default/rsync

5.5启动rsync

service rsync start

5.6设置account-server服务 /etc/swift/account-server.conf

cat >/etc/swift/account-server.conf <<EOF

[DEFAULT]

bind_ip = 172.16.0.2-172.16.0.5

workers = 2

[pipeline:main]

pipeline = account-server

[app:account-server]

use = egg:swift#account

[account-replicator]

[account-auditor]

[account-reaper]

EOF

5.7设置container-server服务/etc/swift/container-server.conf

cat >/etc/swift/container-server.conf <<EOF

[DEFAULT]

bind_ip = 172.16.0.2-172.16.0.5

workers = 2

[pipeline:main]

pipeline = container-server

[app:container-server]

use = egg:swift#container

[container-replicator]

[container-updater]

[container-auditor]

[container-sync]

EOF

5.8设置object-server服务/etc/swift/object-server.conf

cat >/etc/swift/object-server.conf <<EOF

[DEFAULT]

bind_ip = 172.16.0.2-172.16.0.5

workers = 2

[pipeline:main]

pipeline = object-server

[app:object-server]

use = egg:swift#object

[object-replicator]

[object-updater]

[object-auditor]

EOF

5.9启动各个存储节点的各个服务

swift-init all start

6.安装keystone参考其他文档

 

7.验证安装

7.1查看租户状态

swift -v -V 2.0 -A http://172.16.0.1:5000/v2.0 -U openstackDemo:adminUser -K secretword stat

结果如下：

7.2上传文件到swift

swift -v -V 2.0 -A http://172.16.0.1:5000/v2.0 -U openstackDemo:adminUser -K secretword nihao ./insall.txt

7.3查看各个存储节点的对象分布