swift 多节点部署

http://www.openstack.org.cn/bbs/forum.php?mod=viewthread&tid=264 一、安装环境： 代理节点： Ubuntu　11.04　server hostname：proxy eht0：192.168.2.220 eth1：10.0.0.2 存储节点一： Ubuntu　11.04　server hostname：storage1 eht0：10.0.0.3 存储节点二： Ubuntu　11.04　server hostname：storage2 eht0：10.0.0.4 存储节点三： Ubuntu　11.04　server hostname：storage3 eht0：10.0.0.5 二、安装代理节点： 1. 安装 Swift 的先决条件 apt-get install python-software-properties add-apt-repository ppa:swift-core/ppa apt-get update apt-get install swift openssh-server 2. 创建 Swift 的工作目录 mkdir -p /etc/swift chown -R swift:swift /etc/swift/ 3. 创建/etc/swift/swift.conf cat >/etc/swift/swift.conf <<EOF [swift-hash] # random unique string that can never change (DO NOT LOSE) swift_hash_path_suffix = `od -t x8 -N 8 -A n </dev/random` EOF 4. 安装swift-proxy服务 apt-get install swift-proxy memcached 5. 创建SSL自签名证书 cert cd /etc/swift openssl req -new -x509 -nodes -out cert.crt -keyout cert.key 6. 配置 memcached ，监听默认的端口。推荐配置为内部的、非公用的IP网络地址。编辑 /etc/memcached.conf perl -pi -e "s/-l 127.0.0.1/-l $PROXY_LOCAL_NET_IP/" /etc/memcached.conf 7. 重启 memcached server: service memcached restart 8. 创建 /etc/swift/proxy-server.conf （其中$PROXY_LOCAL_NET_IP 使用10.0.0.2代替 ）: cat >/etc/swift/proxy-server.conf <<EOF [DEFAULT] cert_file = /etc/swift/cert.crt key_file = /etc/swift/cert.key bind_port = 8080 workers = 8 user = swift [pipeline:main] pipeline = healthcheck cache swauth proxy-server [app:proxy-server] use = egg:swift#proxy allow_account_management = true [filter:swauth] use = egg:swift#swauth default_swift_cluster = local#https://$PROXY_LOCAL_NET_IP:8080/v1 # Highly recommended to change this key to something else! super_admin_key = swauthkey [filter:healthcheck] use = egg:swift#healthcheck [filter:cache] use = egg:swift#memcache memcache_servers = <PROXY_LOCAL_NET_IP>:11211 EOF 9. 创建 account, container 和 object ring。swift-ring-builder 命令创建相应的.builder文件会有几个参数。18代表partition的数目会到2^18幂，这个数字取决于你希望一个ring中会有多少个partition。3代表每个object的副本数。最后一个参数1，代表一个partition至少在1小时之后才能被移动。: cd /etc/swift swift-ring-builder account.builder create 18 3 1 swift-ring-builder container.builder create 18 3 1 swift-ring-builder object.builder create 18 3 1 10. ring 需要记录每个 Store Node /srv/node 中记录的存储设备 swift-ring-builder account.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6002/sda6 100 swift-ring-builder container.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6001/sda6 100 swift-ring-builder object.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6000/sda6 100 11. 确认 ring 的内容是否正确: swift-ring-builder account.builder swift-ring-builder container.builder swift-ring-builder object.builder 12. 平衡(Rebalance) ring swift-ring-builder account.builder rebalance swift-ring-builder container.builder rebalance swift-ring-builder object.builder rebalance 13. 确保所有的配置文件都属于 swift 用户 chown -R swift:swift /etc/swift 14. 开启 Proxy 服务: swift-init proxy start 三、安装配置存储节点 1. 安装 Swift 的先决条件 apt-get install python-software-properties add-apt-repository ppa:swift-core/release apt-get update apt-get install swift openssh-server 2. 创建 Swift 的工作目录 mkdir -p /etc/swift chown -R swift:swift /etc/swift/ 3. 从proxy节点中复制/etc/swift/swift.conf，/etc/swift/object.ring.gz，/etc/swift/container.ring.gz，/etc/swift/account.ring.gz到/etc/swift目录 scp 10.0.0.2:/etc/swift/swift.conf /etc/swift/ scp 10.0.0.2:/etc/swift/object.ring.gz /etc/swift/ scp 10.0.0.2:/etc/swift/container.ring.gz /etc/swift/ scp 10.0.0.2:/etc/swift/account.ring.gz /etc/swift/ 4. 安装配置 Storage node 相关 apt-get install swift-account swift-container swift-object xfsprogs 5. 对 Storage node 的每一个存储设备, 安装 XFS 逻辑卷  cfdisk#创建一个新的分区 reboot mkfs.xfs -i size=1024 /dev/sda6 echo "/dev/sda6 /srv/node/sda6 xfs noatime,nodiratime,nobarrier,logbufs=8 0 0" >> /etc/fstab mkdir -p /srv/node/sda6 mount /srv/node/sda6 chown -R swift:swift /srv/node 6. 创建 /etc/rsyncd.conf cat >/etc/rsyncd.conf <<EOF uid = swift gid = swift log file = /var/log/rsyncd.log pid file = /var/run/rsyncd.pid address = $STORAGE_LOCAL_NET_IP [account] max connections = 2 path = /srv/node/ read only = false lock file = /var/lock/account.lock [container] max connections = 2 path = /srv/node/ read only = false lock file = /var/lock/container.lock [object] max connections = 2 path = /srv/node/ read only = false lock file = /var/lock/object.lock EOF 7. 修改 /etc/default/rsync 中的 RSYNC_ENABLE= 属性 perl -pi -e 's/RSYNC_ENABLE=false/RSYNC_ENABLE=true/' /etc/default/rsync 8. 开启 rsync 守护进程 service rsync start 9. 创建 /etc/swift/account-server.conf cat >/etc/swift/account-server.conf <<EOF [DEFAULT] bind_ip =$STORAGE_LOCAL_NET_IP workers = 2 [pipeline:main] pipeline = account-server [app:account-server] use = egg:swift#account [account-replicator] [account-auditor] [account-reaper] EOF 10. 创建 /etc/swift/container-server.conf cat >/etc/swift/container-server.conf <<EOF [DEFAULT] bind_ip =$STORAGE_LOCAL_NET_IP workers = 2 [pipeline:main] pipeline = container-server [app:container-server] use = egg:swift#container [container-replicator] [container-updater] [container-auditor] EOF 11. 创建 /etc/swift/object-server.conf cat >/etc/swift/object-server.conf <<EOF [DEFAULT] bind_ip = $STORAGE_LOCAL_NET_IP workers = 2 [pipeline:main] pipeline = object-server [app:object-server] use = egg:swift#object [object-replicator] [object-updater] [object-auditor] EOF 12. 开启 storage 服务 swift-init all start 四、 创建swift管理员帐户并测试 1. 创建一个管理员帐户（account = system, username = root, password = testpass），Make sure to replace swauthkey with whatever super_admin key you assigned in the proxy-server.conf file above. swauth-prep -A https://proxy:8080/auth/ -K swauthkey swauth-add-user -A https://proxy:8080/auth/ -K swauthkey -a system root testpass 2. 获得 X-Storage-Url 和 X-Auth-Token curl -k -v -H 'X-Storage-User: system:root' -H 'X-Storage-Pass: testpass' https://proxy:8080/auth/v1.0 ++++++++++++++++++++++++++++++ root@proxy:/etc/swift# curl -k -v -H 'X-Storage-User: system:root' -H 'X-Storage-Pass: testpass'https://proxy:8080/auth/v1.0                                            * About to connect() to proxy port 8080 (#0) *   Trying 127.0.1.1... connected * Connected to proxy (127.0.1.1) port 8080 (#0) * successfully set certificate verify locations: *   CAfile: none   CApath: /etc/ssl/certs * SSLv3, TLS handshake, Client hello (1): * SSLv3, TLS handshake, Server hello (2): * SSLv3, TLS handshake, CERT (11): * SSLv3, TLS handshake, Server finished (14): * SSLv3, TLS handshake, Client key exchange (16): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSL connection using AES256-SHA * Server certificate: *        subject: C=CN; ST=HB; L=WH; O=CGT; OU=cloud; CN=james; emailAddress=jameszhou@itspiderman.com *        start date: 2011-08-08 08:25:15 GMT *        expire date: 2011-09-07 08:25:15 GMT *        common name: james (does not match 'proxy') *        issuer: C=CN; ST=HB; L=WH; O=CGT; OU=cloud; CN=james; emailAddress=jameszhou@itspiderman.com *        SSL certificate verify result: self signed certificate (18), continuing anyway. > GET /auth/v1.0 HTTP/1.1 > User-Agent: curl/7.21.3 (x86_64-pc-linux-gnu) libcurl/7.21.3 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18 > Host: proxy:8080 > Accept: */* > X-Storage-User: system:root > X-Storage-Pass: testpass > < HTTP/1.1 200 OK < X-Storage-Url: https://10.0.0.2:8080/v1/AUTH_17140a0f-8e59-416d-ad70-6b97a8209b2c < X-Storage-Token: AUTH_tk5da88107b667402097a36e9ebf64688b < X-Auth-Token: AUTH_tk5da88107b667402097a36e9ebf64688b < Content-Length: 112 < Date: Mon, 08 Aug 2011 08:57:51 GMT < * Connection #0 to host proxy left intact * Closing connection #0 * SSLv3, TLS alert, Client hello (1): {"storage": {"default": "local", "local": "https://10.0.0.2:8080/v1/AUTH_17140a0f-8e59-416d-ad70-6b97a8209b2c"}}root@proxy:/e              ++++++++++++++++++++++++++++++++++ 3. 检测是否可以使用 HEAD 命令查询 account 状态 curl -k -v -H 'X-Auth-Token: AUTH_tk5da88107b667402097a36e9ebf64688b' https://10.0.0.2:8080/v1/AUTH_17140a0f-8e59-416d-ad70-6b97a8209b2c 4. 检测 st 命令是否工作正常 (输出应为 zero containers, zero objects, 和 zero bytes): st -A https://proxy:8080/auth/v1.0 -U system:root -K testpass stat 5. 上传文件到myfiles容器中 st -A https://proxy:8080/auth/v1.0 -U system:root -K testpass upload myfiles test.txt 6. 下载myfiles容器中的所有文件 st -A https://proxy:8080/auth/v1.0 -U system:root -K testpass download myfiles 7. 下载myfiles容器中指定文件 st -A https://proxy:8080/auth/v1.0 -U system:root -K testpass download myfiles test.txt 8. 保存ring的.builder 构建文件到container ‘builders’ st -A https://proxy:8080/auth/v1.0 -U system:root -K testpass upload builders /etc/swift/*.builder 9. 列出特定用户的容器 st -A https://proxy:8080/auth/v1.0 -U system:root -K testpass list 10. 查看container ‘builders’中的 object列表 st -A https://proxy:8080/auth/v1.0 -U system:root -K testpass list builders 11. 下载container ‘builders’中的object st -A https://proxy:8080/auth/v1.0 -U system:root -K testpass download builders ------------------------------------------------ root@proxy:~# more /etc/network/interfaces # This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo iface lo inet loopback # The primary network interface auto eth0 iface eth0 inet static address 192.168.2.220 netmask 255.255.255.0 gateway 192.168.2.1 auto eth1 iface eth1 inet static address 10.0.0.2 netmask 255.255.255.0 root@proxy:~# ifconfig eth0      Link encap:Ethernet  HWaddr 00:1f:d0:ad:95:50           inet addr:192.168.2.220  Bcast:192.168.2.255  Mask:255.255.255.0           inet6 addr: fe80::21f:d0ff:fead:9550/64 Scope:Link           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1           RX packets:1026 errors:0 dropped:53 overruns:0 frame:0           TX packets:800 errors:0 dropped:0 overruns:0 carrier:0           collisions:0 txqueuelen:1000           RX bytes:99692 (99.6 KB)  TX bytes:164044 (164.0 KB)           Interrupt:42 Base address:0xc000 eth1      Link encap:Ethernet  HWaddr 00:05:5d:65:dc:02           inet addr:10.0.0.2  Bcast:10.0.0.255  Mask:255.255.255.0           inet6 addr: fe80::205:5dff:fe65:dc02/64 Scope:Link           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1           RX packets:408 errors:0 dropped:0 overruns:0 frame:0           TX packets:596 errors:0 dropped:0 overruns:0 carrier:0           collisions:0 txqueuelen:1000           RX bytes:49544 (49.5 KB)  TX bytes:67430 (67.4 KB)           Interrupt:19 Base address:0x4000 lo        Link encap:Local Loopback           inet addr:127.0.0.1  Mask:255.0.0.0           inet6 addr: ::1/128 Scope:Host           UP LOOPBACK RUNNING  MTU:16436  Metric:1           RX packets:411 errors:0 dropped:0 overruns:0 frame:0           TX packets:411 errors:0 dropped:0 overruns:0 carrier:0           collisions:0 txqueuelen:0           RX bytes:59865 (59.8 KB)  TX bytes:59865 (59.8 KB)