openstack queens版本环境搭建（十一）：安装swift对象存储服务

最新推荐文章于 2022-03-05 12:02:38 发布

itachi-uchiha

最新推荐文章于 2022-03-05 12:02:38 发布

阅读量1.4k

点赞数 2

分类专栏：云计算-IaaS 文章标签： openstack

本文链接：https://blog.csdn.net/avatar_2009/article/details/108201198

版权

云计算-IaaS 专栏收录该内容

22 篇文章 5 订阅

订阅专栏

Object Storage service, code-named: swift

OpenStack对象存储是一个多租户对象存储系统。它具有高度的可扩展性，可以通过RESTful HTTP API以低成本管理大量非结构化数据。

安装和配置（控制节点）

先决条件

$ . /etc/openstack/admin-openrc

创建swift用户

openstack user create --domain default --password-prompt swift

User Password:你的密码

将admin角色绑定给swift用户

openstack role add --project service --user swift admin

##验证

openstack role assignment list |grep `openstack user list |grep swift|awk '{print $2}'`

创建swift服务，object-store类型

openstack service create --name swift \

--description "OpenStack Object Storage" object-store

创建swift服务的endpoint

openstack endpoint create --region RegionOne \
  object-store public http://controller-150:8080/v1/AUTH_%\(project_id\)s

openstack endpoint create --region RegionOne \
  object-store internal http://controller-150:8080/v1/AUTH_%\(project_id\)s

openstack endpoint create --region RegionOne \
  object-store admin http://controller-150:8080/v1

验证配置

openstack service list |grep swift

openstack endpoint list --service swift

安装和配置组件

yum install openstack-swift-proxy python-swiftclient \

python-keystoneclient python-keystonemiddleware \

memcached -y

注意：上面的安装包部分已经被安装了。

下载存储proxy的配置文件

curl -o /etc/swift/proxy-server.conf https://opendev.org/openstack/swift/raw/branch/stable/queens/etc/proxy-server.conf-sample

编辑配置proxy-server.conf文件

vi /etc/swift/proxy-server.conf

[DEFAULT]

...

bind_port = 8080

user = swift

swift_dir = /etc/swift

[pipeline:main]

pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk ratelimit authtoken keystoneauth container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server

配置模板默认的pipeline多了如下配置：

listing_formats tempurl tempauth copy symlink

[app:proxy-server]

use = egg:swift#proxy

…

node_timeout = 30[xiao1]

...

account_autocreate = True

[filter:keystoneauth]

use = egg:swift#keystoneauth

...

operator_roles = admin,user

[filter:authtoken]

paste.filter_factory = keystonemiddleware.auth_token:filter_factory

...

www_authenticate_uri = http://controller-150:5000

auth_url = http://controller-150:5000

memcached_servers = controller-150:11211

auth_type = password

project_domain_id = default

user_domain_id = default

project_name = service

username = swift

password = 你的密码

delay_auth_decision = True

[filter:cache]

use = egg:swift#memcache

...

memcache_servers = controller-150:11211

具体参见：

https://docs.openstack.org/swift/queens/install/controller-install-rdo.html

安装和配置（对象存储节点）

最好，创建2个node，分别挂载2个device。这里我创建了1个node，挂载了4块device，用1*4来模拟2*2的场景。因为object 最少需要4个device+3个replica。

先决条件

安装依赖包

yum install xfsprogs rsync -y

格式化/dev/sdb、/dev/sdc盘（object节点有5块硬盘，分别200g）

mkfs.xfs /dev/sdb

mkfs.xfs /dev/sdc

mkfs.xfs /dev/sdd

mkfs.xfs /dev/sde

创建挂载点目录

mkdir -p /srv/node/sdb

mkdir -p /srv/node/sdc

mkdir -p /srv/node/sdd

mkdir -p /srv/node/sde

将磁盘挂载配置添加到/etc/fstab，重启系统将自动挂载

vi /etc/fstab

/dev/sdb /srv/node/sdb xfs noatime,nodiratime,nobarrier,logbufs=8 0 2

/dev/sdc /srv/node/sdc xfs noatime,nodiratime,nobarrier,logbufs=8 0 2

/dev/sdd /srv/node/sdd xfs noatime,nodiratime,nobarrier,logbufs=8 0 2

/dev/sde /srv/node/sde xfs noatime,nodiratime,nobarrier,logbufs=8 0 2

手动挂载设备

mount /srv/node/sdb

mount /srv/node/sdc

mount /srv/node/sdd

mount /srv/node/sde

验证

df -hT

Filesystem Type Size Used Avail Use% Mounted on

/dev/sdb xfs 200G 33M 200G 1% /srv/node/sdb

/dev/sdc xfs 200G 33M 200G 1% /srv/node/sdc

/dev/sdd xfs 200G 33M 200G 1% /srv/node/sdd

/dev/sde xfs 200G 33M 200G 1% /srv/node/sde

配置rsyncd服务的配置文件，包含下面的配置内容

vi /etc/rsyncd.conf

uid = swift

gid = swift

log file = /var/log/rsyncd.log

pid file = /var/run/rsyncd.pid

address = 172.5.1.153[W用2]

[account]

max connections = 2[W用3]

path = /srv/node/

read only = False

lock file = /var/lock/account.lock

[container]

max connections = 2

path = /srv/node/

read only = False

lock file = /var/lock/container.lock

[object]

max connections = 2

path = /srv/node/

read only = False

lock file = /var/lock/object.lock

rsync服务请求没有认证，所以在生产环境上，MANAGEMENT_INTERFACE_IP_ADDRESS要使用私网地址。这里我们没有使用192.168.11.153，使用了172.5.1.153。

配置rsync服务开机自启动，并启动rsync服务

systemctl enable rsyncd.service

systemctl start rsyncd.service && systemctl status rsyncd.service

安装和配置组件

安装组件

yum install openstack-swift-account openstack-swift-container \

openstack-swift-object -y

下载swift服务的相关配置文件（先对已有配置文件进行备份）

mv account-server.conf{,.bak}

mv container-server.conf{,.bak}

mv object-server.conf{,.bak}

curl -o /etc/swift/account-server.conf https://opendev.org/openstack/swift/raw/branch/stable/queens/etc/account-server.conf-sample

curl -o /etc/swift/container-server.conf https://opendev.org/openstack/swift/raw/branch/stable/queens/etc/container-server.conf-sample

curl -o /etc/swift/object-server.conf https://opendev.org/openstack/swift/raw/branch/stable/queens/etc/object-server.conf-sample

配置/etc/swift/account-server.conf

vi /etc/swift/account-server.conf

MANAGEMENT_INTERFACE_IP_ADDRESS 使用storage node的管理IP，这里用172.5.1.153

[DEFAULT]

...

bind_ip = 172.5.1.153

bind_port = 6202

user = swift

swift_dir = /etc/swift

devices = /srv/node

mount_check = True

[pipeline:main]

pipeline = healthcheck recon account-server

[filter:recon]

use = egg:swift#recon

...

recon_cache_path = /var/cache/swift

配置container-server.conf

vi /etc/swift/container-server.conf

[DEFAULT]

...

bind_ip = 172.5.1.153

bind_port = 6201

user = swift

swift_dir = /etc/swift

devices = /srv/node

mount_check = True

[pipeline:main]

pipeline = healthcheck recon container-server

[filter:recon]

use = egg:swift#recon

...

recon_cache_path = /var/cache/swift

配置object-server.conf

vi /etc/swift/object-server.conf

[DEFAULT]

...

bind_ip = 172.5.1.153

bind_port = 6200

user = swift

swift_dir = /etc/swift

devices = /srv/node

mount_check = True

[pipeline:main]

pipeline = healthcheck recon object-server

[filter:recon]

use = egg:swift#recon

...

recon_cache_path = /var/cache/swift

recon_lock_path = /var/lock

确保/srv/node目录结构的归属是swift:swift

chown -R swift:swift /srv/node

创建recon目录，配置适当的权限

# mkdir -p /var/cache/swift

# chown -R root:swift /var/cache/swift

# chmod -R 775 /var/cache/swift

具体参见：

https://docs.openstack.org/swift/queens/install/storage-install-rdo.html

添加防火墙策略

rsync：873

object-server: 6200

container-server:6201

account-server:6202

swift-proxy：8080

# firewall-cmd --add-port 873/tcp --add-port 6200-6202/tcp --add-port 8080/tcp --permanent

# firewall-cmd --reload && firewall-cmd --list-port

创建并初始化对象Ring&结束安装并验证（控制节点）

在启动对象存储服务之前，必须创建初始帐户、容器和对象环。ring builder创建配置文件，每个节点使用这些配置文件来确定和部署存储体系结构。为简单起见，本指南使用:

one region two zones with 2^10 (1024) maximum partitions,

3 replicas of each object

对于Object Storage，a partition表示a directory on a storage device，而不是传统的分区表。

但是，我这只做了一个object node，所以只能是1 region + 1 zone + 1replica of each object

后来在该object node上追加了2个device，最后的配置时：1region+2zone+3replica of each object。

创建account ring 帐户环

切到swift配置目录

cd /etc/swift

创建account.builder 库文件（base file）

swift-ring-builder account.builder create 10 3 1

将storage节点添加到ring

STORAGE_NODE_MANAGEMENT_INTERFACE_IP_ADDRESS：172.5.1.153

DEVICE_NAME：/dev/sdb、/dev/sdc、/dev/sdd、/dev/sde

DEVICE_WEIGHT：100

swift-ring-builder account.builder add \

--region 1 --zone 1 --ip 172.5.1.153 --port 6202 \

--device sdb --weight 100

swift-ring-builder account.builder add \

--region 1 --zone 1 --ip 172.5.1.153 --port 6202 \

--device sdc --weight 100

swift-ring-builder account.builder add \

--region 1 --zone 2 --ip 172.5.1.153 --port 6202 \

--device sdd --weight 100

swift-ring-builder account.builder add \

--region 1 --zone 2 --ip 172.5.1.153 --port 6202 \

--device sde --weight 100

验证account环内容

# swift-ring-builder account.builder

account.builder, build version 2[W用4] , id 528a28f48a3445a686b4004279a0abf9

1024 partitions, 3.000000 replicas, 1 regions, 1 zones, 2 devices, 100.00 balance, 0.00 dispersion

The minimum number of hours before a partition can be reassigned is 1 (0:00:00 remaining)

The overload factor is 0.00% (0.000000)

Ring file account.ring.gz not found, probably it hasn't been written yet[W用5]

Devices: id region zone ip address:port replication ip:port name weight partitions balance flags meta

0 1 1 172.5.1.153:6202 172.5.1.153:6202 sdb 100.00 0 -100.00

1 1 1 172.5.1.153:6202 172.5.1.153:6202 sdc 100.00 0 -100.00

Rebalance 帐户环

swift-ring-builder account.builder rebalance

注意：执行上面的命令会失败[W用6] ，因为：Replica count of 3.0 requires more than 2 devices，我们只有2个device，互为副本只有1个replica。

追加设备后再执行成功：

# swift-ring-builder account.builder rebalance

Reassigned 3072 (300.00%) partitions. Balance is now 0.00. Dispersion is now 0.00

创建container ring容器环

cd /etc/swift

创建container.builder 库文件（base file）

swift-ring-builder container.builder create 10 3 1

将storage节点添加到ring

swift-ring-builder container.builder add \

--region 1 --zone 1 --ip 172.5.1.153 --port 6201 --device sdb --weight 100

swift-ring-builder container.builder add \

--region 1 --zone 1 --ip 172.5.1.153 --port 6201 --device sdc --weight 100

swift-ring-builder container.builder add \

--region 1 --zone 2 --ip 172.5.1.153 --port 6201 --device sdd --weight 100

swift-ring-builder container.builder add \

--region 1 --zone 2 --ip 172.5.1.153 --port 6201 --device sde --weight 100

验证container环内容

swift-ring-builder container.builder

Rebalance 容器环

swift-ring-builder container.builder rebalance

Reassigned 3072 (300.00%) partitions. Balance is now 0.00. Dispersion is now 0.00

创建object ring 对象环

cd /etc/swift

创建object.builder 库文件（base file）

swift-ring-builder object.builder create 10 3 1

将storage节点添加到ring

swift-ring-builder object.builder add \

--region 1 --zone 1 --ip 172.5.1.153 --port 6200 --device sdb --weight 100

swift-ring-builder object.builder add \

--region 1 --zone 1 --ip 172.5.1.153 --port 6200 --device sdc --weight 100

swift-ring-builder object.builder add \

--region 1 --zone 2 --ip 172.5.1.153 --port 6200 --device sdd --weight 100

swift-ring-builder object.builder add \

--region 1 --zone 2 --ip 172.5.1.153 --port 6200 --device sde --weight 100

验证container环内容

swift-ring-builder object.builder

Rebalance 容器环

swift-ring-builder object.builder rebalance

Reassigned 3072 (300.00%) partitions. Balance is now 0.00. Dispersion is now 0.00

分发ring配置文件到各个Storage节点

复制account.ring.gz, container.ring.gz, and object.ring.gz 到所有的storage节点的/etc/swift目录。

scp account.ring.gz container.ring.gz object.ring.gz 192.168.11.153:/etc/swift/

具体参见：

https://docs.openstack.org/swift/queens/install/

结束安装（控制节点）

下载/etc/swift/swift.conf文件

cd /etc/swift

mv swift.conf{,.bak}

curl -o /etc/swift/swift.conf \

https://opendev.org/openstack/swift/raw/branch/stable/queens/etc/swift.conf-sample

编辑/etc/swift/swift.conf

vi /etc/swift/swift.conf

swift_hash_path_suffix和swift_hash_path_prefix作为哈希算法的一部分用于确定数据在集群中的位置。

这些值应该保持机密，并且在部署集群之后不能更改，丢失。

只能使用可打印字符：python -c "import string; print(string.printable)"

这里我们使用下面的默认值即可，不过我做了自定义：

HASH_PATH_SUFFIX：HPS_liuyl_001

HASH_PATH_PREFIX：HPP_liuyl_001

[swift-hash]

...

swift_hash_path_suffix = HPS_liuyl_001

swift_hash_path_prefix = HPP_liuyl_001

[storage-policy:0]

...

name = Policy-0

default = yes

分发到各个Storage节点的/etc/swift/目录

scp swift.conf 192.168.11.153:/etc/swift/

所有节点（controller+object）配置恰当的权限

chown -R root:swift /etc/swift

controller启动swift-proxy服务

查看一下Memcached的状态，如果已经启动且enabled可以忽略，否则和openstack-swift-proxy一起执行一遍

systemctl status memcached

systemctl enable openstack-swift-proxy.service

systemctl start openstack-swift-proxy.service && systemctl status openstack-swift-proxy.service

object节点启动如下服务

account相关

systemctl enable openstack-swift-account.service openstack-swift-account-auditor.service \

openstack-swift-account-reaper.service openstack-swift-account-replicator.service

systemctl start openstack-swift-account.service openstack-swift-account-auditor.service \

openstack-swift-account-reaper.service openstack-swift-account-replicator.service \

&& systemctl status openstack-swift-account.service openstack-swift-account-auditor.service \

openstack-swift-account-reaper.service openstack-swift-account-replicator.service

container相关

systemctl enable openstack-swift-container.service \

openstack-swift-container-auditor.service openstack-swift-container-replicator.service \

openstack-swift-container-updater.service

systemctl start openstack-swift-container.service \

openstack-swift-container-auditor.service openstack-swift-container-replicator.service \

openstack-swift-container-updater.service \

&& systemctl status openstack-swift-container.service \

openstack-swift-container-auditor.service openstack-swift-container-replicator.service \

openstack-swift-container-updater.service

object相关

systemctl enable openstack-swift-object.service openstack-swift-object-auditor.service \

openstack-swift-object-replicator.service openstack-swift-object-updater.service

验证

Object节点操作

如果使用redhat7或者centos7 需在object上执行下面的操作，否则selinux限制。

# ll -dZ /srv/node

drwxr-xr-x. swift swift unconfined_u:object_r:var_t:s0 /srv/node

# chcon -R system_u:object_r:swift_data_t:s0 /srv/node

Controller节点操作

$ . /etc/openstack/demo-openrc

##查看服务状态

$ swift stat

Account: AUTH_bc68b3c0986643d593c5fee1b66a9b3c

Containers: 0

Objects: 0

Bytes: 0

X-Put-Timestamp: 1563181623.33845

X-Timestamp: 1563181623.33845

X-Trans-Id: tx72cc80dcd99f4d1092d8d-005d2c4237

Content-Type: text/plain; charset=utf-8

X-Openstack-Request-Id: tx72cc80dcd99f4d1092d8d-005d2c4237

##创建container1容器

$ openstack container create container1

##上传一个文件到container1容器

FILE:/etc/openstack/demo-openrc

openstack object create container1 /etc/openstack/demo-openrc

默认上传到container上的文件名称和即本地文件名称，通过--name重命名文件或者指定目录，如下：

$ openstack object create container1 down-demo-openrc --name test-dir-2/down-demo-openrc-2

创建的文件名最好不要从/开始，如此会导致object在horizon中显示异常：

##查看容器container1中的文件

$ openstack object list container1

+---------------------------------+

| Name |

+---------------------------------+

| /etc/openstack/demo-openrc |

+---------------------------------+

##下载容器container1中的某个文件

FILE:/etc/openstack/demo-openrc

openstack object save container1 /etc/openstack/demo-openrc

目标文件名默认是object name，所以上面的命令会把demo-openrc写到/etc/openstack/下，名称为：demo-openrc。但是，普通用户没有写入权限，会报错：

[Errno 13] Permission denied: u'/etc/openstack/demo-openrc'

通过--file指定目标文件名称，不指定目录，默认为当前目录。

openstack object save container1 /etc/openstack/demo-openrc --file xxxx

指定文件名为-，则会将文件内容打印到stdout

openstack object save container1 /etc/openstack/demo-openrc --file -

export OS_PROJECT_DOMAIN_NAME=Default

export OS_USER_DOMAIN_NAME=Default

export OS_PROJECT_NAME=demo

……

再次查询swift状态

$ swift stat

Account: AUTH_bc68b3c0986643d593c5fee1b66a9b3c

Containers: 1

Objects: 1

Bytes: 271

Containers in policy "policy-0": 1

Objects in policy "policy-0": 1

Bytes in policy "policy-0": 271

X-Account-Project-Domain-Id: default

X-Openstack-Request-Id: txd9fa1719578e4d75a9987-005d2c4600

X-Timestamp: 1563181753.87945

X-Trans-Id: txd9fa1719578e4d75a9987-005d2c4600

Content-Type: application/json; charset=utf-8

Accept-Ranges: bytes

同时有个问题：swift stat状态查询的object个数和openstack object list container_name查询出来的不一致。

[xiao1]这里默认超时时间10s，如果上传大文件，controller和swift node之间网络传输速度慢，容易超时。

设置为30s，刚刚能上传1.5G大小文件。上传8.8G的Windows镜像还是超时失败。

[W用2]Storage node的管理IP，鉴于安全考虑这里使用172.5.1.153

[W用3]max_connections=2，是配置手册上的，这里我创建了4个device，应该改成4.下同

[W用4]创建几个device，这里就是几，后来追加2个device，这里变成4.

[W用5]Rebalance后account.ring.gz生成。

[W用6]所以在153node上追加了sdd、sde两块设备，模拟2*2场景

itachi-uchiha

关注

2
点赞
踩
3

收藏

觉得还不错? 一键收藏
0
评论
复制链接

分享到 QQ

分享到新浪微博

扫一扫

专栏目录

openstack queens版本 环境搭建（十一）：安装swift对象存储服务

Object Storage service, code-named: swift

安装和配置（控制节点）

先决条件

安装和配置组件

安装和配置（对象存储节点）

先决条件

安装和配置组件

添加防火墙策略

创建并初始化对象Ring&结束安装并验证（控制节点）

创建account ring 帐户环

创建container ring容器环

创建object ring 对象环

结束安装（控制节点）

验证

openstack queens版本环境搭建（十一）：安装swift对象存储服务