基于opens tack的swift存储搭建过程

平台服务器安装ubuntu-14.04.4-server-amd64.iso。(貌似现在支持16.04？最好使用这个版本)

使用swift 3.0.0实现。

感觉国内对于swift讲解很少强烈推荐一本书（云存储系统——swift的原理、架构及实现，人民邮电的，感觉讲的非常好）

表1.1：节点ip配置

节点	eth0	硬盘
controller	10.80.118.87
storage1	10.82.25.128	10G sdb
storage 2	10.82.27.134	10G sdb
storage 3	10.82.13.135	10G sdb

设置root密码（以下都在root内执行）

Sudo passwd root

2.1 主机名配置

所有节点主机名分别设置为：controller、storage1、storage2、storage3。

所有节点/etc/hosts，添加：

ip   controller

ip   storage1

ip   storage2

ip   storage3

 

设置完之后，重启服务器，使配置生效。（或者source一下也行）

2.2、防火墙配置

所有节点，在/etc/rc.local添加iptables –F（清除链表中的规则，相当于初始化）

2.3、系统源更新制作

apt-get install software-properties-common

add-apt-repositorycloud-archive:mitaka   (仓库)

apt-get update 

apt-get dist-upgrade

重启服务器

apt-get install python-openstackclient

三、数据库安装

在controller节点安装数据库。

1、安装数据库

apt-get install mariadb-serverpython-pymysql （mysql的分支，性能却由于mysql）

2、配置数据库

在/etc/mysql/conf.d/路径下，创建openstack.cnf，内容如下所示。

[mysqld]

bind-address=10.80.118.87（controller节点IP）

default-storage-engine=innodb

innodb_file_per_table

collation-server=utf8_general_ci

character-set-server=utf8

 

执行命令

service mysql restart

mysql_secure_installation

四、Memcache配置

再控制节点执行。

apt-get install memcached python-memcache   （缓存）

编辑 /etc/memcached.conf

-l 10.80.118.87

偶尔报错（重启服务sudo: unable to resolve host10-9-154-112）可能是/etc/hosts没有配置好

service memcached restart

五、认证服务

5.1、数据库添加认证信息

mysql –u root –p

CREATE DATABASE keystone;

GRANT ALL PRIVILEGES ON keystone.* TO'keystone'@'localhost' IDENTIFIED BY '123456';

GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' IDENTIFIED BY '123456';

5.2、认证服务安装配置

在控制节点操作。

取消keystone开机自启动。

echo "manual" > /etc/init/keystone.override

下载认证服务

apt-get install keystone apache2 libapache2-mod-wsgi

 

编辑/etc/keystone/keystone.conf

[DEFAULT]

admin_token = 123456

[database]

connection = mysql+pymysql://keystone:123456@controller/keystone

[token]

provider = fernet

其它项保持不变。

同步数据库，初始化fernet key。

su -s /bin/sh -c "keystone-manage db_sync" keystone

keystone-manage fernet_setup --keystone-user keystone--keystone-group keystone

 

5.3、Apache HTTP server服务配置

管理节点操作

编辑/etc/apache2/apache2.conf ，添加

ServerName controller

创建/etc/apache2/sites-available/wsgi-keystone.conf ，内容如下：

Listen 5000

Listen 35357

 

<VirtualHost *:5000>

   WSGIDaemonProcess keystone-public processes=5 threads=1 user=keystonegroup=xdkeystone display-name=%{GROUP}

   WSGIProcessGroup keystone-public

   WSGIScriptAlias / /usr/bin/keystone-wsgi-public

   WSGIApplicationGroup %{GLOBAL}

   WSGIPassAuthorization On

   <IfVersion >= 2.4>

     ErrorLogFormat "%{cu}t %M"

   </IfVersion>

    ErrorLog/var/log/apache2/keystone.log

    CustomLog/var/log/apache2/keystone_access.log combined

 

   <Directory /usr/bin>

       <IfVersion >= 2.4>

           Require all granted

       </IfVersion>

       <IfVersion < 2.4>

           Order allow,deny

           Allow from all

       </IfVersion>

   </Directory>

</VirtualHost>

 

<VirtualHost *:35357>

   WSGIDaemonProcess keystone-admin processes=5 threads=1 user=keystonegroup=keystone display-name=%{GROUP}

   WSGIProcessGroup keystone-admin

   WSGIScriptAlias / /usr/bin/keystone-wsgi-admin

   WSGIApplicationGroup %{GLOBAL}

   WSGIPassAuthorization On

   <IfVersion >= 2.4>

     ErrorLogFormat "%{cu}t %M"

   </IfVersion>

    ErrorLog/var/log/apache2/keystone.log

    CustomLog/var/log/apache2/keystone_access.log combined

 

   <Directory /usr/bin>

       <IfVersion >= 2.4>

           Require all granted

       </IfVersion>

       <IfVersion < 2.4>

            Order allow,deny

           Allow from all

       </IfVersion>

   </Directory>

</VirtualHost>

 

创建连接

ln -s /etc/apache2/sites-available/wsgi-keystone.conf/etc/apache2/sites-enabled

重启服务

service apache2 restart

删除文件

rm -f /var/lib/keystone/keystone.db

5.4、认证服务添加

控制节点执行。

export OS_TOKEN=123456

export OS_TOKEN=123456

export OS_URL=http://controller:35357/v3

export OS_IDENTITY_API_VERSION=3

 

openstack service create --name keystone--description "OpenStack Identity" identity

openstack endpoint create --region RegionOneidentity public http://controller:5000/v3

openstack endpoint create --region RegionOneidentity internal http://controller:5000/v3

openstack endpoint create --region RegionOneidentity admin http://controller:35357/v3

 

openstack domain create --description "DefaultDomain" default

 

openstack project create --domain default  --description "Admin Project" admin

openstack user create --domain default--password-prompt admin

openstack role create admin

openstack role add --project admin --user adminadmin              没有显示表格的

openstack project create --domain default   --description "Service Project"service

openstack project create --domain default  --description "Demo Project" demo

openstack user create --domain default  --password-prompt demo

openstack role create user

openstack role add --project demo --user demo user

5.5、验证

unset OS_TOKEN OS_URL

openstack --os-auth-url http://controller:35357/v3 --os-project-domain-namedefault --os-user-domain-name default  --os-project-nameadmin --os-username admin token issue

Password:

openstack --os-auth-urlhttp://controller:5000/v3  --os-project-domain-name default --os-user-domain-name default  --os-project-name demo --os-username demotoken issue

Password:

5.6、环境变量

 

创建admin-openrc ：

export OS_PROJECT_DOMAIN_NAME=default

export OS_USER_DOMAIN_NAME=default

export OS_PROJECT_NAME=admin

export OS_USERNAME=admin

export OS_PASSWORD=123456

export OS_AUTH_URL=http://controller:35357/v3

export OS_IDENTITY_API_VERSION=3

export OS_IMAGE_API_VERSION=2

创建demo-openrc：

export OS_PROJECT_DOMAIN_NAME=default

export OS_USER_DOMAIN_NAME=default

export OS_PROJECT_NAME=demo

export OS_USERNAME=demo

export OS_PASSWORD=123456

export OS_AUTH_URL=http://controller:5000/v3

export OS_IDENTITY_API_VERSION=3

export OS_IMAGE_API_VERSION=2

 

执行以下命令验证：

. admin-openrc

openstack token issue

六、存储服务

 6.1、控制节点

1、添加用户信息

执行以下命令：

. admin-openrc

openstack user create --domain default--password-prompt swift

openstack role add --project service --user swiftadmin

 

openstack service create --name swift --description"OpenStack Object Storage" object-store

openstack endpoint create --region RegionOneobject-store public http://controller:8080/v1/AUTH_%\(tenant_id\)s

 

openstack endpoint create --region RegionOne  object-store internalhttp://controller:8080/v1/AUTH_%\(tenant_id\)s

 

openstack endpoint create --region RegionOne   object-store admin http://controller:8080/v1

2、安装控制服务

apt-get install swift swift-proxypython-swiftclient  python-keystoneclient python-keystonemiddleware  memcached

mkdir /etc/swift

curl -o /etc/swift/proxy-server.confhttps://git.openstack.org/cgit/openstack/swift/plain/etc/proxy-server.conf-sample?h=stable/mitaka

编辑/etc/swift/proxy-server.conf

 

[DEFAULT]

bind_port = 8080

user = swift

swift_dir = /etc/swift

[pipeline:main]

pipeline = catch_errors gatekeeper healthcheckproxy-logging cache container_sync bulk ratelimit authtoken keystoneauthcontainer-quotas account-quotas slo dlo versioned_writes proxy-loggingproxy-server

[app:proxy-server]

use = egg:swift#proxy

account_autocreate = true

[filter:keystoneauth]

use = egg:swift#keystoneauth

operator_roles = admin,user

[filter:authtoken]

[filter:authtoken]

paste.filter_factory =keystonemiddleware.auth_token:filter_factory

auth_uri = http://controller:5000

auth_url = http://controller:35357

memcached_servers = controller:11211

auth_type = password

project_domain_name = default

user_domain_name = default

project_name = service

username = swift

password = 123456

delay_auth_decision = True

[filter:cache]

use = egg:swift#memcache

memcache_servers = controller:11211

6.2、存储节点

 

1、安装

apt-get install xfsprogs rsync

 

mkfs.xfs /dev/sdb   (这一部可能会报错，因为不同机子下的分区叫法不一样。。。你需要自己去看一下  fdisk -l )

mkdir -p /srv/node/sdb

编辑/etc/fstab，添加

/dev/sdb /srv/node/sdb xfsnoatime,nodiratime,nobarrier,logbufs=8 0 2

mount /srv/node/sdb

编辑/etc/rsyncd.conf

uid = swift

gid = swift

log file = /var/log/rsyncd.log

pid file = /var/run/rsyncd.pid

address = MANAGEMENT_INTERFACE_IP_ADDRESS   存储节点ip

 

[account]

max connections = 2

path = /srv/node/

read only = False

lock file = /var/lock/account.lock

 

[container]

max connections = 2

path = /srv/node/

read only = False

lock file = /var/lock/container.lock

 

[object]

max connections = 2

path = /srv/node/

read only = False

lock file = /var/lock/object.lock

 

更改/etc/default/rsync 

RSYNC_ENABLE=true

重启服务

service rsync start

 

apt-get install swift swift-account swift-containerswift-object

curl -o /etc/swift/account-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/account-server.conf-sample?h=stable/mitaka

# curl -o /etc/swift/container-server.confhttps://git.openstack.org/cgit/openstack/swift/plain/etc/container-server.conf-sample?h=stable/mitaka

# curl -o /etc/swift/object-server.confhttps://git.openstack.org/cgit/openstack/swift/plain/etc/object-server.conf-sample?h=stable/mitaka

 

编辑文件 /etc/swift/account-server.conf 

[DEFAULT]

bind_ip =MANAGEMENT_INTERFACE_IP_ADDRESS   存储节点ip

bind_port =6002

user = swift

swift_dir =/etc/swift

devices =/srv/node

mount_check =True

 [pipeline:main]

pipeline =healthcheck recon account-server

[filter:recon]

use =egg:swift#recon

recon_cache_path= /var/cache/swift

 

 

编辑文件/etc/swift/container-server.conf

[DEFAULT]

bind_ip = MANAGEMENT_INTERFACE_IP_ADDRESS

bind_port =6001

user = swift

swift_dir =/etc/swift

devices =/srv/node

mount_check =True

[pipeline:main]

pipeline =healthcheck recon container-server

[filter:recon]

use =egg:swift#recon

recon_cache_path= /var/cache/swift

 

 

编辑文件/etc/swift/object-server.conf

[DEFAULT]

bind_ip =MANAGEMENT_INTERFACE_IP_ADDRESS

bind_port =6000

user = swift

swift_dir =/etc/swift

devices =/srv/node

mount_check =True

[pipeline:main]

pipeline =healthcheck recon object-server

[filter:recon]

use =egg:swift#recon

recon_cache_path= /var/cache/swift

recon_lock_path= /var/lock

 

chown -Rswift:swift /srv/node

mkdir -p/var/cache/swift

chown -Rroot:swift /var/cache/swift

chmod -R 775/var/cache/swift

 

 

6.3、初始化

管理节点执行：

cd  /etc/swift

swift-ring-builderaccount.builder create 10 3 1

swift-ring-buildercontainer.builder create 10 3 1

swift-ring-builderobject.builder create 10 3 1

 

确定好后执行(以下IP是三个存储结点的IP)

swift-ring-builderaccount.builder   add --region 1 --zone 1--ip 10.82.25.128 --port 6002   --device sdb--weight 100

swift-ring-builderaccount.builder   add --region 1 --zone 1--ip 10.82.27.134 --port 6002   --device sdb--weight 100

swift-ring-builderaccount.builder   add --region 1 --zone 1--ip 10.82.13.135 --port 6002   --device sdb--weight 100

 

swift-ring-buildercontainer.builder  add --region 1 --zone1 --ip 10.82.25.128 --port 6001  --device sdb --weight 100

swift-ring-buildercontainer.builder  add --region 1 --zone1 --ip 10.82.27.134 --port 6001  --device sdb --weight 100

swift-ring-buildercontainer.builder  add --region 1 --zone1 --ip 10.82.13.135 --port 6001  --device sdb --weight 100

 

swift-ring-builderobject.builder  add --region 1 --zone 1--ip 10.82.25.128 --port 6000  --device sdb--weight 100

swift-ring-builderobject.builder  add --region 1 --zone 1--ip 10.82.27.134 --port 6000  --device sdb--weight 100

swift-ring-builderobject.builder  add --region 1 --zone 1--ip 10.82.13.135 --port 6000  --device sdb--weight 100

 

swift-ring-builderaccount.builder

swift-ring-buildercontainer.builder

swift-ring-builderobject.builder

 

swift-ring-builderaccount.builder rebalance

swift-ring-buildercontainer.builder rebalance

swift-ring-builderobject.builder rebalance

 将account.ring.gz, container.ring.gz,and object.ring.gz 拷贝至存储节点 /etc/swift 

 

curl -o/etc/swift/swift.conf    https://git.openstack.org/cgit/openstack/swift/plain/etc/swift.conf-sample?h=stable/mitaka

 

 

编辑 /etc/swift/swift.conf

[swift-hash]

...

swift_hash_path_suffix= HASH_PATH_SUFFIX

swift_hash_path_prefix= HASH_PATH_PREFIX

[storage-policy:0]

...

name =Policy-0

default = yes

将swift.conf 拷贝至 存储节点 /etc/swift 

 

所有节点chown -R root:swift /etc/swift

 

管理节点

# servicememcached restart

# serviceswift-proxy restart

 

存储节点

swift-initall start

如果不成功的话关闭一下防火墙

ufw disable

6.5、客户端测试

客户端执行

. demo-openrc，引入环境变量用户信息。

 swift stat，查看用户账户信息。

swift upload baisl_container1 baisl_file1：将根目录下文件baisl_file1，上传至云存储，baisl_container1文件夹中。baisl_container1不存在，自动创建。

swift list：查看已有的一层文件夹。

post: 创造一个文件夹，stat查看用户或者文件属性。upload/download/delete:上传/下载/删除数据。