How to Install GlusterFS

Prerequisites

• At least two machines, each of them has at least 1G memory.
• At least two disks, one for the base OS and one that we will use as a Gluster “brick”.

Installing Gluster

Notice: The following steps need run on each nodes.

1. Download the packages (modify URL for your specific release and architecture).

 http://download.gluster.org/pub/gluster/glusterfs/3.4/3.4.5/RHEL/epel-6.5/x86_64/

2. Download the dependency package

 ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/billcavalieri:/QEMU/RedHat_RHEL-6/x86_64/xfsprogs-3.1.1-4.1.x86_64.rpm

3. Install the packages

 # rpm -e openssl-1.0.0-27.el6_4.2.x86_64 --nodeps
 # rpm -ivh openssl-1.0.1e-16.el6_5.7.x86_64.rpm –nodeps
 # yum install xfsprogs-3.1.1-4.1.x86_64.rpm
 # yum install glusterfs-libs-3.5.2-1.el6.x86_64.rpm glusterfs-cli-3.5.2-1.el6.x86_64.rpm
 # yum install glusterfs-api-3.5.2-1.el6.x86_64.rpm glusterfs-3.5.2-1.el6.x86_64.rpm glusterfs-fuse-3.5.2-1.el6.x86_64.rpm glusterfs-server-3.5.2-1.el6.x86_64.rp

4. Start the services, and set to start at boot

 # /etc/init.d/glusterd start
 # chkconfig glusterd on

Pre-Configuring

Notice: the following steps need run on both nodes.

1. Check the current mode of SELinux.

 # getenforce

If the current mode is “enforcing” mode, need to change it to either “permissive” or “disabled” mode. Otherwise, skip this step.

 # setenforce  0

Change “/etc/selinux/config” and make “SELINUX=permissive” or “SELINUX=disabled” in it to make the SELinux changes permanent.

2. Enable server can communicate with each other.

3. Partition the disk, assuming that you are a brick at /dev/vdb.

3.1. fdisk /dev/vdb and create a single partition:

 # fdisk /dev/vdb
 Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
 Building a new DOS disklabel with disk identifier 0xa0399863.
 Changes will remain in memory only, until you decide to write them.
 After that, of course, the previous content won't be recoverable.
 Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
 WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
 switch off the mode (command 'c') and change display units to sectors (command 'u').
 Command (m for help):

3.2. Press n,then press Enter.The following information is displayed:

 Command action
    e   extended
    p   primary partition (1-4)

3.3. Press p and press Enter. The following information is displayed:

  Partition number (1-4):

3.4. Press 1 and press Enter. The following information is displayed:

 First cylinder (1-2080, default 1):

3.5. Press Enter and continue. The following information is displayed:

 Using default value 1
 Last cylinder, +cylinders or +size{K,M,G} (1-2080, default 2080):

3.6. Press Enter. The following information is displayed:

 Using default value 2080
     Command (m for help):

3.7. Press p to check and then press w to save.

 Command (m for help): p
 Disk /dev/vdb: 1073 MB, 1073741824 bytes
 16 heads, 63 sectors/track, 2080 cylinders
 Units = cylinders of 1008 * 512 = 516096 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytes
 Disk identifier: 0xa0399863

 Device Boot      Start         End      Blocks   Id  System
 /dev/vdb1               1        2080     1048288+  83  Linux
 Command (m for help): w

4. Format the partition:

 # mkfs.xfs -f -i size=512 -n size=8192 -d su=64k,sw=4 /dev/vdb1

5. Mount the partition:

 # mkdir -p /export/disk1 && mount -t xfs  -o  noatime,inode64 /dev/vdb1  /export/disk1 && mkdir -p /export/disk1/brick

6. Add en entry to /etc/fstab:

 # echo "/dev/vdb1 /export/disk1 xfs defaults,noatime,inode64 0 0"  >> /etc/fstab

Configuring Gluster Server

Choose a server to be your "primary" node. The following steps just only run in "primary" node. 1. Configure the trusted pool. Remember that the trusted pool is the term used to define a cluster of nodes in Gluster.

 # gluster peer probe <hostname_of_node2>
   peer probe: success.

2. Set up a replicated volume:

 # gluster volume create gv0 replica 2 <hostname_of_node1>:/export/disk1/brick <hostname_of_node2>:/export/disk1/brick

3. Check the volume we created

 # gluster volume info

And you should see results similar to the following:

 Volume Name: gv0
 Type: Replicate
 Volume ID: 8bc3e96b-a1b6-457d-8f7a-a91d1d4dc019
 Status: Created
 Number of Bricks: 1 x 2 = 2
 Transport-type: tcp
 Bricks:
 Brick1: node01:/export/disk1/brick
 Brick2: node02:/export/disk1/brick

4. Start the volume

 # gluster volume start gv0

5. Set the volume options.(Option)

 # gluster volume set gv0 performance.cache-size 16GB
 # gluster volume set gv0 performance.io-thread-count 64
 # gluster volume set gv0 performance.cache-refresh-timeout 60
 # gluster volume set gv0 performance.client-io-threads on
 # gluster volume set gv0 eager-lock on

Configuring Gluster Client

Notice: the following steps need run on both nodes.

1. Add the FUSE loadable kernel module(LKM) to the Linux kernel:

 # modprobe fuse

2. Verify that the FUSE module is loaded:

 # dmesg |grep fuse
 fuse init (API version 7.13)

3. Create the mount directory:

 # mkdir /bmsccontents

4. Mount glusterfs.

 # mount -t glusterfs <hostname_node_1>:/gv0 /bmsccontents

FAQ

1. Gluster can’t start if the old info exists.

 # cd /var/lib/glusterd/
 # rm -rf glusterd.info
 # rm -rf vols/*
 # cd /var/lib/glusterd/glustershd
 # rm -rf glustershd-server.vol
 # /etc/init.d/glusterd restart

2. Stop & delete volume

 # gluster volume stop gv0 force
 # gluster volume delete gv0

3. Expanding Volumes
1. On the first server in the cluster, probe the server to which you want to add the new brick using the following command:

 # gluster peer probe HOSTNAME

2. Add the brick using the following command:

 # gluster volume add-brick gv0  <hostname_of_node>:/export/disk1/brick

3. Check the volume information using the following command:

 # gluster volume info

4. Re-balance the volume to ensure that all files are visible at the mount point.

 # gluster volume rebalance gv0 start

4. Path or a prefix of it is already part of a volume
1. For the directory (or any parent directories) that was formerly part of a volume. Don't worry if it says that the attribute does not exist. As long as it doesn't exist, you're in good shape.

 # setfattr -x trusted.glusterfs.volume-id $brick_path
 # setfattr -x trusted.gfid $brick_path
 # rm -rf $brick_path/.glusterfs

2. Finally, restart glusterd to ensure it's not "remembering" the old bricks.