GlusterFS: {path} or a prefix of it is already part of a volume
Starting with GlusterFS 3.3, one change has been the check to see if a directory (or any of it's ancestors) is already part of a volume. This is causing many support questions in #gluster.
This was implemented because if you remove a brick from a volume and continue to use the volume, you can get file into a state where re-adding a former brick can cause all sort of problems, many of which can result in data loss.
If you're going to reuse a brick, make sure you know what you're doing.
The Solution
For the directory (or any parent directories) that was formerly part of a volume, simply:
setfattr -x trusted.glusterfs.volume-id $brick_path setfattr -x trusted.gfid $brick_path rm -rf $brick_path/.glusterfs
Don't worry if it says that the attribute does not exist. As long as it doesn't exist, you're in good shape.
Finally, restart glusterd to ensure it's not "remembering" the old bricks.
See the bugzilla entry for more details and see Jeff Darcy's article for more information about how GlusterFS uses extended attributes.
[root@test glusterfs]# gluster volume create hello replica 3 test.144:/data0/glusterfs test.145:/data0/glusterfs test.146:/data0/glusterfs
volume create: hello: success: please start the volume to access data
[root@test glusterfs]# gluster volume delete hello
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: hello: success
[root@test glusterfs]# gluster volume create gfs replica 3 test.144:/data0/glusterfs test.145:/data0/glusterfs test.146:/data0/glusterfs
volume create: gfs: failed: Staging failed on test.144. Error: /data0/glusterfs is already part of a volume
Staging failed on test.146. Error: /data0/glusterfs is already part of a volume
Staging failed on test.145. Error: /data0/glusterfs is already part of a volume
[root@test glusterfs]# ssh test.144 'setfattr -x trusted.glusterfs.volume-id /data0/glusterfs/ && setfattr -x trusted.gfid /data0/glusterfs/ && rm -rf /data0/glusterfs/..glusterfs'
setfattr: /data0/glusterfs/: No such attribute
[root@test glusterfs]# ssh test.145 'setfattr -x trusted.glusterfs.volume-id /data0/glusterfs/ && setfattr -x trusted.gfid /data0/glusterfs/ && rm -rf /data0/glusterfs/..glusterfs'
setfattr: /data0/glusterfs/: No such attribute
[root@test glusterfs]# ssh test.146 'setfattr -x trusted.glusterfs.volume-id /data0/glusterfs/ && setfattr -x trusted.gfid /data0/glusterfs/ && rm -rf /data0/glusterfs/..glusterfs'
setfattr: /data0/glusterfs/: No such attribute
[root@test glusterfs]#
[root@test glusterfs]# gluster volume create gfs replica 3 test.144:/data0/glusterfs test.145:/data0/glusterfs test.146:/data0/glusterfs
volume create: gfs: success: please start the volume to access data
[root@test glusterfs]#