应用挂载beegfs指定目录_BeeGFS元数据组织方式和路径寻址分析

元数据根节点的确定和获取

管理节点处理元数据节点的心跳信息时,如果发现目前没有Root节点,则会在已经注册的节点中选择ID最小的那个注册为元数据Root节点(这些信息最后都会保存在磁盘上):

// fhgfs_mgmtd\source\components\HeartbeatManager.cpp

/**

* @param rootIDHint empty string to auto-define root or a nodeID that is assumed to be the root

* @return true if a new root node has been defined

*/

bool HeartbeatManager::initRootNode(NumNodeID rootIDHint, bool rootIsBuddyMirrored)

{

// be careful: this method is also called from other threads

// note: after this method, the root node might still be undefined (this is normal)

bool setRootRes = false;

if( (rootIDHint != 0) || (metaNodes->getSize()) )

{ // check whether root has already been set

if(rootIDHint == 0)

rootIDHint = metaNodes->getLowestNodeID();

// set root to lowest ID (if no other root was set yet)

setRootRes = metaNodes->setRootNodeNumID(rootIDHint, false, rootIsBuddyMirrored);

if(setRootRes)

{ // new root set

log.log(Log_CRITICAL, "New root directory metadata node: " +

Program::getApp()->getMetaNodes()->getNodeIDWithTypeStr(rootIDHint) );

notifyAsyncAddedNode("", rootIDHint, NODETYPE_Meta); /* (real string ID will

be retrieved by notifier before sending the heartbeat) */

}

}

return setRootRes;

}

客户端在处理与管理节点的心跳信息时,会从中获取并设置元数据Root节点信息:

// fhgfs_client_module\source\common\net\message\nodes\HeartbeatMsgEx.c

/**

* Handles the contained root information.

*/

void __HeartbeatMsgEx_processIncomingRoot(HeartbeatMsgEx* this, App* app)

{

Logger* log = App_getLogger(app);

const char* logContext = "Heartbeat incoming (root)";

NodeStoreEx* metaNodes;

bool setRootRes;

NodeOrGroup rootOwner = this->rootIsBuddyMirrored

? NodeOrGroup_fromGroup(this->rootNumID.value)

: NodeOrGroup_fromNode(this->rootNumID);

NumNodeID rootNumID = HeartbeatMsgEx_getRootNumID(this);

// check whether root info is defined

if( (HeartbeatMsgEx_getNodeType(this) != NODETYPE_Meta) || (NumNodeID_isZero(&rootNumID)))

return;

// try to apply the contained root info

metaNodes = App_getMetaNodes(app);

setRootRes = NodeStoreEx_setRootOwner(metaNodes, rootOwner, false);

if(setRootRes)

{ // found the very first root

Logger_logFormatted(log, Log_CRITICAL, logContext, "Root (by Heartbeat): %hu",

HeartbeatMsgEx_getRootNumID(this) );

}

}

元数据根目录的确定和获取

客户端在挂载文件系统,初始化超级块时,会向元数据Root节点获取根目录的DirEntry信息:

// fhgfs_client_module\source\filesystem\FhgfsOpsSuper.c

/**

* Fill the file system superblock (vfs object)

*/

int FhgfsOps_fillSuper(struct super_block* sb, void* rawMountOptions, int silent)

{

...

// init root inode

memset(&kstat, 0, sizeof(struct kstat) );

kstat.ino = BEEGFS_INODE_ROOT_INO;

kstat.mode = S_IFDIR | 0777; // allow access for everyone

kstat.atime = kstat.mtime = kstat.ctime = current_fs_time(sb);

kstat.uid = FhgfsCommon_getCurrentKernelUserID();

kstat.gid = FhgfsCommon_getCurrentKernelGroupID();

kstat.blksize = Config_getTuneInodeBlockSize(cfg);

kstat.nlink = 1;

// root entryInfo is always updated when someone asks for it (so we just set dummy values here)

EntryInfo_init(&entryInfo, NodeOrGroup_fromGroup(0), StringTk_strDup(""), StringTk_strDup(""),

StringTk_strDup(""), DirEntryType_DIRECTORY, 0);

rootInode = __FhgfsOps_newInode(sb, &kstat, 0, &entryInfo, &iSizeHints);

if(!rootInode || IS_ERR(rootInode) )

{

__FhgfsOps_destructFsInfo(sb);

return IS_ERR(rootInode) ? PTR_ERR(rootInode) : -ENOMEM;

}

rootDentry = d_make_root(rootInode);

if(!rootDentry)

{

__FhgfsOps_destructFsInfo(sb);

return -ENOMEM;

}

...

}

// fhgfs_client_module\source\common\toolkit\MetadataTk.c

/**

* @param outEntryInfo contained values will be kalloced (on success) and need to be kfreed with

* FhgfsInode_freeEntryMinInfoVals() later.

*/

bool MetadataTk_getRootEntryInfoCopy(App* app, EntryInfo* outEntryInfo)

{

NodeStoreEx* nodes = App_getMetaNodes(app);

NodeOrGroup rootOwner = NodeStoreEx_getRootOwner(nodes);

const char* parentEntryID = StringTk_strDup("");

const char* entryID = StringTk_strDup(META_ROOTDIR_ID_STR);

const char* dirName = StringTk_strDup("");

DirEntryType entryType = (DirEntryType) DirEntryType_DIRECTORY;

/* Even if rootOwner is invalid, we still init outEntryInfo and malloc as FhGFS

* policy says that kfree(NULL) is not allowed (the kernel allows it). */

EntryInfo_init(outEntryInfo, rootOwner, parentEntryID, entryID, dirName, entryType, 0);

return NodeOrGroup_valid(rootOwner);

}

元数据节点在初始化时,会先后在BuddyMirror和非BuddyMirror目录查找ID名为root的根目录DirEntry文件:

// fhgfs_common\source\common\storage\Metadata.h

#define META_ROOTDIR_ID_STR "root" /* initial file system entry point */

// fhgfs_meta\source\app\App.cpp

void App::initRootDir(NumNodeID localNodeNumID)

{

// try to load root dir from disk (through metaStore) or create a new one

this->metaStore = new MetaStore();

// try to reference root directory with buddy mirroring

rootDir = this->metaStore->referenceDir(META_ROOTDIR_ID_STR, true, true);

// if that didn't work try to reference non-buddy-mirrored root dir

if (!rootDir)

{

rootDir = this->metaStore->referenceDir(META_ROOTDIR_ID_STR, false, true);

}

if(rootDir)

{ // loading succeeded (either with or without mirroring => init rootNodeID

this->log->log(Log_NOTICE, "Root directory loaded.");

NumNodeID rootDirOwner = rootDir->getOwnerNodeID();

bool rootIsBuddyMirrored = rootDir->getIsBuddyMirrored();

// try to set rootDirOwner as root node

if((rootDirOwner != 0) && metaNodes->setRootNodeNumID(rootDirOwner, false,

rootIsBuddyMirrored) )

{ // new root node accepted (check if rootNode is localNode)

NumNodeID primaryRootDirOwner;

if (rootIsBuddyMirrored)

primaryRootDirOwner = NumNodeID(

metaBuddyGroupMapper->getPrimaryTargetID(rootDirOwner.val() ) );

else

primaryRootDirOwner = rootDirOwner;

if(localNodeNumID == primaryRootDirOwner)

{

log->log(Log_CRITICAL, "I got root (by possession of root directory)");

if (rootIsBuddyMirrored)

log->log(Log_CRITICAL, "Root directory is mirrored");

}

else

log->log(Log_CRITICAL,

"Root metadata server (by possession of root directory): " + rootDirOwner.str());

}

}

else

{ // failed to load root directory => create a new rootDir (not mirrored)

this->log->log(Log_CRITICAL,

"This appears to be a new storage directory. Creating a new root dir.");

UInt16Vector stripeTargets;

unsigned defaultChunkSize = this->cfg->getTuneDefaultChunkSize();

unsigned defaultNumStripeTargets = this->cfg->getTuneDefaultNumStripeTargets();

Raid0Pattern stripePattern(defaultChunkSize, stripeTargets, defaultNumStripeTargets);

DirInode newRootDir(META_ROOTDIR_ID_STR,

S_IFDIR | S_IRWXU | S_IRWXG | S_IRWXO, 0, 0, NumNodeID(), stripePattern, false);

this->metaStore->makeDirInode(newRootDir);

this->rootDir = this->metaStore->referenceDir(META_ROOTDIR_ID_STR, false, true);

if(!this->rootDir)

{ // error

this->log->logErr("Failed to store root directory. Unable to proceed.");

throw InvalidConfigException("Failed to store root directory");

}

}

}

元数据ID文件命名规则

每个目录或者文件都会有一个ID文件,而这个ID文件的名字组成结构为--:

// fhgfs_common\source\common\toolkit\StorageTk.h

class StorageTk

{

/**

* Generate ID for new fs entry (i.e. file or dir).

*/

static std::string generateFileID(const NumNodeID localNodeID)

{

/* note: we assume here that the clock doesn't jump backwards between restarts of

the daemon (and that there always is at least one second between restarts) and that we

don't need more than 2^32 IDs per second (sustained) */

uint64_t nextID = idCounter.increase();

// note on idCounter value: high 32bits are timestamp, low 32bits are sequential counter

/* note on switching time/counter in string representation: having the timestamp first is

bad for strcmp() and such things, which the underlying fs might need to do - because in

that order, the first characters of entryIDs/filenames would always be similar. */

uint32_t counterPart = (uint32_t) nextID;

uint32_t timestampPart = (uint32_t) (nextID >> STORAGETK_FILEID_TIMESTAMP_SHIFTBITS);

return StringTk::uintToHexStr(counterPart) + "-" + StringTk::uintToHexStr(timestampPart)

+ "-" + localNodeID.strHex();

}

}

元数据组织结构测试

清空和查看数据目录

$ rm /mnt/beegfs/* -rf

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f

/beegfs_meta/meta1/inodes/35/5B/disposal

/beegfs_meta/meta1/inodes/38/51/root

/beegfs_meta/meta2/inodes/35/5B/disposal

/beegfs_meta/meta2/inodes/38/51/root

/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal

/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

创建和查看空文件

$ touch /mnt/beegfs/test001

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f

/beegfs_meta/meta1/inodes/35/5B/disposal

/beegfs_meta/meta1/inodes/38/51/root

/beegfs_meta/meta2/inodes/35/5B/disposal

/beegfs_meta/meta2/inodes/38/51/root

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2

/beegfs_meta/meta1/dentries/38/51/root/test001

/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal

/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

$ ll /beegfs_meta/meta1/dentries/38/51/root/test001 -i

3804241 -rw-r--r-- 2 root root 0 4月 26 10:44 /beegfs_meta/meta1/dentries/38/51/root/test001

$ ll /beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2 -i

3804241 -rw-r--r-- 2 root root 0 4月 26 10:44 /beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2

$ touch /mnt/beegfs/test002

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f

/beegfs_meta/meta1/inodes/35/5B/disposal

/beegfs_meta/meta1/inodes/38/51/root

/beegfs_meta/meta2/inodes/35/5B/disposal

/beegfs_meta/meta2/inodes/38/51/root

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2

/beegfs_meta/meta1/dentries/38/51/root/test002

/beegfs_meta/meta1/dentries/38/51/root/test001

/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal

/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

创建和查看一级目录

$ mkdir /mnt/beegfs/dir001

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f

/beegfs_meta/meta1/inodes/35/5B/disposal

/beegfs_meta/meta1/inodes/38/51/root

/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

/beegfs_meta/meta2/inodes/35/5B/disposal

/beegfs_meta/meta2/inodes/38/51/root

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2

/beegfs_meta/meta1/dentries/38/51/root/test002

/beegfs_meta/meta1/dentries/38/51/root/dir001

/beegfs_meta/meta1/dentries/38/51/root/test001

/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal

/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

$ ll -i /beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

6293289 -rw-r--r-- 1 root root 0 4月 26 10:45 /beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

$ ll -i /beegfs_meta/meta1/dentries/38/51/root/dir001

3804243 -rw-r--r-- 1 root root 0 4月 26 10:45 /beegfs_meta/meta1/dentries/38/51/root/dir001

$ getfattr -n user.fhgfs /beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

getfattr: Removing leading '/' from absolute path names

# file: beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

user.fhgfs=0sBAMKAAIAAAAAAAAA7UEAANlwwlwAAAAA2XDCXAAAAABGx+RcAAAAAEbH5FwAAAAAAAAAAAAAAAABAAAADAAAADAtNUNDMjcwRDQtMgAAAAAEAAAAcm9vdAAAAAACAAAAAgAAABoAAAABAAAAAAAIAAEABAAAAAgAAAAAAAAA

$ getfattr -n user.fhgfs /beegfs_meta/meta1/dentries/38/51/root/dir001

getfattr: Removing leading '/' from absolute path names

# file: beegfs_meta/meta1/dentries/38/51/root/dir001

user.fhgfs=0sAgMQAAEAAAAMAAAAMC01Q0MyNzBENC0yAAAAAAIAAAA=

创建和查看非空文件

$ dd if=/dev/zero of=/mnt/beegfs/test00 bs=1K count=1

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f

/beegfs_meta/meta1/inodes/35/5B/disposal

/beegfs_meta/meta1/inodes/38/51/root

/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

/beegfs_meta/meta2/inodes/35/5B/disposal

/beegfs_meta/meta2/inodes/38/51/root

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2

/beegfs_meta/meta1/dentries/38/51/root/test002

/beegfs_meta/meta1/dentries/38/51/root/test00

/beegfs_meta/meta1/dentries/38/51/root/dir001

/beegfs_meta/meta1/dentries/38/51/root/test001

/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal

/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2

$ dd if=/dev/zero of=/mnt/beegfs/test00 bs=1K count=1024

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f

/beegfs_meta/meta1/inodes/35/5B/disposal

/beegfs_meta/meta1/inodes/38/51/root

/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

/beegfs_meta/meta2/inodes/35/5B/disposal

/beegfs_meta/meta2/inodes/38/51/root

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2

/beegfs_meta/meta1/dentries/38/51/root/test002

/beegfs_meta/meta1/dentries/38/51/root/test00

/beegfs_meta/meta1/dentries/38/51/root/dir001

/beegfs_meta/meta1/dentries/38/51/root/test001

/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal

/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2

$ dd if=/dev/zero of=/mnt/beegfs/test00 bs=1K count=4096

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f

/beegfs_meta/meta1/inodes/35/5B/disposal

/beegfs_meta/meta1/inodes/38/51/root

/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

/beegfs_meta/meta2/inodes/35/5B/disposal

/beegfs_meta/meta2/inodes/38/51/root

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2

/beegfs_meta/meta1/dentries/38/51/root/test002

/beegfs_meta/meta1/dentries/38/51/root/test00

/beegfs_meta/meta1/dentries/38/51/root/dir001

/beegfs_meta/meta1/dentries/38/51/root/test001

/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal

/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2

$ dd if=/dev/zero of=/mnt/beegfs/test00 bs=1K count=16384

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f

/beegfs_meta/meta1/inodes/35/5B/disposal

/beegfs_meta/meta1/inodes/38/51/root

/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

/beegfs_meta/meta2/inodes/35/5B/disposal

/beegfs_meta/meta2/inodes/38/51/root

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2

/beegfs_meta/meta1/dentries/38/51/root/test002

/beegfs_meta/meta1/dentries/38/51/root/test00

/beegfs_meta/meta1/dentries/38/51/root/dir001

/beegfs_meta/meta1/dentries/38/51/root/test001

/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal

/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2

$ find /beegfs_data/data[123]/{buddymir,chunks} -type f | xargs -i ls -lh {}

-rw-rw-rw- 1 root root 5.0M 5月 5 16:44 /beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2

-rw-rw-rw- 1 root root 5.5M 5月 5 16:44 /beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2

-rw-rw-rw- 1 root root 5.5M 5月 5 16:44 /beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2

$ dd if=/dev/zero of=/mnt/beegfs/test00 bs=1K count=65536

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f

/beegfs_meta/meta1/inodes/35/5B/disposal

/beegfs_meta/meta1/inodes/38/51/root

/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

/beegfs_meta/meta2/inodes/35/5B/disposal

/beegfs_meta/meta2/inodes/38/51/root

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2

/beegfs_meta/meta1/dentries/38/51/root/test002

/beegfs_meta/meta1/dentries/38/51/root/test00

/beegfs_meta/meta1/dentries/38/51/root/dir001

/beegfs_meta/meta1/dentries/38/51/root/test001

/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal

/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2

$ find /beegfs_data/data[123]/{buddymir,chunks} -type f | xargs -i ls -lh {}

-rw-rw-rw- 1 root root 21M 5月 5 16:46 /beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2

-rw-rw-rw- 1 root root 22M 5月 5 16:46 /beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2

-rw-rw-rw- 1 root root 22M 5月 5 16:46 /beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2

创建和查看多级目录

$ touch /mnt/beegfs/dir001/test005

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f

/beegfs_meta/meta1/inodes/35/5B/disposal

/beegfs_meta/meta1/inodes/38/51/root

/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

/beegfs_meta/meta2/inodes/35/5B/disposal

/beegfs_meta/meta2/inodes/38/51/root

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2

/beegfs_meta/meta1/dentries/38/51/root/test002

/beegfs_meta/meta1/dentries/38/51/root/test00

/beegfs_meta/meta1/dentries/38/51/root/dir001

/beegfs_meta/meta1/dentries/38/51/root/test001

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005

/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal

/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2

$ mkdir /mnt/beegfs/dir002

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f

/beegfs_meta/meta1/inodes/35/5B/disposal

/beegfs_meta/meta1/inodes/38/51/root

/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2

/beegfs_meta/meta2/inodes/35/5B/disposal

/beegfs_meta/meta2/inodes/38/51/root

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2

/beegfs_meta/meta1/dentries/38/51/root/test002

/beegfs_meta/meta1/dentries/38/51/root/dir002

/beegfs_meta/meta1/dentries/38/51/root/test00

/beegfs_meta/meta1/dentries/38/51/root/dir001

/beegfs_meta/meta1/dentries/38/51/root/test001

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005

/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal

/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2

$ mkdir /mnt/beegfs/dir001/dir003

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f

/beegfs_meta/meta1/inodes/35/5B/disposal

/beegfs_meta/meta1/inodes/38/51/root

/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2

/beegfs_meta/meta2/inodes/35/5B/disposal

/beegfs_meta/meta2/inodes/38/51/root

/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2

/beegfs_meta/meta1/dentries/38/51/root/test002

/beegfs_meta/meta1/dentries/38/51/root/dir002

/beegfs_meta/meta1/dentries/38/51/root/test00

/beegfs_meta/meta1/dentries/38/51/root/dir001

/beegfs_meta/meta1/dentries/38/51/root/test001

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003

/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal

/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2

$ mkdir /mnt/beegfs/dir001/dir003/dir004

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f

/beegfs_meta/meta1/inodes/35/5B/disposal

/beegfs_meta/meta1/inodes/38/51/root

/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2

/beegfs_meta/meta2/inodes/35/5B/disposal

/beegfs_meta/meta2/inodes/38/51/root

/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2

/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2

/beegfs_meta/meta1/dentries/38/51/root/test002

/beegfs_meta/meta1/dentries/38/51/root/dir002

/beegfs_meta/meta1/dentries/38/51/root/test00

/beegfs_meta/meta1/dentries/38/51/root/dir001

/beegfs_meta/meta1/dentries/38/51/root/test001

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003

/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004

/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal

/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2

$ mkdir /mnt/beegfs/dir001/dir003/dir004/dir005

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f

/beegfs_meta/meta1/inodes/35/5B/disposal

/beegfs_meta/meta1/inodes/38/51/root

/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

/beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5

/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2

/beegfs_meta/meta2/inodes/35/5B/disposal

/beegfs_meta/meta2/inodes/38/51/root

/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2

/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2

/beegfs_meta/meta1/dentries/38/51/root/test002

/beegfs_meta/meta1/dentries/38/51/root/dir002

/beegfs_meta/meta1/dentries/38/51/root/test00

/beegfs_meta/meta1/dentries/38/51/root/dir001

/beegfs_meta/meta1/dentries/38/51/root/test001

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003

/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004

/beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005

/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal

/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2

$ touch /mnt/beegfs/dir001/dir003/dir004/dir005/test006

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f

/beegfs_meta/meta1/inodes/35/5B/disposal

/beegfs_meta/meta1/inodes/38/51/root

/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

/beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5

/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2

/beegfs_meta/meta2/inodes/35/5B/disposal

/beegfs_meta/meta2/inodes/38/51/root

/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2

/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2

/beegfs_meta/meta1/dentries/38/51/root/test002

/beegfs_meta/meta1/dentries/38/51/root/dir002

/beegfs_meta/meta1/dentries/38/51/root/test00

/beegfs_meta/meta1/dentries/38/51/root/dir001

/beegfs_meta/meta1/dentries/38/51/root/test001

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003

/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/#fSiDs#/0-5CDA6D39-2

/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006

/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004

/beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005

/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal

/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2

$ mkdir /mnt/beegfs/dir001/dir003/dir004/dir005/dir006

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f

/beegfs_meta/meta1/inodes/35/5B/disposal

/beegfs_meta/meta1/inodes/38/51/root

/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

/beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5

/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2

/beegfs_meta/meta2/inodes/35/5B/disposal

/beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2

/beegfs_meta/meta2/inodes/38/51/root

/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2

/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2

/beegfs_meta/meta1/dentries/38/51/root/test002

/beegfs_meta/meta1/dentries/38/51/root/dir002

/beegfs_meta/meta1/dentries/38/51/root/test00

/beegfs_meta/meta1/dentries/38/51/root/dir001

/beegfs_meta/meta1/dentries/38/51/root/test001

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003

/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/#fSiDs#/0-5CDA6D39-2

/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/dir006

/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006

/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004

/beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005

/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal

/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2

$ mkdir /mnt/beegfs/dir001/dir003/dir004/dir005/dir006/dir007

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f

/beegfs_meta/meta1/inodes/35/5B/disposal

/beegfs_meta/meta1/inodes/38/51/root

/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

/beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5

/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2

/beegfs_meta/meta2/inodes/35/5B/disposal

/beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2

/beegfs_meta/meta2/inodes/38/51/root

/beegfs_meta/meta2/inodes/67/63/0-5CEFCA37-5

/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2

/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5

/beegfs_meta/meta2/inodes/78/78/0-5CE4C714-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2

/beegfs_meta/meta1/dentries/38/51/root/test002

/beegfs_meta/meta1/dentries/38/51/root/dir002

/beegfs_meta/meta1/dentries/38/51/root/test00

/beegfs_meta/meta1/dentries/38/51/root/dir001

/beegfs_meta/meta1/dentries/38/51/root/test001

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir010

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003

/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/#fSiDs#/0-5CDA6D39-2

/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/dir006

/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006

/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/dir007

/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004

/beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005

/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal

/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2

$ touch /mnt/beegfs/dir001/dir003/dir004/dir005/dir006/test008

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f

/beegfs_meta/meta1/inodes/35/5B/disposal

/beegfs_meta/meta1/inodes/38/51/root

/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

/beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5

/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2

/beegfs_meta/meta2/inodes/35/5B/disposal

/beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2

/beegfs_meta/meta2/inodes/38/51/root

/beegfs_meta/meta2/inodes/67/63/0-5CEFCA37-5

/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2

/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5

/beegfs_meta/meta2/inodes/78/78/0-5CE4C714-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2

/beegfs_meta/meta1/dentries/38/51/root/test002

/beegfs_meta/meta1/dentries/38/51/root/dir002

/beegfs_meta/meta1/dentries/38/51/root/test00

/beegfs_meta/meta1/dentries/38/51/root/dir001

/beegfs_meta/meta1/dentries/38/51/root/test001

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir010

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003

/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/#fSiDs#/0-5CDA6D39-2

/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/dir006

/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006

/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/#fSiDs#/0-5CEFCC92-5

/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/dir007

/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/test008

/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004

/beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005

/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal

/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2

$ touch /mnt/beegfs/dir001/dir003/dir004/dir005/dir006/dir007/test009

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f

/beegfs_meta/meta1/inodes/35/5B/disposal

/beegfs_meta/meta1/inodes/38/51/root

/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

/beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5

/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2

/beegfs_meta/meta2/inodes/35/5B/disposal

/beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2

/beegfs_meta/meta2/inodes/38/51/root

/beegfs_meta/meta2/inodes/67/63/0-5CEFCA37-5

/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2

/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5

/beegfs_meta/meta2/inodes/78/78/0-5CE4C714-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2

/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2

/beegfs_meta/meta1/dentries/38/51/root/test002

/beegfs_meta/meta1/dentries/38/51/root/dir002

/beegfs_meta/meta1/dentries/38/51/root/test00

/beegfs_meta/meta1/dentries/38/51/root/dir001

/beegfs_meta/meta1/dentries/38/51/root/test001

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir010

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003

/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/#fSiDs#/0-5CDA6D39-2

/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/dir006

/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006

/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/#fSiDs#/0-5CEFCC92-5

/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/dir007

/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/test008

/beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/test009

/beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/#fSiDs#/0-5CEFCD46-5

/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004

/beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005

/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal

/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2

/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f -exec ls -l {} \;

-rw-r--r-- 1 root root 0 4月 16 17:48 /beegfs_meta/meta1/inodes/35/5B/disposal

-rw-r--r-- 1 root root 0 4月 16 17:48 /beegfs_meta/meta1/inodes/38/51/root

-rw-r--r-- 1 root root 0 4月 26 10:45 /beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

-rw-r--r-- 1 root root 0 5月 14 15:23 /beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5

-rw-r--r-- 1 root root 0 5月 14 15:12 /beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2

-rw-r--r-- 1 root root 0 4月 16 21:10 /beegfs_meta/meta2/inodes/35/5B/disposal

-rw-r--r-- 1 root root 0 5月 14 15:34 /beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2

-rw-r--r-- 1 root root 0 4月 16 21:10 /beegfs_meta/meta2/inodes/38/51/root

-rw-r--r-- 1 root root 0 5月 14 15:14 /beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2

-rw-r--r-- 1 root root 0 5月 14 15:15 /beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5

-rw-r--r-- 2 root root 0 4月 26 10:45 /beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2

-rw-r--r-- 2 root root 0 4月 26 10:44 /beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2

-rw-r--r-- 2 root root 0 4月 26 10:47 /beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2

-rw-r--r-- 2 root root 0 4月 26 10:45 /beegfs_meta/meta1/dentries/38/51/root/test002

-rw-r--r-- 1 root root 0 5月 14 15:12 /beegfs_meta/meta1/dentries/38/51/root/dir002

-rw-r--r-- 2 root root 0 4月 26 10:47 /beegfs_meta/meta1/dentries/38/51/root/test00

-rw-r--r-- 1 root root 0 4月 26 10:45 /beegfs_meta/meta1/dentries/38/51/root/dir001

-rw-r--r-- 2 root root 0 4月 26 10:44 /beegfs_meta/meta1/dentries/38/51/root/test001

-rw-r--r-- 2 root root 0 5月 5 16:47 /beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2

-rw-r--r-- 2 root root 0 5月 5 16:47 /beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005

-rw-r--r-- 1 root root 0 5月 14 15:14 /beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003

-rw-r--r-- 2 root root 0 5月 14 15:25 /beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/#fSiDs#/0-5CDA6D39-2

-rw-r--r-- 1 root root 0 5月 14 15:34 /beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/dir006

-rw-r--r-- 2 root root 0 5月 14 15:25 /beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006

-rw-r--r-- 1 root root 0 5月 14 15:15 /beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004

-rw-r--r-- 1 root root 0 5月 14 15:23 /beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005

-rw-r--r-- 1 root root 0 4月 16 17:48 /beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal

-rw-r--r-- 1 root root 0 4月 16 21:10 /beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

-rw-rw-rw- 1 root root 22020096 5月 5 16:46 /beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2

-rw-rw-rw- 1 root root 22544384 5月 5 16:46 /beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2

-rw-rw-rw- 1 root root 22544384 5月 5 16:46 /beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2

测试总结分析

对于每个文件,在其dentries父目录的ID目录中有一个同名的DirEntry文件,以及#fSiDs#子目录中的以ID为名称的硬链接;

对于每个目录,在其dentries父目录的ID目录中有一个同名的DirEntry文件,以及inodes目录中的以ID为名称的DirInode文件;

如果一个目录和其父目录不在一个元数据节点,则其DirEntry文件在父目录所在节点,其DirInode文件在其被真正存储的节点;

无论是inodes还是dentry目录,里面都是有两级Hash目录,每一级有128个,根据DirEntry和DirInode的ID进行Hash。而ID文件是根据时间戳生成,所以理论上可以做到很好的均衡。而根目录的ID固定为root,所以总是在38/51这个Hash目录中。

/mnt/beegfs/dir001

/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

/beegfs_meta/meta1/dentries/38/51/root/dir001

/mnt/beegfs/dir002

/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2

/beegfs_meta/meta1/dentries/38/51/root/dir002

/mnt/beegfs/dir001/dir003

/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2

/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003

/mnt/beegfs/dir001/dir003/dir004

/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5

/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004

/mnt/beegfs/dir001/dir003/dir004/dir005

/beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5

/beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005

/mnt/beegfs/dir001/dir003/dir004/dir005/dir006

/beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2

/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006

$ ll -i /beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2

3279957 -rw-r--r-- 1 root root 0 5月 14 15:34 /beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2

$ ll -i /beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006

2362456 -rw-r--r-- 2 root root 0 5月 14 15:25 /beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006

/mnt/beegfs/dir001/dir003/dir004/dir005/dir006/test008

/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/#fSiDs#/0-5CEFCC92-5

/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/test008

/mnt/beegfs/dir001/dir003/dir004/dir005/dir006/dir007

/beegfs_meta/meta2/inodes/67/63/0-5CEFCA37-5

/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/dir007

/mnt/beegfs/dir001/dir003/dir004/dir005/dir006/dir007/test009

/beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/#fSiDs#/0-5CEFCD46-5

/beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/test009

$ ll -i /beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/#fSiDs#/0-5CEFCD46-5

1707090 -rw-r--r-- 2 root root 0 5月 30 20:32 /beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/#fSiDs#/0-5CEFCD46-5

$ ll -i /beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/test009

1707090 -rw-r--r-- 2 root root 0 5月 30 20:32 /beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/test009

多元数据节点的分配

在元数据节点每次创建目录时,如果有多个元数据节点,优先选择剩余空间容量多的:

// fhgfs_meta\source\net\message\storage\creating\MkDirMsgEx.cpp

std::unique_ptr<:responsestate> MkDirMsgEx::mkDirPrimary(ResponseContext& ctx)

{

...

metaCapacityPools->chooseStorageTargets(numDesiredTargets, minNumRequiredTargets,

&getPreferredNodes(), &newOwnerNodes);

...

}

/**

* @param numTargets number of desired targets

* @param minNumRequiredTargets the minimum number of targets the caller needs; ususally 1

* for RAID-0 striping and 2 for mirroring (so this parameter is intended to avoid problems when

* there is only 1 target left in the normal pool and the user has mirroring turned on).

* @param preferredTargets may be NULL

*/

void NodeCapacityPools::chooseStorageTargets(unsigned numTargets, unsigned minNumRequiredTargets,

const UInt16List* preferredTargets, UInt16Vector* outTargets)

{

SafeRWLock lock(&rwlock, SafeRWLock_READ);

if(!preferredTargets || preferredTargets->empty() )

{ // no preference => start with first pool that contains any targets

if(pools[CapacityPool_NORMAL].size() )

{

chooseStorageNodesNoPref(pools[CapacityPool_NORMAL],

numTargets, outTargets);

if(outTargets->size() >= minNumRequiredTargets)

goto unlock_and_exit;

}

/* note: no "else if" here, because we want to continue with next pool if we didn't find

enough targets for minNumRequiredTargets in previous pool */

if(pools[CapacityPool_LOW].size() )

{

chooseStorageNodesNoPref(pools[CapacityPool_LOW],

numTargets - outTargets->size(), outTargets);

if(outTargets->size() >= minNumRequiredTargets)

goto unlock_and_exit;

}

chooseStorageNodesNoPref(pools[CapacityPool_EMERGENCY],

numTargets, outTargets);

}

else

{

// caller has preferred targets, so we can't say a priori whether nodes will be found or not

// in a pool. our strategy here is to automatically allow non-preferred targets before

// touching the emergency pool.

std::set chosenTargets;

// try normal and low pool with preferred targets...

chooseStorageNodesWithPref(pools[CapacityPool_NORMAL], numTargets, preferredTargets, false,

outTargets, chosenTargets);

if(outTargets->size() >= minNumRequiredTargets)

goto unlock_and_exit;

chooseStorageNodesWithPref(pools[CapacityPool_LOW], numTargets - outTargets->size(),

preferredTargets, false, outTargets, chosenTargets);

if(!outTargets->empty() )

goto unlock_and_exit;

/* note: currently, we cannot just continue here with non-empty outTargets (even if

"outTargets->size() < minNumRequiredTargets"), because we would need a mechanism to exclude

the already chosen outTargets for that (e.g. like an inverted preferredTargets list). */

// no targets yet - allow non-preferred targets before using emergency pool...

chooseStorageNodesWithPref(pools[CapacityPool_NORMAL], numTargets, preferredTargets, true,

outTargets, chosenTargets);

if(outTargets->size() >= minNumRequiredTargets)

goto unlock_and_exit;

chooseStorageNodesWithPref(pools[CapacityPool_LOW], numTargets - outTargets->size(),

preferredTargets, true, outTargets, chosenTargets);

if(!outTargets->empty() )

goto unlock_and_exit;

/* still no targets available => we have to try the emergency pool (first with preference,

then without preference) */

chooseStorageNodesWithPref(pools[CapacityPool_EMERGENCY], numTargets, preferredTargets, false,

outTargets, chosenTargets);

if(!outTargets->empty() )

goto unlock_and_exit;

chooseStorageNodesWithPref(pools[CapacityPool_EMERGENCY], numTargets, preferredTargets, true,

outTargets, chosenTargets);

}

unlock_and_exit:

lock.unlock();

}

相关的配置选项和说明如下:

$ vi /etc/beegfs/beegfs-client.conf

tunePreferredMetaFile =

tunePreferredStorageFile =

# [tunePreferredMetaFile], [tunePreferredStorageFile]

# Path to a text file that contains the numeric IDs of preferred storage targets

# and metadata servers. These will be preferred when the client creates new file

# system entries. This is useful e.g. to take advantage of data locality in the

# case of multiple data centers. If unspecified, all available targets and

# servers will be used equally.

# Usage: One targetID per line for storage servers, one nodeID per line for

# metadata servers.

# Note: TargetIDs and nodeIDs can be queried with the beegfs-ctl tool.

# Default:

$ vi /etc/beegfs/beegfs-mgmtd.conf

tuneMetaDynamicPools = true

tuneMetaInodesLowLimit = 10M

tuneMetaInodesEmergencyLimit = 1M

tuneMetaSpaceLowLimit = 10G

tuneMetaSpaceEmergencyLimit = 3G

tuneStorageDynamicPools = true

tuneStorageInodesLowLimit = 10M

tuneStorageInodesEmergencyLimit = 1M

tuneStorageSpaceLowLimit = 1T

tuneStorageSpaceEmergencyLimit = 20G

# [tune{Meta,Storage}DynamicPools]

# Temporarily raise the Low/Emergency limits if the spread (difference in free

# capacity between the targets with the most and the least free space) becomes

# too large. This will move targets to a lower pool earlier if there are other

# targets with much more free capacity.

# [tune{Meta,Storage}{Space,Inodes}LowLimit]

# [tune{Meta,Storage}{Space,Inodes}EmergencyLimit]

# The free space pool thresholds. If a metadata or storage target is below a

# threshold, it will only be used to store new files and directories when no

# higher class targets are available (i.e. while there are targets in the

# "normal" pool, no targets from the "low" pool will be used.)

# Note: Preferred target settings of a client will be ignored if it helps to

# avoid using targets from the emergency class.

# Default: Space: Meta: 10G/3G; Storage: 512G/10G

# Inodes: Meta: 10M/1M; Storage: 10M/1M

# [tune{Meta,Storage}{Space,Inodes}NormalSpreadThreshold]

# [tune{Meta,Storage}{Space,Inodes}LowSpreadThreshold]

# [tune{Meta,Storage}{Space,Inodes}LowDynamicLimit]

# [tune{Meta,Storage}{Space,Inodes}EmergencyDynamicLimit]

# Only effective if tune{Meta,Storage}DynamicPools is enabled.

# Whenever the spread (see above) of the free capacities in the normal / low

# class of storage targets is above this threshold, the StorageLowLimit /

# StorageEmergencyLimit is temporarily raised to StorageLowDynamicLimit /

# StorageEmergencyDynamicLimit.

# When the {Normal,Low}SpreadThreshold values are set to 0, the value from the

# corresponding {Low,Emergency}Limit is used for the spread threshold.

# When the {Low,Emergency}DynamicLimits are set to 0, they are automatically

# assumed as two times the corresponding {Low,Emergency}(non-dynamic)Limit.

# Default: 0

寻址过程总结

管理节点在收到元数据节点的的心跳信息时,如果没有元数据Root节点,则选择ID较小的作为元数据Root节点;

客户端在Mount时,会初始化超级块,同时向管理节点询问元数据Root节点的ID,然后向元数据Root节点获取DirEntry元数据;

元数据节点初始化时,会先依次寻找buddymir/inodes/38/51/root,inodes/38/51/root来确定根目录的元数据;

根据Root节点中Root的目录的元数据信息中包含的OwnerID(确定目录所在节点)和EntryID(根据Hash确定目录路径),就可以找到该目录的元数据位置;

该目录所有文件的DirEntry文件中包含有文件的条带信息,以及数据分布的存储节点;

该目录所有子目录的DirEntry文件的元数据中包含子目录的OwnerID(即所在节点的NodeID)和EntryID,据此找到子目录的元数据目录,不断迭代,即可找到最终的文件。

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值