BeeGFS元資料組織方式和路徑定址分析

阿新 • • 發佈：2019-09-08

元資料根節點的確定和獲取

管理節點處理元資料節點的心跳資訊時，如果發現目前沒有Root節點，則會在已經註冊的節點中選擇ID最小的那個註冊為元資料Root節點（這些資訊最後都會儲存在磁碟上）：

// fhgfs_mgmtd\source\components\HeartbeatManager.cpp

/**
 * @param rootIDHint empty string to auto-define root or a nodeID that is assumed to be the root
 * @return true if a new root node has been defined
 */
bool HeartbeatManager::initRootNode(NumNodeID rootIDHint, bool rootIsBuddyMirrored)
{
   // be careful: this method is also called from other threads
   // note: after this method, the root node might still be undefined (this is normal)

   bool setRootRes = false;

   if( (rootIDHint != 0) || (metaNodes->getSize()) )
   { // check whether root has already been set

      if(rootIDHint == 0)
         rootIDHint = metaNodes->getLowestNodeID();

      // set root to lowest ID (if no other root was set yet)
      setRootRes = metaNodes->setRootNodeNumID(rootIDHint, false, rootIsBuddyMirrored);

      if(setRootRes)
      { // new root set
         log.log(Log_CRITICAL, "New root directory metadata node: " +
            Program::getApp()->getMetaNodes()->getNodeIDWithTypeStr(rootIDHint) );

         notifyAsyncAddedNode("", rootIDHint, NODETYPE_Meta); /* (real string ID will
            be retrieved by notifier before sending the heartbeat) */
      }
   }

   return setRootRes;
}

客戶端在處理與管理節點的心跳資訊時，會從中獲取並設定元資料Root節點資訊：

// fhgfs_client_module\source\common\net\message\nodes\HeartbeatMsgEx.c

/**
 * Handles the contained root information.
 */
void __HeartbeatMsgEx_processIncomingRoot(HeartbeatMsgEx* this, App* app)
{
   Logger* log = App_getLogger(app);
   const char* logContext = "Heartbeat incoming (root)";

   NodeStoreEx* metaNodes;
   bool setRootRes;
   NodeOrGroup rootOwner = this->rootIsBuddyMirrored
      ? NodeOrGroup_fromGroup(this->rootNumID.value)
      : NodeOrGroup_fromNode(this->rootNumID);
   NumNodeID rootNumID = HeartbeatMsgEx_getRootNumID(this);

   // check whether root info is defined
   if( (HeartbeatMsgEx_getNodeType(this) != NODETYPE_Meta) || (NumNodeID_isZero(&rootNumID)))
      return;

   // try to apply the contained root info

   metaNodes = App_getMetaNodes(app);

   setRootRes = NodeStoreEx_setRootOwner(metaNodes, rootOwner, false);

   if(setRootRes)
   { // found the very first root
      Logger_logFormatted(log, Log_CRITICAL, logContext, "Root (by Heartbeat): %hu",
         HeartbeatMsgEx_getRootNumID(this) );
   }
}

元資料根目錄的確定和獲取

客戶端在掛載檔案系統，初始化超級塊時，會向元資料Root節點獲取根目錄的DirEntry資訊：

// fhgfs_client_module\source\filesystem\FhgfsOpsSuper.c

/**
 * Fill the file system superblock (vfs object)
 */
int FhgfsOps_fillSuper(struct super_block* sb, void* rawMountOptions, int silent)
{
...
   // init root inode

   memset(&kstat, 0, sizeof(struct kstat) );

   kstat.ino = BEEGFS_INODE_ROOT_INO;
   kstat.mode = S_IFDIR | 0777; // allow access for everyone
   kstat.atime = kstat.mtime = kstat.ctime = current_fs_time(sb);
   kstat.uid = FhgfsCommon_getCurrentKernelUserID();
   kstat.gid = FhgfsCommon_getCurrentKernelGroupID();
   kstat.blksize = Config_getTuneInodeBlockSize(cfg);
   kstat.nlink = 1;

   // root entryInfo is always updated when someone asks for it (so we just set dummy values here)
   EntryInfo_init(&entryInfo, NodeOrGroup_fromGroup(0), StringTk_strDup(""), StringTk_strDup(""),
      StringTk_strDup(""), DirEntryType_DIRECTORY, 0);

   rootInode = __FhgfsOps_newInode(sb, &kstat, 0, &entryInfo, &iSizeHints);
   if(!rootInode || IS_ERR(rootInode) )
   {
      __FhgfsOps_destructFsInfo(sb);
      return IS_ERR(rootInode) ? PTR_ERR(rootInode) : -ENOMEM;
   }

   rootDentry = d_make_root(rootInode);
   if(!rootDentry)
   {
      __FhgfsOps_destructFsInfo(sb);
      return -ENOMEM;
   }
...
}

// fhgfs_client_module\source\common\toolkit\MetadataTk.c

/**
 * @param outEntryInfo contained values will be kalloced (on success) and need to be kfreed with
 * FhgfsInode_freeEntryMinInfoVals() later.
 */
bool MetadataTk_getRootEntryInfoCopy(App* app, EntryInfo* outEntryInfo)
{
   NodeStoreEx* nodes = App_getMetaNodes(app);

   NodeOrGroup rootOwner = NodeStoreEx_getRootOwner(nodes);
   const char* parentEntryID = StringTk_strDup("");
   const char* entryID = StringTk_strDup(META_ROOTDIR_ID_STR);
   const char* dirName = StringTk_strDup("");
   DirEntryType entryType = (DirEntryType) DirEntryType_DIRECTORY;

   /* Even if rootOwner is invalid, we still init outEntryInfo and malloc as FhGFS
    * policy says that kfree(NULL) is not allowed (the kernel allows it). */

   EntryInfo_init(outEntryInfo, rootOwner, parentEntryID, entryID, dirName, entryType, 0);

   return NodeOrGroup_valid(rootOwner);
}

元資料節點在初始化時，會先後在BuddyMirror和非BuddyMirror目錄查詢ID名為root的根目錄DirEntry檔案：

// fhgfs_common\source\common\storage\Metadata.h

#define META_ROOTDIR_ID_STR            "root" /* initial file system entry point */


// fhgfs_meta\source\app\App.cpp

void App::initRootDir(NumNodeID localNodeNumID)
{
   // try to load root dir from disk (through metaStore) or create a new one

   this->metaStore = new MetaStore();

   // try to reference root directory with buddy mirroring
   rootDir = this->metaStore->referenceDir(META_ROOTDIR_ID_STR, true, true);

   // if that didn't work try to reference non-buddy-mirrored root dir
   if (!rootDir)
   {
      rootDir = this->metaStore->referenceDir(META_ROOTDIR_ID_STR, false, true);
   }

   if(rootDir)
   { // loading succeeded (either with or without mirroring => init rootNodeID
      this->log->log(Log_NOTICE, "Root directory loaded.");

      NumNodeID rootDirOwner = rootDir->getOwnerNodeID();
      bool rootIsBuddyMirrored = rootDir->getIsBuddyMirrored();

      // try to set rootDirOwner as root node
      if((rootDirOwner  != 0) && metaNodes->setRootNodeNumID(rootDirOwner, false,
         rootIsBuddyMirrored) )
      { // new root node accepted (check if rootNode is localNode)
         NumNodeID primaryRootDirOwner;
         if (rootIsBuddyMirrored)
            primaryRootDirOwner = NumNodeID(
               metaBuddyGroupMapper->getPrimaryTargetID(rootDirOwner.val() ) );
         else
            primaryRootDirOwner = rootDirOwner;

         if(localNodeNumID == primaryRootDirOwner)
         {
            log->log(Log_CRITICAL, "I got root (by possession of root directory)");
            if (rootIsBuddyMirrored)
               log->log(Log_CRITICAL, "Root directory is mirrored");
         }
         else
            log->log(Log_CRITICAL,
               "Root metadata server (by possession of root directory): " + rootDirOwner.str());
      }
   }
   else
   { // failed to load root directory => create a new rootDir (not mirrored)
      this->log->log(Log_CRITICAL,
         "This appears to be a new storage directory. Creating a new root dir.");

      UInt16Vector stripeTargets;
      unsigned defaultChunkSize = this->cfg->getTuneDefaultChunkSize();
      unsigned defaultNumStripeTargets = this->cfg->getTuneDefaultNumStripeTargets();
      Raid0Pattern stripePattern(defaultChunkSize, stripeTargets, defaultNumStripeTargets);

      DirInode newRootDir(META_ROOTDIR_ID_STR,
         S_IFDIR | S_IRWXU | S_IRWXG | S_IRWXO, 0, 0, NumNodeID(), stripePattern, false);

      this->metaStore->makeDirInode(newRootDir);
      this->rootDir = this->metaStore->referenceDir(META_ROOTDIR_ID_STR, false, true);

      if(!this->rootDir)
      { // error
         this->log->logErr("Failed to store root directory. Unable to proceed.");
         throw InvalidConfigException("Failed to store root directory");
      }
   }
}

元資料ID檔案命名規則

每個目錄或者檔案都會有一個ID檔案，而這個ID檔案的名字組成結構為<counterPart>-<timestampPart>-<localNodeID>：

// fhgfs_common\source\common\toolkit\StorageTk.h
class StorageTk
{
      /**
       * Generate ID for new fs entry (i.e. file or dir).
       */
      static std::string generateFileID(const NumNodeID localNodeID)
      {
         /* note: we assume here that the clock doesn't jump backwards between restarts of
          the daemon (and that there always is at least one second between restarts) and that we
          don't need more than 2^32 IDs per second (sustained) */

         uint64_t nextID = idCounter.increase();

         // note on idCounter value: high 32bits are timestamp, low 32bits are sequential counter

         /* note on switching time/counter in string representation: having the timestamp first is
          bad for strcmp() and such things, which the underlying fs might need to do - because in
          that order, the first characters of entryIDs/filenames would always be similar. */

         uint32_t counterPart = (uint32_t) nextID;
         uint32_t timestampPart = (uint32_t) (nextID >> STORAGETK_FILEID_TIMESTAMP_SHIFTBITS);

         return StringTk::uintToHexStr(counterPart) + "-" + StringTk::uintToHexStr(timestampPart)
            + "-" + localNodeID.strHex();
      }
}

元資料組織結構測試

清空和檢視資料目錄

$ rm /mnt/beegfs/* -rf

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

建立和檢視空檔案

$ touch /mnt/beegfs/test001

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

$ ll /beegfs_meta/meta1/dentries/38/51/root/test001 -i          
3804241 -rw-r--r-- 2 root root 0 4月  26 10:44 /beegfs_meta/meta1/dentries/38/51/root/test001

$ ll /beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2 -i
3804241 -rw-r--r-- 2 root root 0 4月  26 10:44 /beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2

$ touch /mnt/beegfs/test002

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal

建立和檢視一級目錄

$ mkdir /mnt/beegfs/dir001

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal



$ ll -i /beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
6293289 -rw-r--r-- 1 root root 0 4月  26 10:45 /beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2

$ ll -i /beegfs_meta/meta1/dentries/38/51/root/dir001
3804243 -rw-r--r-- 1 root root 0 4月  26 10:45 /beegfs_meta/meta1/dentries/38/51/root/dir001

$ getfattr -n user.fhgfs  /beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2 
getfattr: Removing leading '/' from absolute path names
# file: beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
user.fhgfs=0sBAMKAAIAAAAAAAAA7UEAANlwwlwAAAAA2XDCXAAAAABGx+RcAAAAAEbH5FwAAAAAAAAAAAAAAAABAAAADAAAADAtNUNDMjcwRDQtMgAAAAAEAAAAcm9vdAAAAAACAAAAAgAAABoAAAABAAAAAAAIAAEABAAAAAgAAAAAAAAA

$ getfattr -n user.fhgfs /beegfs_meta/meta1/dentries/38/51/root/dir001
getfattr: Removing leading '/' from absolute path names
# file: beegfs_meta/meta1/dentries/38/51/root/dir001
user.fhgfs=0sAgMQAAEAAAAMAAAAMC01Q0MyNzBENC0yAAAAAAIAAAA=

建立和檢視非空檔案

$ dd if=/dev/zero of=/mnt/beegfs/test00 bs=1K count=1

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2


$ dd if=/dev/zero of=/mnt/beegfs/test00 bs=1K count=1024

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2


$ dd if=/dev/zero of=/mnt/beegfs/test00 bs=1K count=4096

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2


$ dd if=/dev/zero of=/mnt/beegfs/test00 bs=1K count=16384

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2


$ find /beegfs_data/data[123]/{buddymir,chunks} -type f | xargs -i ls -lh {}
-rw-rw-rw- 1 root root 5.0M 5月   5 16:44 /beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
-rw-rw-rw- 1 root root 5.5M 5月   5 16:44 /beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
-rw-rw-rw- 1 root root 5.5M 5月   5 16:44 /beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2


$ dd if=/dev/zero of=/mnt/beegfs/test00 bs=1K count=65536

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2


$ find /beegfs_data/data[123]/{buddymir,chunks} -type f | xargs -i ls -lh {}                            
-rw-rw-rw- 1 root root 21M 5月   5 16:46 /beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
-rw-rw-rw- 1 root root 22M 5月   5 16:46 /beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
-rw-rw-rw- 1 root root 22M 5月   5 16:46 /beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2

建立和檢視多級目錄

$ touch /mnt/beegfs/dir001/test005

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2



$ mkdir /mnt/beegfs/dir002

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/dir002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2


$ mkdir /mnt/beegfs/dir001/dir003

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/dir002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2


$ mkdir /mnt/beegfs/dir001/dir003/dir004

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2
/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/dir002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003
/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2


$ mkdir /mnt/beegfs/dir001/dir003/dir004/dir005

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5
/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2
/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/dir002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003
/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004
/beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2


$ touch /mnt/beegfs/dir001/dir003/dir004/dir005/test006

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5
/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2
/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/dir002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/#fSiDs#/0-5CDA6D39-2
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006
/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004
/beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2


$ mkdir /mnt/beegfs/dir001/dir003/dir004/dir005/dir006

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5
/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2
/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/dir002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/#fSiDs#/0-5CDA6D39-2
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/dir006
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006
/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004
/beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2


$ mkdir /mnt/beegfs/dir001/dir003/dir004/dir005/dir006/dir007

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5
/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta2/inodes/67/63/0-5CEFCA37-5
/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2
/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5
/beegfs_meta/meta2/inodes/78/78/0-5CE4C714-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/dir002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir010
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/#fSiDs#/0-5CDA6D39-2
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/dir006
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006
/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/dir007
/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004
/beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2


$ touch /mnt/beegfs/dir001/dir003/dir004/dir005/dir006/test008

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5
/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta2/inodes/67/63/0-5CEFCA37-5
/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2
/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5
/beegfs_meta/meta2/inodes/78/78/0-5CE4C714-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/dir002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir010
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/#fSiDs#/0-5CDA6D39-2
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/dir006
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006
/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/#fSiDs#/0-5CEFCC92-5
/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/dir007
/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/test008
/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004
/beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2


$ touch /mnt/beegfs/dir001/dir003/dir004/dir005/dir006/dir007/test009

$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5
/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta2/inodes/67/63/0-5CEFCA37-5
/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2
/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5
/beegfs_meta/meta2/inodes/78/78/0-5CE4C714-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/dir002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir010
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/#fSiDs#/0-5CDA6D39-2
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/dir006
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006
/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/#fSiDs#/0-5CEFCC92-5
/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/dir007
/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/test008
/beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/test009
/beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/#fSiDs#/0-5CEFCD46-5
/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004
/beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2


$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f -exec ls -l {} \;
-rw-r--r-- 1 root root 0 4月  16 17:48 /beegfs_meta/meta1/inodes/35/5B/disposal
-rw-r--r-- 1 root root 0 4月  16 17:48 /beegfs_meta/meta1/inodes/38/51/root
-rw-r--r-- 1 root root 0 4月  26 10:45 /beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
-rw-r--r-- 1 root root 0 5月  14 15:23 /beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5
-rw-r--r-- 1 root root 0 5月  14 15:12 /beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
-rw-r--r-- 1 root root 0 4月  16 21:10 /beegfs_meta/meta2/inodes/35/5B/disposal
-rw-r--r-- 1 root root 0 5月  14 15:34 /beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2
-rw-r--r-- 1 root root 0 4月  16 21:10 /beegfs_meta/meta2/inodes/38/51/root
-rw-r--r-- 1 root root 0 5月  14 15:14 /beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2
-rw-r--r-- 1 root root 0 5月  14 15:15 /beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5
-rw-r--r-- 2 root root 0 4月  26 10:45 /beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
-rw-r--r-- 2 root root 0 4月  26 10:44 /beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
-rw-r--r-- 2 root root 0 4月  26 10:47 /beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
-rw-r--r-- 2 root root 0 4月  26 10:45 /beegfs_meta/meta1/dentries/38/51/root/test002
-rw-r--r-- 1 root root 0 5月  14 15:12 /beegfs_meta/meta1/dentries/38/51/root/dir002
-rw-r--r-- 2 root root 0 4月  26 10:47 /beegfs_meta/meta1/dentries/38/51/root/test00
-rw-r--r-- 1 root root 0 4月  26 10:45 /beegfs_meta/meta1/dentries/38/51/root/dir001
-rw-r--r-- 2 root root 0 4月  26 10:44 /beegfs_meta/meta1/dentries/38/51/root/test001
-rw-r--r-- 2 root root 0 5月   5 16:47 /beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
-rw-r--r-- 2 root root 0 5月   5 16:47 /beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
-rw-r--r-- 1 root root 0 5月  14 15:14 /beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003
-rw-r--r-- 2 root root 0 5月  14 15:25 /beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/#fSiDs#/0-5CDA6D39-2
-rw-r--r-- 1 root root 0 5月  14 15:34 /beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/dir006
-rw-r--r-- 2 root root 0 5月  14 15:25 /beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006
-rw-r--r-- 1 root root 0 5月  14 15:15 /beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004
-rw-r--r-- 1 root root 0 5月  14 15:23 /beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005
-rw-r--r-- 1 root root 0 4月  16 17:48 /beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
-rw-r--r-- 1 root root 0 4月  16 21:10 /beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
-rw-rw-rw- 1 root root 22020096 5月   5 16:46 /beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
-rw-rw-rw- 1 root root 22544384 5月   5 16:46 /beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
-rw-rw-rw- 1 root root 22544384 5月   5 16:46 /beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2

測試總結分析

對於每個檔案，在其dentries父目錄的ID目錄中有一個同名的DirEntry檔案，以及#fSiDs#子目錄中的以ID為名稱的硬連結；
對於每個目錄，在其dentries父目錄的ID目錄中有一個同名的DirEntry檔案，以及inodes目錄中的以ID為名稱的DirInode檔案；
如果一個目錄和其父目錄不在一個元資料節點，則其DirEntry檔案在父目錄所在節點，其DirInode檔案在其被真正儲存的節點；
無論是inodes還是dentry目錄，裡面都是有兩級Hash目錄，每一級有128個，根據DirEntry和DirInode的ID進行Hash。而ID檔案是根據時間戳生成，所以理論上可以做到很好的均衡。而根目錄的ID固定為root，所以總是在38/51這個Hash目錄中。

/mnt/beegfs/dir001
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta1/dentries/38/51/root/dir001

/mnt/beegfs/dir002
/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
/beegfs_meta/meta1/dentries/38/51/root/dir002

/mnt/beegfs/dir001/dir003
/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003

/mnt/beegfs/dir001/dir003/dir004
/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5
/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004

/mnt/beegfs/dir001/dir003/dir004/dir005
/beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5
/beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005

/mnt/beegfs/dir001/dir003/dir004/dir005/dir006
/beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006

$ ll -i /beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2
3279957 -rw-r--r-- 1 root root 0 5月  14 15:34 /beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2

$ ll -i /beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006
2362456 -rw-r--r-- 2 root root 0 5月  14 15:25 /beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006


/mnt/beegfs/dir001/dir003/dir004/dir005/dir006/test008
/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/#fSiDs#/0-5CEFCC92-5
/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/test008


/mnt/beegfs/dir001/dir003/dir004/dir005/dir006/dir007
/beegfs_meta/meta2/inodes/67/63/0-5CEFCA37-5
/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/dir007


/mnt/beegfs/dir001/dir003/dir004/dir005/dir006/dir007/test009
/beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/#fSiDs#/0-5CEFCD46-5
/beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/test009


$ ll -i /beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/#fSiDs#/0-5CEFCD46-5
1707090 -rw-r--r-- 2 root root 0 5月  30 20:32 /beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/#fSiDs#/0-5CEFCD46-5

$ ll -i /beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/test009
1707090 -rw-r--r-- 2 root root 0 5月  30 20:32 /beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/test009

多元資料節點的分配

在元資料節點每次建立目錄時，如果有多個元資料節點，優先選擇剩餘空間容量多的：

// fhgfs_meta\source\net\message\storage\creating\MkDirMsgEx.cpp

std::unique_ptr<MkDirMsgEx::ResponseState> MkDirMsgEx::mkDirPrimary(ResponseContext& ctx)
{
...
   metaCapacityPools->chooseStorageTargets(numDesiredTargets, minNumRequiredTargets,
      &getPreferredNodes(), &newOwnerNodes);
...
}


/**
 * @param numTargets number of desired targets
 * @param minNumRequiredTargets the minimum number of targets the caller needs; ususally 1
 * for RAID-0 striping and 2 for mirroring (so this parameter is intended to avoid problems when
 * there is only 1 target left in the normal pool and the user has mirroring turned on).
 * @param preferredTargets may be NULL
 */
void NodeCapacityPools::chooseStorageTargets(unsigned numTargets, unsigned minNumRequiredTargets,
   const UInt16List* preferredTargets, UInt16Vector* outTargets)
{
   SafeRWLock lock(&rwlock, SafeRWLock_READ);

   if(!preferredTargets || preferredTargets->empty() )
   { // no preference => start with first pool that contains any targets
      if(pools[CapacityPool_NORMAL].size() )
      {
         chooseStorageNodesNoPref(pools[CapacityPool_NORMAL],
            numTargets, outTargets);

         if(outTargets->size() >= minNumRequiredTargets)
            goto unlock_and_exit;
      }

      /* note: no "else if" here, because we want to continue with next pool if we didn't find
         enough targets for minNumRequiredTargets in previous pool */

      if(pools[CapacityPool_LOW].size() )
      {
         chooseStorageNodesNoPref(pools[CapacityPool_LOW],
            numTargets - outTargets->size(), outTargets);

         if(outTargets->size() >= minNumRequiredTargets)
            goto unlock_and_exit;
      }

      chooseStorageNodesNoPref(pools[CapacityPool_EMERGENCY],
         numTargets, outTargets);
   }
   else
   {
      // caller has preferred targets, so we can't say a priori whether nodes will be found or not
      // in a pool. our strategy here is to automatically allow non-preferred targets before
      // touching the emergency pool.

      std::set<uint16_t> chosenTargets;

      // try normal and low pool with preferred targets...

      chooseStorageNodesWithPref(pools[CapacityPool_NORMAL], numTargets, preferredTargets, false,
                                 outTargets, chosenTargets);

      if(outTargets->size() >= minNumRequiredTargets)
         goto unlock_and_exit;

      chooseStorageNodesWithPref(pools[CapacityPool_LOW], numTargets - outTargets->size(),
                                 preferredTargets, false, outTargets, chosenTargets);

      if(!outTargets->empty() )
         goto unlock_and_exit;

      /* note: currently, we cannot just continue here with non-empty outTargets (even if
         "outTargets->size() < minNumRequiredTargets"), because we would need a mechanism to exclude
         the already chosen outTargets for that (e.g. like an inverted preferredTargets list). */

      // no targets yet - allow non-preferred targets before using emergency pool...

      chooseStorageNodesWithPref(pools[CapacityPool_NORMAL], numTargets, preferredTargets, true,
                                 outTargets, chosenTargets);

      if(outTargets->size() >= minNumRequiredTargets)
         goto unlock_and_exit;

      chooseStorageNodesWithPref(pools[CapacityPool_LOW], numTargets - outTargets->size(),
                                 preferredTargets, true, outTargets, chosenTargets);

      if(!outTargets->empty() )
         goto unlock_and_exit;

      /* still no targets available => we have to try the emergency pool (first with preference,
         then without preference) */

      chooseStorageNodesWithPref(pools[CapacityPool_EMERGENCY], numTargets, preferredTargets, false,
         outTargets, chosenTargets);
      if(!outTargets->empty() )
         goto unlock_and_exit;

      chooseStorageNodesWithPref(pools[CapacityPool_EMERGENCY], numTargets, preferredTargets, true,
         outTargets, chosenTargets);
   }


unlock_and_exit:

   lock.unlock();
}

相關的配置選項和說明如下：

$ vi /etc/beegfs/beegfs-client.conf

tunePreferredMetaFile         =
tunePreferredStorageFile      =

# [tunePreferredMetaFile], [tunePreferredStorageFile]
# Path to a text file that contains the numeric IDs of preferred storage targets
# and metadata servers. These will be preferred when the client creates new file
# system entries. This is useful e.g. to take advantage of data locality in the
# case of multiple data centers. If unspecified, all available targets and
# servers will be used equally.
# Usage: One targetID per line for storage servers, one nodeID per line for
#    metadata servers.
# Note: TargetIDs and nodeIDs can be queried with the beegfs-ctl tool.
# Default: <none>


$ vi /etc/beegfs/beegfs-mgmtd.conf

tuneMetaDynamicPools                   = true
tuneMetaInodesLowLimit                 = 10M
tuneMetaInodesEmergencyLimit           = 1M
tuneMetaSpaceLowLimit                  = 10G
tuneMetaSpaceEmergencyLimit            = 3G
tuneStorageDynamicPools                = true
tuneStorageInodesLowLimit              = 10M
tuneStorageInodesEmergencyLimit        = 1M
tuneStorageSpaceLowLimit               = 1T
tuneStorageSpaceEmergencyLimit         = 20G

# [tune{Meta,Storage}DynamicPools]
# Temporarily raise the Low/Emergency limits if the spread (difference in free
# capacity between the targets with the most and the least free space) becomes
# too large. This will move targets to a lower pool earlier if there are other
# targets with much more free capacity.

# [tune{Meta,Storage}{Space,Inodes}LowLimit]
# [tune{Meta,Storage}{Space,Inodes}EmergencyLimit]
# The free space pool thresholds. If a metadata or storage target is below a
# threshold, it will only be used to store new files and directories when no
# higher class targets are available (i.e. while there are targets in the
# "normal" pool, no targets from the "low" pool will be used.)
# Note: Preferred target settings of a client will be ignored if it helps to
#    avoid using targets from the emergency class.
# Default: Space: Meta: 10G/3G; Storage: 512G/10G
#          Inodes: Meta: 10M/1M; Storage: 10M/1M

# [tune{Meta,Storage}{Space,Inodes}NormalSpreadThreshold]
# [tune{Meta,Storage}{Space,Inodes}LowSpreadThreshold]
# [tune{Meta,Storage}{Space,Inodes}LowDynamicLimit]
# [tune{Meta,Storage}{Space,Inodes}EmergencyDynamicLimit]
# Only effective if tune{Meta,Storage}DynamicPools is enabled.
# Whenever the spread (see above) of the free capacities in the normal / low
# class of storage targets is above this threshold, the StorageLowLimit /
# StorageEmergencyLimit is temporarily raised to StorageLowDynamicLimit /
# StorageEmergencyDynamicLimit.
# When the {Normal,Low}SpreadThreshold values are set to 0, the value from the
# corresponding {Low,Emergency}Limit is used for the spread threshold.
# When the {Low,Emergency}DynamicLimits are set to 0, they are automatically
# assumed as two times the corresponding {Low,Emergency}(non-dynamic)Limit.
# Default: 0

定址過程總結

管理節點在收到元資料節點的的心跳資訊時，如果沒有元資料Root節點，則選擇ID較小的作為元資料Root節點；
客戶端在Mount時，會初始化超級塊，同時向管理節點詢問元資料Root節點的ID，然後向元資料Root節點獲取DirEntry元資料；
元資料節點初始化時，會先依次尋找buddymir/inodes/38/51/root，inodes/38/51/root來確定根目錄的元資料；
根據Root節點中Root的目錄的元資料資訊中包含的OwnerID（確定目錄所在節點）和EntryID（根據Hash確定目錄路徑），就可以找到該目錄的元資料位置；
該目錄所有檔案的DirEntry檔案中包含有檔案的條帶資訊，以及資料分佈的儲存節點；
該目錄所有子目錄的DirEntry檔案的元資料中包含子目錄的OwnerID（即所在節點的NodeID）和EntryID，據此找到子目錄的元資料目錄，不斷迭代，即可找到最

BeeGFS元資料組織方式和路徑定址分析

元資料根節點的確定和獲取管理節點處理元資料節點的心跳資訊時，如果發現目前沒有Root節點，則會在已經註冊的節點中選擇ID最小的

ceph radosgw 物件定址和元資料組織

1. 物件定址 rgw物件定址的過程就是“拼接物件串“的過程，再結合bucket.pool就能對應到rados物件。 1.1 普通物件 1.2 multipart 物件 1.3 非multipart versioned物件 1.4 multipart version

小談——讀取web資源文件的方式和路徑問題

put mage 文件 ons 控制臺文件的 web doget web-inf 讀取web資源文件的方式 a): 采用servletContext對象獲得. 　　優點：任意文件，任意路徑都可獲得　　缺點：必須在web環境下 // 拿到全局對象

hbase 文件組織方式和多級索引機制

使用 dfs star 超過文件數據讀寫 -c BE 存在轉自 https://blog.csdn.net/u010039929/article/details/74217793 並且參照了 https://blog.csdn.net/songjifei/art

hive的資料組織格式和基本操作

hive的資料組織格式: 庫：管理資料不同模組的資料最好放在不同的資料庫中 &n

程序間資料通訊方式和特點

由於不同的程序執行在各自不同的記憶體空間中．一方對於變數的修改另一方是無法感知的．因此．程序之間的資訊傳遞不可能通過變數或其它資料結構直接進行，只能通過程序間通訊來完成。根據程序通訊時資訊量大小的不同，可以將程序通訊劃分為兩大型別：控制資訊的通訊和大批資料資訊的通訊．前者

分組交換資料報方式和虛電路方式的異同點

組交換是在報文交換和線路交換基礎上發展起來的技術，結合了兩者的優點。分組交換採用兩種不同的方法來管理被傳輸的分組流：資料報和虛電路。資料報（Datagram) 面向無連線的資料傳輸,工作過程類似於報文交換。採用資料報方式傳輸時，被傳輸的分組稱為資料報。資料報的前部增加

[資料結構]散列表-連結法和開放定址法線性探查

在介紹hash表之前首先提到直接定址表但是由於實際上儲存在字典裡的關鍵字集合K比實際上所有可能的關鍵字的全域U要小的多，因此散列表所需要的儲存空間比直接定址表要小的多通過雜湊函式 h:U -> {0,1,2…m-1} 其中m 遠小於|U| 但是對於h

【計算機網路】資料鏈路層（一）：資料幀和物理定址

資料鏈路層是物理層的上層，物理層是把電腦連線起來的物理手段，它主要規定了網路的一些電氣屬性，其作用是負責傳送0和1的電訊號。資料鏈路層位於物理層的上層，簡單的闡述它的作用就是確定0和1的分組方式。下

資料結構實驗三間接定址學生資訊

#include<iostream> using namespace std; const int Maxsize = 100; template<

磁碟定址（CHS定址方式、LBA定址方式）

目前市場上的硬碟價格越來越低，相比以前同樣價格買的硬碟容量越來越大。但是更換硬碟時，也要注意你的PC主機板是否支援目前的大容量硬碟。計算機的發展一直受著硬碟容量限制或BIOS（軟體系統，主機板是其硬體載體）訪問容量限制的制約。要明白這個原因我們需要對IDE介面硬碟的工作方式做一個介紹，即使到現在我們今天的I

鏈地址法和開放定址法，求等概率下查詢成功時的平均查詢長度

問題描述：演算法與資料結構的一個題目，用鏈地址法和開放定址法，求等概率情況下查詢成功時的平均查詢長度已知一組關鍵字(13,20,85,52,8),雜湊函式為：H(key)=key MOD 6

linux 路由表和路由定址過程

1 概述每一個linux系統中都具有IP路由表，它儲存了本地計算機可以到達的網路目的地址範圍和如何到達的路由資訊。路由表是TCP/IP通訊的基礎，本地計算機上的任何TCP/IP通訊都受到路由表的控制。 2 Linux 核心的路由表通過 route

bzoj 2337 [HNOI2011]XOR和路徑【高斯消元+dp】

name 直接 ring size scanf 高斯消元 str pre hnoi 首先，我們發現，因為是無向圖，所以相連的點之間是有“依賴性”的，所以不能直接用dp求解。因為是xor，所以按位處理，於是列線性方程組，設$ x[i] $為點i到n異或和為1的期望，因為從1

BZOJ2337 [HNOI2011]XOR和路徑【概率dp + 高斯消元】

但是 ++ clu alt HA exit rep 等於找到題目題解突然get到這樣路徑期望的題目八成是高斯消元因為路徑上的dp往往具有後效性，這就形成了一個方程組對於本題來說，直接對權值dp很難找到突破口但是由於異或是位獨立的，我們考慮求出每一位的期望設

bzoj 2337 [HNOI2011]XOR和路徑高斯消元+期望dp

return set 時間復雜度 += memset als \n line sum 題面題目傳送門解法既然有異或，那麽我們把每一位單獨考慮一下先枚舉是二進制的第幾位，然後設$f_i$表示點$i$這一位為1的概率是多少顯然，可以列出一個方程註意，自環的出

資料表示、定址方式與指令系統_計算機系統結構第二章_自考本科

概要:計算機系統結構第二章資料表示、定址方式與指令系統知識點小結一、資料表示（應用） 1、資料表示的定義，資料表示與資料結構的關係定義：能由計算機硬體識別和引用的資料型別，表現再它有對這種型別的資料進行操作的指令和運算部件 &nbs

[BZOJ2337][HNOI2011]XOR和路徑(概率+高斯消元)

直接不容易算，考慮拆成位處理。設f[i]表示i到n的期望路徑異或和（僅考慮某一位），則$f[y]=\sum\limits_{exist\ x1\to y=0}\frac{f[x1]}{d[x1]}+\sum\limits_{exist\ x2\to y=1}\frac{1-f[x2]}{d[x2]}$。

蔡高廳老師 - 高等數學閱讀筆記 - 14 定積分 -定積分的換元法 - 廣義積分和伽馬函式（65、66）

不定積分有第一，第二換元，定積分只有換元：周期函式的定積分定積分的分部積分法利用歸納法和分部積分法

js和原生應用常用的資料互動方式

場景1 在原生app中經常會使用到H5頁面，比如說電商中的活動頁，一些電商中的詳情頁，等等...這些頁面都有一個特點，那就是在未來修改的可能性，和一次性的機率特別的大。所以用H5的頁面是最睿智的一種選擇。一旦使用了H5那麼就少不了和原生開發的一些互動（Android, IOS）如下的方案能夠幫助你解決。

BeeGFS元資料組織方式和路徑定址分析

元資料根節點的確定和獲取

元資料根目錄的確定和獲取

元資料ID檔案命名規則

元資料組織結構測試

清空和檢視資料目錄

建立和檢視空檔案

建立和檢視一級目錄

建立和檢視非空檔案

建立和檢視多級目錄

測試總結分析

多元資料節點的分配

定址過程總結

相關推薦