ipfs, mfs & unixfs

UnixFS

UnixFs is a unix-like filesystem utilities on top of an ipld merkledag

• io
  The io subpackage provides helpers for reading files and manipulating directories. The DagReader takes a reference to a unixfs file and returns a file handle that can be read from and seeked through. The Directory interface allows you to easily read items in a directory, add items to a directory, and do lookups.

• mod
  The mod subpackage implements a DagModifier type that can be used to write to an existing unixfs file, or create a new one. The logic for this is significantly more complicated than for the dagreader, so its a separate type. (TODO: maybe it still belongs in the io subpackage though?)

• hamt
  The hamt subpackage implements a CHAMP hamt that is used in unixfs directory sharding.

• archive
  The archive subpackage implements a tar importer and exporter. The objects created here are not officially unixfs, but in the future, this may be integrated more directly.

UnixFs file from a IpldNode

func NewUnixfsFile(ctx context.Context, dserv ipld.DAGService, nd ipld.Node) (files.Node, error) {
	switch dn := nd.(type) {
	case *dag.ProtoNode:
		fsn, err := ft.FSNodeFromBytes(dn.Data())
		if err != nil {
			return nil, err
		}
		if fsn.IsDir() {
			return newUnixfsDir(ctx, dserv, dn)
		}
		if fsn.Type() == ft.TSymlink {
			return files.NewLinkFile(string(fsn.Data()), nil), nil
		}

	case *dag.RawNode:
	default:
		return nil, errors.New("unknown node type")
	}

	dr, err := uio.NewDagReader(ctx, nd, dserv)
	if err != nil {
		return nil, err
	}

	return &ufsFile{
		DagReader: dr,
	}, nil
}

Conversion from a block to a file

• Block format is checked in ipld.Decode. 3 formats are registered so far:

func init() {
	ipld.Register(cid.DagProtobuf, DecodeProtobufBlock)
	ipld.Register(cid.Raw, DecodeRawBlock)
	ipld.Register(cid.DagCBOR, ipldcbor.DecodeBlock)
}

MFS

mutable IPFS filesystem. Root is initialized as:

dsk := datastore.NewKey("/local/filesroot")
rootDS.Put(dsk, c.Bytes());

// Files loads persisted MFS root
func Files(mctx helpers.MetricsCtx, lc fx.Lifecycle, repo repo.Repo, dag format.DAGService) (*mfs.Root, error) {
	dsk := datastore.NewKey("/local/filesroot")
	pf := func(ctx context.Context, c cid.Cid) error {
		rootDS := repo.Datastore()
		if err := rootDS.Sync(blockstore.BlockPrefix); err != nil {
			return err
		}
		if err := rootDS.Sync(filestore.FilestorePrefix); err != nil {
			return err
		}

		if err := rootDS.Put(dsk, c.Bytes()); err != nil {
			return err
		}
		return rootDS.Sync(dsk)
	}

	var nd *merkledag.ProtoNode
	val, err := repo.Datastore().Get(dsk)
	ctx := helpers.LifecycleCtx(mctx, lc)

	switch {
	case err == datastore.ErrNotFound || val == nil:
		nd = unixfs.EmptyDirNode()
		err := dag.Add(ctx, nd)
		if err != nil {
			return nil, fmt.Errorf("failure writing to dagstore: %s", err)
		}
	case err == nil:
		c, err := cid.Cast(val)
		if err != nil {
			return nil, err
		}

		rnd, err := dag.Get(ctx, c)
		if err != nil {
			return nil, fmt.Errorf("error loading filesroot from DAG: %s", err)
		}

		pbnd, ok := rnd.(*merkledag.ProtoNode)
		if !ok {
			return nil, merkledag.ErrNotProtobuf
		}

		nd = pbnd
	default:
		return nil, err
	}

	root, err := mfs.NewRoot(ctx, dag, nd, pf)

	lc.Append(fx.Hook{
		OnStop: func(ctx context.Context) error {
			return root.Close()
		},
	})

	return root, err
}

Conversion from a MFS path to a file

mfs.Lookup will iterate the mfs.Root to get the path. mfs.Root is created when mfs initialized.

mfsRoot

QmbxJou6T8R7b6xuqEC5jR8hDJa9EqEEVkYFfskawQDvar is the mfs.Root block, created in the begining. We can see this block actually contain some links to the files/directories…

ipfs object get QmbxJou6T8R7b6xuqEC5jR8hDJa9EqEEVkYFfskawQDvar
{“Links”:[{“Name”:“e4”,“Hash”:“QmVD5F2sxesn57H7uCJtmKMdzdMuzJ67FgXiVhaeoBt5TA”,“Size”:23598},{“Name”:“f2”,“Hash”:“QmRFNu2WqHiKp2nLZ8qMcs1ZMMps8EGEeFP3acvuYgDFay”,“Size”:74},{“Name”:“f3”,“Hash”:“QmW9BbU9Tiq8dMeoANVeSrSAevawHGDNQVbidpKFToSwZg”,“Size”:74},{“Name”:“f4”
,“Hash”:“QmRFNu2WqHiKp2nLZ8qMcs1ZMMps8EGEeFP3acvuYgDFay”,“Size”:74},{“Name”:“f5”,“Hash”:“QmRFNu2WqHiKp2nLZ8qMcs1ZMMps8EGEeFP3acvuYgDFay”,“Size”:74},{“Name”:“f6”,“Hash”:“QmW9BbU9Tiq8dMeoANVeSrSAevawHGDNQVbidpKFToSwZg”,“Size”:74},{“Name”:“file”,“Hash”:“QmRFNu2WqHiKp2
nLZ8qMcs1ZMMps8EGEeFP3acvuYgDFay”,“Size”:74},{“Name”:“test1”,“Hash”:“QmTSMdpEMakBGdamtWqzApdj4zQdAeA9NdiFn1DSv51Knu”,“Size”:70947}],“Data”:"\u0008\u0001"}

ipfs files ls
e4
f2
f3
f4
f5
f6
file
test1

Samples

Take a look at at a MFS file:

ipfs files read /f2
“hello world”

Check out the CID of this MFS file:

ipfs files stat /f2
QmRFNu2WqHiKp2nLZ8qMcs1ZMMps8EGEeFP3acvuYgDFay
Size: 16
CumulativeSize: 74
ChildBlocks: 1
Type: file

Check out the block of the CID, it has a link pointing to a block:

ipfs dag get QmRFNu2WqHiKp2nLZ8qMcs1ZMMps8EGEeFP3acvuYgDFay
{“data”:“CAIYECAQ”,“links”:[{“Cid”:{"/":“QmQX9Q2YhruUmdFR2wmUftRC9GSb8mwsh3idUSUW2AkpQ1”},“Name”:"",“Size”:24}]}

This block has the data， json encoded:

ipfs dag get QmQX9Q2YhruUmdFR2wmUftRC9GSb8mwsh3idUSUW2AkpQ1
{“data”:“CAASECJoZWxsbyB3b3JsZCIgDQoYEA==”,“links”:[]}

or, text format:

ipfs object get QmQX9Q2YhruUmdFR2wmUftRC9GSb8mwsh3idUSUW2AkpQ1
{“Links”:[],“Data”:"\u0008\u0000\u0012\u0010"hello world" \r\n\u0018\u0010"}