ipfs, unixFs & DAG iteration

The way to get all blocks of IPLD

Let’s consider we have an IPLD node and we build a a UnixFs file from it, then how do we read the file content at once:

	node, err := dagRO.Get(ctx, Cid)
	if err != nil {
		return err
	}
	file, err := unixfile.NewUnixfsFile(ctx, dagRO, node)
	if err != nil {
		return err
	}

	r := files.ToFile(file)
	if r == nil {
		return errors.New("file is not regular")
	}
	var buffer [1024000]byte
	var sum int64

	t := time.Now()
	for {
		n, err := r.Read(buffer[:])
		sum += int64(n)
		fmt.Println("read", n, err)
		if err != nil {
			fmt.Println("bad read", err)
			break
		}
	}

As shown above, UnixFs creats a io.reader of our file. Therefore, what happened when we kick reading it?

github.com/ipfs/go-ipld-format.GetNodes at daghelpers.go:47
github.com/ipfs/go-ipld-format.(*NavigableIPLDNode).preload at navipld.go:95
github.com/ipfs/go-ipld-format.(*NavigableIPLDNode).FetchChild at navipld.go:52
github.com/ipfs/go-ipld-format.(*Walker).fetchChild at walker.go:325
github.com/ipfs/go-ipld-format.(*Walker).down at walker.go:299
github.com/ipfs/go-ipld-format.(*Walker).Iterate at walker.go:195
github.com/ipfs/go-unixfs/io.(*dagReader).CtxReadFull at dagreader.go:172
github.com/ipfs/go-unixfs/io.(*dagReader).Read at dagreader.go:144
<autogenerated>:2
main.bs at bs.go:109
main.glob..func1 at stub.go:227
github.com/urfave/cli/v2.(*Command).Run at command.go:161
github.com/urfave/cli/v2.(*App).RunContext at app.go:302
github.com/kingwel-xie/xcli.RunCli at cli.go:259
github.com/kingwel-xie/xcli.RunP2PNodeCLI at commands.go:71
main.main at stub.go:208
runtime.main at proc.go:203
runtime.goexit at asm_amd64.s:1357
 - Async stack trace
runtime.rt0_go at asm_amd64.s:220

The io.reader is actually a dagReader, which will iterate the blocks belonged to this IPLD.

ipld-format.GetNodes will start a routine to get the requested Nodes. Each requested node will be linked to a ‘promise’ channel for further notification of getting result. In the routine, ds.GetMany(ctx, dedupedKeys) will try to get the blcoks from local blockstore, or if failed, from the exchange interface aka. bitswap.

// GetNodes returns an array of 'FutureNode' promises, with each corresponding
// to the key with the same index as the passed in keys
func GetNodes(ctx context.Context, ds NodeGetter, keys []cid.Cid) []*NodePromise {
	// promise has a channel named 'done'
	promises := make([]*NodePromise, len(keys))
	for i := range keys {
		promises[i] = NewNodePromise(ctx)
	}	
	dedupedKeys := dedupeKeys(keys)
	go func() {
		ctx, cancel := context.WithCancel(ctx)
		defer cancel()

		nodechan := ds.GetMany(ctx, dedupedKeys)

		for count := 0; count < len(keys); {
			select {
			case opt, ok := <-nodechan:
				if !ok {
					for _, p := range promises {
						p.Fail(ErrNotFound)
					}
					return
				}
				if opt.Err != nil {
					for _, p := range promises {
						p.Fail(opt.Err)
					}
					return
				}
				nd := opt.Node
				c := nd.Cid()
				for i, lnk_c := range keys {
					if c.Equals(lnk_c) {
						count++
						promises[i].Send(nd)
					}
				}
			case <-ctx.Done():
				return
			}
		}
	}()
	return promises
}

In ipld.FetchChild, it will wait for the ‘promise’ notification:

func (nn *NavigableIPLDNode) FetchChild(ctx context.Context, childIndex uint) (NavigableNode, error) {
	// This function doesn't check that `childIndex` is valid, that's
	// the `Walker` responsibility.

	// If we drop to <= preloadSize/2 preloading nodes, preload the next 10.
	for i := childIndex; i < childIndex+preloadSize/2 && i < uint(len(nn.childPromises)); i++ {
		// TODO: Check if canceled.
		if nn.childPromises[i] == nil {
			nn.preload(ctx, i)
			break
		}
	}
	// Here, waiting for the promise.done channle
	child, err := nn.getPromiseValue(ctx, childIndex)

Done