poky: bitbake sstate

 Introduction to the sstate.bbclass

Contents:
    1) Introduction
    2) Usage
    3) Tasks to be sstated
    4) The implementation of storing the sstate file
    5) Re-usage of the sstate file
    6) Pitfalls

1) Introduction
   Stored the task's result and re-use it when re-building, seems like
   ccache and ldat's rpm cache.

2) Usage
   - inherit sstate
     It has been inheritted by default, in meta/conf/distro/defaultsetup.conf
     INHERIT_DISTRO ?= "debian devshell sstate license"

   - setup the SSTATE_DIR, in meta/conf/bitbake.conf:
     SSTATE_DIR ?= "${TOPDIR}/sstate-cache"

   The sstate file would be stored and re-used.

3) Tasks to be sstated
   do_populate_sysroot
   do_populate_lic
   do_package_write_ipk
   do_package_write_deb
   do_package_write_rpm
   do_package
   do_deploy

4) The implementation of storing the sstate file
   In meta/classes/sstate.bbclass:
   21 python () {
   ...
   45     for task in unique_tasks:
   46         namemap.append(bb.data.getVarFlag(task, 'sstate-name', d))
   47         funcs = bb.data.getVarFlag(task, 'prefuncs', d) or ""
   48         funcs = "sstate_task_prefunc " + funcs
   49         bb.data.setVarFlag(task, 'prefuncs', funcs, d)
   50         funcs = bb.data.getVarFlag(task, 'postfuncs', d) or ""
   51         funcs = funcs + " sstate_task_postfunc"
   52         bb.data.setVarFlag(task, 'postfuncs', funcs, d)
   53     d.setVar('SSTATETASKNAMES', " ".join(namemap))
   54 }

   Note the line of:
   48         funcs = "sstate_task_prefunc " + funcs
   51         funcs = funcs + " sstate_task_postfunc"

   The two functions sstate_task_prefunc and sstate_task_postfunc are
   used to store the sstate file.
   
   The sstate_task_prefunc is used to preparing the environment, for
   example, remove the manifest file in tmp/sstate-control.

   And the sstate_task_postfunc:
   426 python sstate_task_postfunc () {
   427     shared_state = sstate_state_fromvars(d)
   428     sstate_install(shared_state, d)
   429     for intercept in shared_state['interceptfuncs']:
   430         bb.build.exec_func(intercept, d)
   431     sstate_package(shared_state, d)
   432 }

   And the sstate_package:
   320 def sstate_package(ss, d):
   ...
   373     bb.build.exec_func('sstate_create_package', d)
   374
   375     bb.siggen.dump_this_task(sstatepkg + ".siginfo", d)

   And the sstate_create_package:
   439 sstate_create_package () {
   440         cd ${SSTATE_BUILDDIR}
   441         # Need to handle empty directories
   442         if [ "$(ls -A)" ]; then
   443                 tar -czf ${SSTATE_PKG} *
   444         else
   445                 tar -cz --file=${SSTATE_PKG} --files-from=/dev/null
   446         fi
   447
   448         cd ${WORKDIR}
   449         rm -rf ${SSTATE_BUILDDIR}
   450 }

   We can see that the sstate file has been stored by tar, e.g:
   sstate-gzip-armv5te-poky-linux-gnueabi-1.4-r0-armv5te-2-079607fa7ece189eb79c856e9a16baa2_package.tgz
   sstate-gzip-armv5te-poky-linux-gnueabi-1.4-r0-armv5te-2-079607fa7ece189eb79c856e9a16baa2_package.tgz.siginfo

5) Re-usage of the sstate file
   - In classes/sstate.bbclass:
   461 BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
   - In bitbake/lib/bb/runqueue.py:
   self.hashvalidate = bb.data.getVar("BB_HASHCHECK_FUNCTION", cfgData, True) or None
   - In the task's bbclass, for example, meta/classes/package.bbclass:
   python do_package_setscene () {
           sstate_setscene(d)
   }
   addtask do_package_setscene

  The sstate_checkhashes is used to check whether there is a valid sstate
  file, if there is one, then save the task name and return it to
  bitbake/lib/bb/runqueue.py, and runqueue.py will set the task which
  need to be run and skip other tasks, for example, only run do_package_rpm
  for the package, and skip do_fetch, do_patch, do_config, do_compile and so on.

  The sstate_setscene will used to restore the sstate file, let's see the
  keypoint code in each file.

  In classes/sstate.bbclass:
  461 BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
  462
  463 def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
  464
  465     ret = []
  466     # This needs to go away, FIXME
  467     mapping = {
  468         "do_populate_sysroot" : "populate-sysroot",
  469         "do_populate_lic" : "populate-lic",
  470         "do_package_write_ipk" : "deploy-ipk",
  471         "do_package_write_deb" : "deploy-deb",
  472         "do_package_write_rpm" : "deploy-rpm",
  473         "do_package" : "package",
  474         "do_deploy" : "deploy",
  475     }
  376
  477     for task in range(len(sq_fn)):
  478         sstatefile = bb.data.expand("${SSTATE_DIR}/" + sq_hashfn[task] + "_" + mapping[sq_task[task]] + ".tgz", d)
  479         sstatefile = sstatefile.replace("${BB_TASKHASH}", sq_hash[task])
  480         if os.path.exists(sstatefile):
  481             bb.debug(2, "SState: Found valid sstate file %s" % sstatefile)
  482             ret.append(task)
  483             continue
  484         else:
  485             bb.debug(2, "SState: Looked for but didn't find file %s" % sstatefile)

  Note the array of "ret" which saves the tasks that we need run and
  return to runqueue.py

  In bitbake/lib/bb/runqueue.py:
  761 class RunQueue:
  762     def __init__(self, cooker, cfgData, dataCache, taskData, targets):
  ...
  769         self.hashvalidate = bb.data.getVar("BB_HASHCHECK_FUNCTION", cfgData, True) or None
  ...
 
  1371 class RunQueueExecuteScenequeue(RunQueueExecute):
  ...
  1458         if self.rq.hashvalidate:
  1459             sq_hash = []
  1460             sq_hashfn = []
  1461             sq_fn = []
  1462             sq_taskname = []
  1463             sq_task = []
  1464             noexec = []
  1465             stamppresent = []
  1466             for task in xrange(len(self.sq_revdeps)):
  ...
  1487                 sq_taskname.append(taskname)
  1488                 sq_task.append(task)
  ...

  At last, the sstate_setscene which is defined in sstate.bbclass and
  invoked by the task's bbclass:
  415 def sstate_setscene(d):
  416     shared_state = sstate_state_fromvars(d)
  417     accelerate = sstate_installpkg(shared_state, d)
  418     if not accelerate:
  419         raise bb.build.FuncFailed("No suitable staging package found")

  The sstate_installpkg:
  147 def sstate_installpkg(ss, d):
  ...
  171     bb.build.exec_func('sstate_unpack_package', d)
  ...

  The sstate_unpack_package:
  455 sstate_unpack_package () {
  456         mkdir -p ${SSTATE_INSTDIR}
  457         cd ${SSTATE_INSTDIR}
  458         tar -xvzf ${SSTATE_PKG}
  459 }

  The sstate_unpack_package will extract the files to the directory, and
  bitbake will go on running the task based on the directory.

6) Pitfalls

   - 'bitbake  -cclean' doesn't remove the sstate file, If you
      need rebuild the package completely, please remove the sstate file
      from the SSTATE_DIR, e.g.:
      rm -f SSTATE_DIR/sstate--*

   - It doesn't calculate the md5sum of the file in the white list:
     BB_HASHTASK_WHITELIST ?= "(.*-cross$|.*-native$|.*-cross-initial$|.*-cross-intermediate$|^virtual:native:.*|^virtual:nativesdk:.*)"
     This would make bitbake hit the sstate file incorrectly for some
     packages, for example, as far as we know is eglibc.

  - Sometimes, it should not hit the sstate file , but it does, this may
    be caused by the problem of the depends file, please file a bug if
    you met such a problem