Bug 1651

This sounds like a really neat way of introducing hard to track down bugs into the system, particularly when you consider that you've got to do this over three different package managers.

Quite a number of the steps have to be redone as soon as you change the image in any way. You have to reindex the packages, you have to build a manifest of what you'll install (since the user may have changed IMAGE_INSTALL or something related), you have to write out all the different forms of the rootfs again once you modify it and you have to unprelink it, then prelink it again. If had mklibs run against it, you're also on a highway to nowhere.

Given all the pieces that have to rerun anyway, is the incremental update such a significant time saver?

Purpose of incremental image generation is not build-time saving, but engineering work flow.

A lot of the quick prototype style workflow is, build an image.. boot it.. work on identifying issues, problems, etc..  make changes to the base system, update the image -- this way any existing scripts, work (debug files), configuration files, etc are not lost.  Boot, see if behavior changed in the expected way and repeat as necessary.  Once the system does what you want, you clean and do a fresh build.

Any incremental filesystem update is based on a specific option, it is not the default behavior of the system.  Even if it was something like:

bitbake -c update_fs core-image-minimal

The reindex of the package feed is easy, we already do that each time.  As long as there is a package database as part of the image, we can use that and the packaging mechanism to trigger an update.  Likely the hardest thing in this will be to simply identify what changed and trigger the update exclusively on what changed.  (It's also very possible this may not work across all packaging configurations and setups, and may be limited based on specific behaviors.)

Here is the Design document:(For the rpm based rootfs)

* Usage:
  1) Add the follow config option to a conf file:
     INC_IMAGE_GEN = "1"

  2) bitbake core-image-sato
     modify a package
     bitbake core-image-sato

  The rootfs ould not be totally removed and re-created in the second generation, it would be simply updated based on the "package"

I considered use the "bitbake -c update_fs core-image-minimal", this needs re-organize the image.bbclass, package_rpm.bbclass and rootfs_rpm.bbclass, but use the configure option INC_IMAGE_GEN would me much easier, and it would be enabled by default.

* Implementation:
  1) Add the follow information to meta-yocto/conf/local.conf.sample:

# Incremental image generation, the rootfs would be totally removed and
# re-created in the second generation by default, but with INC_IMAGE_GEN = "1",
# the rootfs would be kept, and will do update(remove/add some files) on it.
# NOTE: This is not suggested when you want to create a productive rootfs
#INC_IMAGE_GEN = "1"

It would not be enabled by default.

2) Figure out the pkg which need to be removed or re-installed, then use 'rpm -e 
   to remove the old one. Use the rpm's BUILDTIME to determine which pkg has been
   rebuilt.

   Figure out the pkg which is newly added, and use 'rpm -U' to install it.

This is only for the rpm based rootfs, I don't know whether we need this for ipk or deb  based rootfs. And I will send the pull request for the rpm part soon.

test steps and results below:
1. set build env, run "source ./oe-init-build-env poky-git-build".
2. add one line to local.conf of "poky-git-build" dir.
   INC_RPM_IMAGE_GEN = "1"
3. run "bitbake core-image-sato-sdk".
4. get modify time of file in "tmp/work/qemumips-poky-linux/core-image-sato-sdk-1.0-r0/rootfs/dev",change "meta/recipes-sato/images/core-image-sato-sdk.bb",remove "${SATO_IMAGE_FEATURES}", rerun "bitbake core-image-sato-sdk",next get modify time of file in "tmp/work/qemumips-poky-linux/core-image-sato-sdk-1.0-r0/rootfs/dev",the same as before rerun.
5. get modify time of file in "tmp/work/qemumips-poky-linux/core-image-sato-sdk-1.0-r0/rootfs/dev",change "meta/recipes-sato/images/core-image-sato-sdk.bb",add "${SATO_IMAGE_FEATURES}" which removed in step 4, rerun "bitbake core-image-sato-sdk",next get modify time of file in "tmp/work/qemumips-poky-linux/core-image-sato-sdk-1.0-r0/rootfs/dev",the same as before rerun.
6. run "bitbake bzip2 -cclean","rm -f rm -f sstate-cache/sstate-bzip2-*",change "meta/recipes-extended/bzip2/bzip2_1.0.6.bb"(add a comment line in it), rerun "bitbake core-image-sato-sdk",check do_rootfs log "tmp/work/qemumips-poky-linux/core-image-sato-sdk-1.0-r0/temp/log.do_rootfs" ,only bzip2 package update info be found :

...........
Processing task-core-x11-base-dev...
Manifest: /home/liangli/poky-git-build_45/tmp/work/qemumips-poky-linux/core-image-sato-sdk-1.0-r0/rootfs/install/install.manifest
Preparing...                ##################################################
bzip2                       ##################################################
update-alternatives: Linking /home/liangli/poky-git-build_45/tmp/work/qemumips-poky-linux/core-image-sato-sdk-1.0-r0/rootfs//usr/bin/bunzip2 to /usr/bin/bunzip2.bzip2
update-alternatives: Linking /home/liangli/poky-git-build_45/tmp/work/qemumips-poky-linux/core-image-sato-sdk-1.0-r0/rootfs//usr/bin/bzcat to /usr/bin/bzcat.bzip2
bzip2-dev                   ##################################################
Delayed package scriptlet: # liberation-fonts - postinst
............ 

test env: 
host :ubuntu 11.04 x86_64
kernel : 2.6.38-13-generic #53-Ubuntu SMP

git info : 0f4d99d207b224bb9ce23de00a48f795ae20b3a0
git branch :poky/master

This one has been divided into 3 parts: the rpm, deb and ipk, the rpm part has been done, the deb and ipk part will be done in M3:

http://bugzilla.pokylinux.org/show_bug.cgi?id=1893
http://bugzilla.pokylinux.org/show_bug.cgi?id=1894

Here are the git repo for the rpm based rootfs:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=9d1823b2a0193b3c20fa30f0a118b4ee80a20027

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=6bde156c5ce58c8e55cd6f5fd5f87982a919ba34

verified rpm part:
git info : 0f4d99d207b224bb9ce23de00a48f795ae20b3a0
git branch :poky/master