11771 2017-07-07 11:18:14 +0000 unexpected rebuilds during selftesting 2017-07-10 09:40:16 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake unspecified x86 Multiple RESOLVED NOTABUG Undecided normal --- 1 patrick.ohly patrick.ohly poky.bs.watcher poky.watcher --- --- No (bug/feature does not impact docs) oldest_to_newest 74859 0 patrick.ohly 2017-07-07 11:18:14 +0000 In a OE selftest I was trying to avoid cluttering the regular layer with a test recipe. Instead, I created it on-the-fly in the test and added it to BBFILES. However, in that setup, do_prepare_recipe_sysroot and do_rootfs always run again, although none of the tasks signatures change. To reproduce, check out Poky 754c03006943a219ffd417c36ab909b464f57f67 (or any other recent version), then create this file: $ cat meta/lib/oeqa/selftest/cases/imagebuild.py from oeqa.selftest.case import OESelftestTestCase from oeqa.utils.commands import bitbake class ImageBuildTest(OESelftestTestCase): def test_build(self): # Creates a test-specific image recipe to avoid cluttering the actual layer with # a .bb file that is not used elsewhere. pn = 'test-image' bb = pn + '.bb' self.track_for_cleanup(bb) with open(bb, 'w') as f: # Identical to core-image-minimal, but could also have additional settings. f.write('require ${COREBASE}/meta/recipes-core/images/core-image-minimal.bb\n') self.append_config('BBFILES_append = " %s"' % os.path.abspath(bb)) # This shouldn't need to re-run do_rootfs when invoking "oe-selftest -r imagebuild.ImageBuildTest.test_build" # repeatedly, but somehow it does. bitbake(pn, output_log=self.logger) Run "oe-selftest -r imagebuild.ImageBuildTest.test_build" repeatedly. The expected outcome is that after the initial selftest, all following runs don't do anything. Instead I get: 2017-07-07 11:14:24,548 - oe-selftest - INFO - Initialising tasks...done. 2017-07-07 11:14:24,629 - oe-selftest - INFO - NOTE: Executing SetScene Tasks 2017-07-07 11:14:24,632 - oe-selftest - INFO - NOTE: Running setscene task 729 of 729 (/fast/build/poky/x86/test-image.bb:do_populate_lic_setscene) 2017-07-07 11:14:24,861 - oe-selftest - INFO - NOTE: recipe test-image-1.0-r0: task do_populate_lic_setscene: Started 2017-07-07 11:14:24,930 - oe-selftest - INFO - NOTE: recipe test-image-1.0-r0: task do_populate_lic_setscene: Succeeded 2017-07-07 11:14:24,945 - oe-selftest - INFO - NOTE: Executing RunQueue Tasks 2017-07-07 11:14:25,565 - oe-selftest - INFO - NOTE: Running noexec task 1438 of 2388 (/fast/build/poky/x86/test-image.bb:do_fetch) 2017-07-07 11:14:25,633 - oe-selftest - INFO - NOTE: Running task 1636 of 2388 (/fast/build/poky/x86/test-image.bb:do_prepare_recipe_sysroot) 2017-07-07 11:14:25,860 - oe-selftest - INFO - NOTE: recipe test-image-1.0-r0: task do_prepare_recipe_sysroot: Started 2017-07-07 11:14:26,363 - oe-selftest - INFO - NOTE: recipe test-image-1.0-r0: task do_prepare_recipe_sysroot: Succeeded 2017-07-07 11:14:26,378 - oe-selftest - INFO - NOTE: Running task 2381 of 2388 (/fast/build/poky/x86/test-image.bb:do_rootfs) 2017-07-07 11:14:27,426 - oe-selftest - INFO - NOTE: recipe test-image-1.0-r0: task do_rootfs: Started 2017-07-07 11:14:45,449 - oe-selftest - INFO - NOTE: recipe test-image-1.0-r0: task do_rootfs: Succeeded 2017-07-07 11:14:45,466 - oe-selftest - INFO - NOTE: Running task 2382 of 2388 (/fast/build/poky/x86/test-image.bb:do_write_qemuboot_conf) 2017-07-07 11:14:45,772 - oe-selftest - INFO - NOTE: recipe test-image-1.0-r0: task do_write_qemuboot_conf: Started 2017-07-07 11:14:45,786 - oe-selftest - INFO - NOTE: recipe test-image-1.0-r0: task do_write_qemuboot_conf: Succeeded 2017-07-07 11:14:45,796 - oe-selftest - INFO - NOTE: Running task 2383 of 2388 (/fast/build/poky/x86/test-image.bb:do_image) 2017-07-07 11:14:46,134 - oe-selftest - INFO - NOTE: recipe test-image-1.0-r0: task do_image: Started 2017-07-07 11:14:46,820 - oe-selftest - INFO - NOTE: recipe test-image-1.0-r0: task do_image: Succeeded 2017-07-07 11:14:46,836 - oe-selftest - INFO - NOTE: Running task 2384 of 2388 (/fast/build/poky/x86/test-image.bb:do_image_ext4) 2017-07-07 11:14:46,895 - oe-selftest - INFO - NOTE: Running task 2385 of 2388 (/fast/build/poky/x86/test-image.bb:do_image_tar) 2017-07-07 11:14:47,171 - oe-selftest - INFO - NOTE: recipe test-image-1.0-r0: task do_image_ext4: Started 2017-07-07 11:14:47,222 - oe-selftest - INFO - NOTE: recipe test-image-1.0-r0: task do_image_tar: Started 2017-07-07 11:14:47,530 - oe-selftest - INFO - NOTE: recipe test-image-1.0-r0: task do_image_tar: Succeeded 2017-07-07 11:14:47,868 - oe-selftest - INFO - NOTE: recipe test-image-1.0-r0: task do_image_ext4: Succeeded 2017-07-07 11:14:47,883 - oe-selftest - INFO - NOTE: Running task 2386 of 2388 (/fast/build/poky/x86/test-image.bb:do_image_complete) 2017-07-07 11:14:48,220 - oe-selftest - INFO - NOTE: recipe test-image-1.0-r0: task do_image_complete: Started 2017-07-07 11:14:48,816 - oe-selftest - INFO - NOTE: recipe test-image-1.0-r0: task do_image_complete: Succeeded 2017-07-07 11:14:48,830 - oe-selftest - INFO - NOTE: Running task 2387 of 2388 (/fast/build/poky/x86/test-image.bb:do_image_qa) 2017-07-07 11:14:49,130 - oe-selftest - INFO - NOTE: recipe test-image-1.0-r0: task do_image_qa: Started 2017-07-07 11:14:49,136 - oe-selftest - INFO - NOTE: recipe test-image-1.0-r0: task do_image_qa: Succeeded 2017-07-07 11:14:49,142 - oe-selftest - INFO - NOTE: Running noexec task 2388 of 2388 (/fast/build/poky/x86/test-image.bb:do_build) 2017-07-07 11:14:50,263 - oe-selftest - INFO - NOTE: Tasks Summary: Attempted 2388 tasks of which 2378 didn't need to be rerun and all succeeded. 2017-07-07 11:14:50,364 - oe-selftest - INFO - test_build (imagebuild.ImageBuildTest) ... OK (31.114s) 74860 1 patrick.ohly 2017-07-07 11:19:57 +0000 I suspected that perhaps some of the "bitbake" calls done by oe-selftest cause a cleanup of the build directory because the recipe is not currently present, but I've not been able to actually observe or reproduce that manually. Just a thought. 74887 2 richard.purdie 2017-07-08 12:46:50 +0000 (In reply to comment #1) > I suspected that perhaps some of the "bitbake" calls done by oe-selftest > cause a cleanup of the build directory because the recipe is not currently > present, but I've not been able to actually observe or reproduce that > manually. Just a thought. There is code which does this, if something is "unreachable", it gets removed from the build. This is to allow things like users switching between sysvinit and systemd for example. See sstate_eventhandler2[eventmask] = "bb.event.ReachableStamps" in sstate.bbclass. If that code is triggering, the removal is entirely by design. 74891 3 patrick.ohly 2017-07-10 09:40:16 +0000 (In reply to comment #2) > (In reply to comment #1) > > I suspected that perhaps some of the "bitbake" calls done by oe-selftest > > cause a cleanup of the build directory because the recipe is not currently > > present, but I've not been able to actually observe or reproduce that > > manually. Just a thought. > > There is code which does this, if something is "unreachable", it gets > removed from the build. This is to allow things like users switching between > sysvinit and systemd for example. > > See sstate_eventhandler2[eventmask] = "bb.event.ReachableStamps" in > sstate.bbclass. If that code is triggering, the removal is entirely by > design. That's probably it. I tried to trigger the removal with "bitbake -e", but that alone was not enough. I'm not going to investigate further what specific action in oe-selftest triggers the removal. My idea with transient recipes goes against the design, so I'll just have to add some permanent recipe somewhere.