10948 2017-01-20 19:27:47 +0000 buildhistory not reporting srcrev's when using sstate cache 2017-06-16 16:18:59 +0000 1 1 7 Build System, Metadata & Runtime OE-Core Scripts and Tools 2.2 x86 x86 RESOLVED FIXED Medium normal 2.4 M1 1 jon.richardson eduard.bartosh eduard.bartosh ross.burton scott.branden --- --- No (bug/feature does not impact docs) oldest_to_newest 69973 0 jon.richardson 2017-01-20 19:27:47 +0000 I enable buildhistory by adding INHERIT += "buildhistory" to local.conf. When we do a build with the sstate cache disabled, the latest_srcrev files are written properly to the buildhistory directory and running buildhistory-collect-srcrevs returns the SRCREV for all our recipes using AUTOREV. When we run the same commands with a populated sstate cache, there is almost never latest_srcrev files written in buildhistory and the buildhistory-collect-srcrevs returns nothing. Sometimes it returns one the SRCREV of one recipe - not sure why. I tried deleting build/tmp and build/cache but it makes no difference. sstate cache is enabled with SSTATE_DIR and SSTATE_MIRRORS populated. When disabled, neither are specified. Thanks. 69974 1 jon.richardson 2017-01-20 19:50:57 +0000 A bit more info on what we're compiling. When we do a build we are generating a custom image with many recipes included in the image that use AUTOREV. It also builds several bootloaders such as u-boot, plus a couple of others, built at the same time the image is generated by using EXTRA_IMAGEDEPENDS in our machine conf. Even when doing a bitbake <recipe> -c cleanall and rebuilding I don't get SRCREV info. 71122 2 eduard.bartosh 2017-03-08 12:21:49 +0000 Couple of observations and questions - you can't disable sstate cache by commenting out SSTATE_DIR. This will just make sstate to use default location of sstate-cache, which is "${TOPDIR}/sstate-cache" - it looks like the difference comes from SSTATE_MIRRORS, i.e. buildhistory-collect-srcrevs doesn't show AUTOREV revisions when SSTATE_MIRRORS are enabled. Please, confirm it. - Can you show the value of SSTATE_MIRRORS please? 71156 3 jon.richardson 2017-03-09 01:59:32 +0000 Hi Ed. Yes, correct on both points. I can confirm that buildhistory-collect-srcrevs doesn't show AUTOREV revisions when SSTATE_MIRRORS is enabled. The value of the variable to enable sstate is: SSTATE_MIRRORS="file://.* file:///projects/sw/yocto/mirror/sstate-cache/PATH" 71402 4 sjolley.yp.pm 2017-03-17 17:25:49 +0000 It appears the question was answered. 71860 5 eduard.bartosh 2017-03-29 14:07:15 +0000 I'm not able to get any output from buildhistory-collect-srcrevs even with commented out SSTATE_MIRRORS line in local.conf Jonathan, can you suggest some way to reproduce this issue? 71983 6 jon.richardson 2017-04-01 00:22:13 +0000 Not sure why you don't get any output if you're using autorev. I put together a simple test. If you need a real recipe I'll find one I can share and modify it exactly, but I'll assume you have some recipe pointing to a git repo using autorev: my-recipe.bb: SRCREV = "${AUTOREV}" SRC_URI = "git://somewhere.com/repo.git;protocol=ssh;branch=master;" # Force access to git repo every time. PV="git${SRCPV}" 1) In build/conf/local.conf add: INHERIT += "buildhistory" # I have these variables in (conf/distro/include/my-distro-defaults.inc), but local.conf should be fine SSTATE_DIR ?= "${TOPDIR}/../../../new-yocto-cache/sstate-cache" SSTATE_MIRRORS = "" 2) bitbake my-recipe 3) buildhistory-collect-srcrevs SRCREV_pn-my-recipe = "2069d0d124c5d325890a91a6c309693c45143ff9" 4) Populate the sstate cache: - mkdir ~/mirror - cp -fr ../../../new-yocto-cache/sstate-cache /home/user/mirror/ 5) Set SSTATE_MIRRORS to the populated sstate cache: SSTATE_MIRRORS ?= "file://.* file:///home/user/yocto/mirror/sstate-cache/PATH 6) In your build dir: rm -rf buildhistory cache tmp 7) bitbake my-recipe 8) buildhistory-collect-srcrevs <no output> 9) bitbake my-recipe -c cleanall 10) buildhistory-collect-srcrevs <no output> 11) Remove SSTATE_MIRRRORS: SSTATE_MIRRORS = "" 12) bitbake my-recipe -c cleanall; bitbake my-recipe 13) buildhistory-collect-srcrevs SRCREV_pn-my-recipe = "2069d0d124c5d325890a91a6c309693c45143ff9" 72068 7 eduard.bartosh 2017-04-04 14:42:44 +0000 Jonathan, thank you for detailed example. I was able to reproduce this. The reason for it is that latest_srcrev files are written after sources are fetched, i.e. after do_fetch task is executed. This does not happen when sstate mirror already has fetched sources. So, latest_srcrev files are not created and buildhistory-collect-srcrevs reports nothing. 72098 8 eduard.bartosh 2017-04-05 11:23:09 +0000 I've noticed one difference in your example: with enabled SSTATE_MIRRORS you don't build your recipe again after running cleanall task: 9) bitbake my-recipe -c cleanall with disabled SSTATE_MIRRORS you do: 12) bitbake my-recipe -c cleanall; bitbake my-recipe when you build the recipe after cleaning do_fetch task is called and write_srcrev is called too and creates latest_srcrev for your recipe. However, if you don't build a recipe it doesn't happen. Can you try to run bitbake my-recipe when SSTATE_MIRRORS is enabled? It worked just fine for me: $ grep ^SSTATE_MIRRORS conf/local.conf SSTATE_MIRRORS = "file://.* file:///home/ed/yocto/poky/sstate-cache.mirror/PATH" $ rm -rf buildhistory cache tmp $ bitbake lz4 -c cleanall $ buildhistory-collect-srcrevs # core2-64-poky-linux SRCREV_pn-lz4 = "7bb64ff2b69a9f8367de9ab483cdadf42b4c1b65" I'm using modified lz4 recipe. Here is the diff: --- a/meta/recipes-support/lz4/lz4.bb +++ b/meta/recipes-support/lz4/lz4.bb @@ -2,13 +2,15 @@ SUMMARY = "Extremely Fast Compression algorithm" DESCRIPTION = "LZ4 is a very fast lossless compression algorithm, providing compression speed at 400 MB/s per core, scalable with multi-cores CPU. It also features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems." LICENSE = "BSD" -LIC_FILES_CHKSUM = "file://lib/LICENSE;md5=0b0d063f37a4477b54af2459477dcafd" +LIC_FILES_CHKSUM = "file://lib/LICENSE;md5=ebc2ea4814a64de7708f1571904b32cc" -SRCREV = "d86dc916771c126afb797637dda9f6421c0cb998" +#SRCREV = "d86dc916771c126afb797637dda9f6421c0cb998" +SRCREV = "${AUTOREV}" -PV = "131+git${SRCPV}" +#PV = "131+git${SRCPV}" +PV="git${SRCPV}" -SRC_URI = "git://github.com/Cyan4973/lz4.git" +SRC_URI = "git://github.com/Cyan4973/lz4.git;branch=master" 72099 9 eduard.bartosh 2017-04-05 11:42:26 +0000 Sorry, missed 'bitbake lz4' command in previous comment. Here is how it should loook: $ grep ^SSTATE_MIRRORS conf/local.conf SSTATE_MIRRORS = "file://.* file:///home/ed/yocto/poky/sstate-cache.mirror/PATH" $ rm -rf buildhistory cache tmp $ bitbake lz4 -c cleanall $ bitbake lz4 $ buildhistory-collect-srcrevs # core2-64-poky-linux SRCREV_pn-lz4 = "7bb64ff2b69a9f8367de9ab483cdadf42b4c1b65" 72110 10 jon.richardson 2017-04-05 20:23:30 +0000 Hi Ed. Sorry, there were a couple of typos in my instructions. I still see the failure even with lz4. Step 5 had the wrong directory for the sstate cache. Correction below. 5) Set SSTATE_MIRRORS to the populated sstate cache: #SSTATE_MIRRORS ?= "file://.* file:///home/user/yocto/mirror/sstate-cache/PATH SSTATE_MIRRORS ?= "file://.* file:///home/user/mirror/sstate-cache/PATH Step 9 was meant to re-compile the recipe again. 9) bitbake my-recipe -c cleanall; bitbake my-recipe Just want to make sure my instructions didn't mess up your test when the sstate-cache was copied to /home/user/mirror and not /home/user/yocto/mirror. I went back and forth enabling/disabling SSTATE_MIRRORS and it consistently fails. 72138 11 eduard.bartosh 2017-04-06 15:29:36 +0000 I'm not able to reproduce it in my environment. Below is the set of steps I performed following your instructions. I used modified lz4 example on the latest poky master (633ad6c9f436f5d2b6ee1a005b697661a054a394) 1) In build/conf/local.conf I have the following variables: SSTATE_DIR ?= "${TOPDIR}/../sstate-cache" SSTATE_MIRRORS = "" ... INHERIT += "buildhistory" 2) bitbake lz4 3) buildhistory-collect-srcrevs # core2-64-poky-linux SRCREV_pn-lz4 = "7bb64ff2b69a9f8367de9ab483cdadf42b4c1b65" 4) Populate the sstate cache: - cp -fr ../sstate-cache ../sstate-cache.mirror 5) Set SSTATE_MIRRORS to the populated sstate cache: SSTATE_MIRRORS = "file://.* file:///home/ed/yocto/poky/sstate-cache.mirror/PATH" 6) In my build dir: rm -rf buildhistory cache tmp 7) bitbake lz4 8) buildhistory-collect-srcrevs <no output> 9) bitbake lz4 -c cleanall 10) buildhistory-collect-srcrevs buildhistory-collect-srcrevs # core2-64-poky-linux SRCREV_pn-lz4 = "7bb64ff2b69a9f8367de9ab483cdadf42b4c1b65" Please, try to repeat this with the same repository and revision that I used. It might be that the issue is fixed only in master. 72555 12 jon.richardson 2017-04-17 21:12:45 +0000 Thanks, I'll try to reproduce it as soon as I can. Issues beyond my control are preventing me from accessing some of the git repos required when upgrading to the latest. I do see a commit in there that may have fixed it. 72768 13 jon.richardson 2017-04-21 18:53:47 +0000 I upgraded poky to 633ad6c9f436f5d2b6ee1a005b697661a054a394. Unfortunately I can't reproduce your results. buildhistory-collect-srcrevs still doesn't return anything. I tried removing my PREMIRRORS_prepend, DL_DIR, and BB_GENERATE_MIRROR_TARBALLS variables also in case those were affecting it. I tried using SSTATE_MIRRORS set and doing 'bitbake lz4 -c install -f' and it works fine, but it's not surprising because the sstate cache is bypassed. 72943 14 sjolley.yp.pm 2017-04-28 16:31:37 +0000 It appears the question has been answered. Moving back to Ed to work. 72970 15 eduard.bartosh 2017-05-02 07:31:10 +0000 Sorry for the delay. I was on vacation. I'll try to reproduce the issue with absolutely clean build on the latest master. 72974 16 eduard.bartosh 2017-05-02 08:56:57 +0000 I'm still not able to reproduce this issue even with a clean build on the latest poky master (381897c64069ea43d595380a3ae913bcc79cf7e1). Here is a detailed list of what I did: 1. Modify lz4 recipe to use AUTOREV: diff --git a/meta/recipes-support/lz4/lz4.bb b/meta/recipes-support/lz4/lz4.bb index 03c5a7a..9a8e612 100644 --- a/meta/recipes-support/lz4/lz4.bb +++ b/meta/recipes-support/lz4/lz4.bb @@ -2,13 +2,15 @@ SUMMARY = "Extremely Fast Compression algorithm" DESCRIPTION = "LZ4 is a very fast lossless compression algorithm, providing compression speed at 400 MB/s per core, scalable with multi-cores CPU. It also features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems." LICENSE = "BSD" -LIC_FILES_CHKSUM = "file://lib/LICENSE;md5=0b0d063f37a4477b54af2459477dcafd" +LIC_FILES_CHKSUM = "file://lib/LICENSE;md5=ebc2ea4814a64de7708f1571904b32cc" -SRCREV = "d86dc916771c126afb797637dda9f6421c0cb998" +#SRCREV = "d86dc916771c126afb797637dda9f6421c0cb998" +SRCREV = "${AUTOREV}" -PV = "131+git${SRCPV}" +#PV = "131+git${SRCPV}" +PV="git${SRCPV}" -SRC_URI = "git://github.com/Cyan4973/lz4.git" +SRC_URI = "git://github.com/Cyan4973/lz4.git;branch=master" S = "${WORKDIR}/git" 2. Create fresh build directory: $ . oe-init-build-env build-10948 You had no conf/local.conf file. This configuration file has therefore been created for you with some default values. You may wish to edit it to, for example, select a different MACHINE (target hardware). See conf/local.conf for more information as common configuration options are commented. You had no conf/bblayers.conf file. This configuration file has therefore been created for you with some default values. To add additional metadata layers into your configuration please add entries to conf/bblayers.conf. The Yocto Project has extensive documentation about OE including a reference manual which can be found at: http://yoctoproject.org/documentation For more information about OpenEmbedded see their website: http://www.openembedded.org/ ### Shell environment set up for builds. ### You can now run 'bitbake <target>' Common targets are: core-image-minimal core-image-sato meta-toolchain meta-ide-support You can also run generated qemu images with a command like 'runqemu qemux86' 3. Add 2 lines to conf/local.conf: SSTATE_MIRRORS = "" INHERIT += "buildhistory" 4. Build lz4: $ bitbake lz4 WARNING: Host distribution "opensuse-13.2" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution. Parsing recipes: 100% |###############################################################################################################################################| Time: 0:00:20 Parsing of 830 .bb files complete (0 cached, 830 parsed). 1299 targets, 48 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependencies Build Configuration: BB_VERSION = "1.34.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "opensuse-13.2" TARGET_SYS = "i586-poky-linux" MACHINE = "qemux86" DISTRO = "poky" DISTRO_VERSION = "2.3" TUNE_FEATURES = "m32 i586" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "master:381897c64069ea43d595380a3ae913bcc79cf7e1" NOTE: Fetching uninative binary shim from http://downloads.yoctoproject.org/releases/uninative/1.6/x86_64-nativesdk-libc.tar.bz2;sha256sum=2b4fffa308d9f19e0742a1a404ff42495fb50c165e5ca0458cedca157372691a --2017-05-02 11:04:38-- http://downloads.yoctoproject.org/releases/uninative/1.6/x86_64-nativesdk-libc.tar.bz2 Resolving proxy-chain.intel.com (proxy-chain.intel.com)... 163.33.7.140 Connecting to proxy-chain.intel.com (proxy-chain.intel.com)|163.33.7.140|:911... connected. Proxy request sent, awaiting response... 200 OK Length: 2535308 (2.4M) [application/octet-stream] Saving to: ‘/home/ed/git/yocto/poky/build-10948/downloads/uninative/2b4fffa308d9f19e0742a1a404ff42495fb50c165e5ca0458cedca157372691a/x86_64-nativesdk-libc.tar.bz2’ 2017-05-02 11:04:39 (2.99 MB/s) - ‘/home/ed/git/yocto/poky/build-10948/downloads/uninative/2b4fffa308d9f19e0742a1a404ff42495fb50c165e5ca0458cedca157372691a/x86_64-nativesdk-libc.tar.bz2’ saved [2535308/2535308] Initialising tasks: 100% |############################################################################################################################################| Time: 0:00:00 NOTE: Executing SetScene Tasks NOTE: Executing RunQueue Tasks WARNING: gmp-native-6.1.2-r0 do_fetch: Failed to fetch URL https://gmplib.org/download/gmp/gmp-6.1.2.tar.bz2, attempting MIRRORS if available NOTE: Tasks Summary: Attempted 470 tasks of which 0 didn't need to be rerun and all succeeded. Summary: There were 2 WARNING messages shown. 5. Run buildhistory-collect-srcrevs $ buildhistory-collect-srcrevs # i586-poky-linux SRCREV_pn-lz4 = "7bb64ff2b69a9f8367de9ab483cdadf42b4c1b65" 6. Populate the sstate cache: $ cp -fr sstate-cache sstate-cache.mirror 7. Set SSTATE_MIRRORS to the populated sstate cache in conf/local.conf: SSTATE_MIRRORS = "file://.* file:///home/ed/yocto/poky/build-10948/sstate-cache.mirror/PATH" 8. Remove buildhistory, cache and tmp directories: $ rm -rf buildhistory cache tmp 9. Build lz4 again: $ bitbake lz4 WARNING: Host distribution "opensuse-13.2" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution. Parsing recipes: 100% |###############################################################################################################################################| Time: 0:00:20 Parsing of 830 .bb files complete (0 cached, 830 parsed). 1299 targets, 48 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependencies Build Configuration: BB_VERSION = "1.34.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "opensuse-13.2" TARGET_SYS = "i586-poky-linux" MACHINE = "qemux86" DISTRO = "poky" DISTRO_VERSION = "2.3" TUNE_FEATURES = "m32 i586" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "master:381897c64069ea43d595380a3ae913bcc79cf7e1" Initialising tasks: 100% |############################################################################################################################################| Time: 0:00:00 NOTE: Executing SetScene Tasks NOTE: Executing RunQueue Tasks NOTE: Tasks Summary: Attempted 470 tasks of which 469 didn't need to be rerun and all succeeded. Summary: There was 1 WARNING message shown. 10. Run buildhistory-collect-srcrevs to ensure it doesn't produce anything: $ buildhistory-collect-srcrevs 11. Clean all $ bitbake lz4 -c cleanall WARNING: Host distribution "opensuse-13.2" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution. Parsing recipes: 100% |###############################################################################################################################################| Time: 0:00:13 Parsing of 830 .bb files complete (0 cached, 830 parsed). 1299 targets, 48 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependencies Build Configuration: BB_VERSION = "1.34.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "universal-4.8" TARGET_SYS = "i586-poky-linux" MACHINE = "qemux86" DISTRO = "poky" DISTRO_VERSION = "2.3" TUNE_FEATURES = "m32 i586" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "master:381897c64069ea43d595380a3ae913bcc79cf7e1" Initialising tasks: 100% |############################################################################################################################################| Time: 0:00:00 NOTE: Executing RunQueue Tasks NOTE: Tasks Summary: Attempted 3 tasks of which 0 didn't need to be rerun and all succeeded. Summary: There was 1 WARNING message shown. 12. Build lz4 again: $ bitbake lz4 WARNING: Host distribution "opensuse-13.2" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution. Loading cache: 100% |#################################################################################################################################################| Time: 0:00:00 Loaded 1296 entries from dependency cache. Parsing recipes: 100% |###############################################################################################################################################| Time: 0:00:00 Parsing of 830 .bb files complete (829 cached, 1 parsed). 1299 targets, 48 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependencies Build Configuration: BB_VERSION = "1.34.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "universal-4.8" TARGET_SYS = "i586-poky-linux" MACHINE = "qemux86" DISTRO = "poky" DISTRO_VERSION = "2.3" TUNE_FEATURES = "m32 i586" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "master:381897c64069ea43d595380a3ae913bcc79cf7e1" Initialising tasks: 100% |############################################################################################################################################| Time: 0:00:00 Checking sstate mirror object availability: 100% |####################################################################################################################| Time: 0:00:00 NOTE: Executing SetScene Tasks NOTE: Executing RunQueue Tasks NOTE: Tasks Summary: Attempted 470 tasks of which 456 didn't need to be rerun and all succeeded. Summary: There was 1 WARNING message shown. 13. Run buildhistory-collect-srcrevs again: $ buildhistory-collect-srcrevs # i586-poky-linux SRCREV_pn-lz4 = "7bb64ff2b69a9f8367de9ab483cdadf42b4c1b65" Please, try to repeat above steps. It could be that the issue is only reproducible on your machine and/or your host distribution. If you're able to reproduce my results on default machine (qemux86) you can try to change machine and let me know which machine and distro you've used to reproduce the issue. 73287 17 jon.richardson 2017-05-16 21:54:12 +0000 I tried again using qemux86. It still doesn't work for me. It is reproducible on all of our ubuntu 14.04 machines. 73318 18 eduard.bartosh 2017-05-18 10:58:16 +0000 ok, thanks. I'll try to reproduce it on Ubuntu 14.04. 73322 19 eduard.bartosh 2017-05-18 14:28:29 +0000 I couldn't reproduce the issue on Ubuntu 16.04 it works the same way as on OpenSUSE. will try Ubuntu 14.04. 73374 20 eduard.bartosh 2017-05-19 09:10:41 +0000 It's the same on Ubuntu 14.04 - not reproducible. I repeated steps that I mentioned here before. The result is the same for the clean qemux86 build on 3 distros: OpenSUSE 13.2, Ubuntu 16.04 and Ubuntu 14.04 I used Ubuntu 14.04and 16.04 docker images from docker hub. Here is a full console log from Ubuntu 14.04: [ed@594ce71a39e0 poky (master)]$ . oe-init-build-env build-10948 You had no conf/local.conf file. This configuration file has therefore been created for you with some default values. You may wish to edit it to, for example, select a different MACHINE (target hardware). See conf/local.conf for more information as common configuration options are commented. You had no conf/bblayers.conf file. This configuration file has therefore been created for you with some default values. To add additional metadata layers into your configuration please add entries to conf/bblayers.conf. The Yocto Project has extensive documentation about OE including a reference manual which can be found at: http://yoctoproject.org/documentation For more information about OpenEmbedded see their website: http://www.openembedded.org/ ### Shell environment set up for builds. ### You can now run 'bitbake <target>' Common targets are: core-image-minimal core-image-sato meta-toolchain meta-ide-support You can also run generated qemu images with a command like 'runqemu qemux86' [ed@594ce71a39e0 build-10948 (master)]$ [ed@594ce71a39e0 build-10948 (master)]$ [ed@594ce71a39e0 build-10948 (master)]$ [ed@594ce71a39e0 build-10948 (master)]$ vi conf/local.conf [ed@594ce71a39e0 build-10948 (master)]$ vi /tmp/diff [ed@594ce71a39e0 build-10948 (master)]$ cd ../ [ed@594ce71a39e0 poky (master)]$ [ed@594ce71a39e0 poky (master)]$ [ed@594ce71a39e0 poky (master)]$ [ed@594ce71a39e0 poky (master)]$ [ed@594ce71a39e0 poky (master)]$ [ed@594ce71a39e0 poky (master)]$ patch -p1 < /tmp/diff patching file meta/recipes-support/lz4/lz4.bb Reversed (or previously applied) patch detected! Assume -R? [n] ^C [ed@594ce71a39e0 poky (master)]$ [ed@594ce71a39e0 poky (master)]$ [ed@594ce71a39e0 poky (master)]$ [ed@594ce71a39e0 poky (master)]$ cd build-10948/ [ed@594ce71a39e0 build-10948 (master)]$ [ed@594ce71a39e0 build-10948 (master)]$ vi conf/local.conf [ed@594ce71a39e0 build-10948 (master)]$ bitbake lz4 WARNING: Host distribution "ubuntu-14.04" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution. Parsing recipes: 100% |###############################################################################################################################################| Time: 0:00:20 Parsing of 830 .bb files complete (0 cached, 830 parsed). 1299 targets, 48 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependencies Build Configuration: BB_VERSION = "1.34.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "ubuntu-14.04" TARGET_SYS = "i586-poky-linux" MACHINE = "qemux86" DISTRO = "poky" DISTRO_VERSION = "2.3" TUNE_FEATURES = "m32 i586" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "master:381897c64069ea43d595380a3ae913bcc79cf7e1" NOTE: Fetching uninative binary shim from http://downloads.yoctoproject.org/releases/uninative/1.6/x86_64-nativesdk-libc.tar.bz2;sha256sum=2b4fffa308d9f19e0742a1a404ff42495fb50c165e5ca0458cedca157372691a --2017-05-19 08:07:51-- http://downloads.yoctoproject.org/releases/uninative/1.6/x86_64-nativesdk-libc.tar.bz2 Resolving proxy-chain.intel.com (proxy-chain.intel.com)... 163.33.7.139 Connecting to proxy-chain.intel.com (proxy-chain.intel.com)|163.33.7.139|:911... connected. Proxy request sent, awaiting response... 200 OK Length: 2535308 (2.4M) [application/octet-stream] Saving to: ‘/home/ed/git/yocto/poky/build-10948/downloads/uninative/2b4fffa308d9f19e0742a1a404ff42495fb50c165e5ca0458cedca157372691a/x86_64-nativesdk-libc.tar.bz2’ 2017-05-19 08:07:52 (3.04 MB/s) - ‘/home/ed/git/yocto/poky/build-10948/downloads/uninative/2b4fffa308d9f19e0742a1a404ff42495fb50c165e5ca0458cedca157372691a/x86_64-nativesdk-libc.tar.bz2’ saved [2535308/2535308] Initialising tasks: 100% |############################################################################################################################################| Time: 0:00:00 NOTE: Executing SetScene Tasks NOTE: Executing RunQueue Tasks WARNING: elfutils-native-0.168-r0 do_fetch: Failed to fetch URL ftp://sourceware.org/pub/elfutils/0.168/elfutils-0.168.tar.bz2, attempting MIRRORS if available WARNING: expat-native-2.2.0-r0 do_fetch: Failed to fetch URL http://downloads.sourceforge.net/expat/expat-2.2.0.tar.bz2, attempting MIRRORS if available NOTE: Tasks Summary: Attempted 470 tasks of which 0 didn't need to be rerun and all succeeded. Summary: There were 3 WARNING messages shown. [ed@594ce71a39e0 build-10948 (master)]$ [ed@594ce71a39e0 build-10948 (master)]$ [ed@594ce71a39e0 build-10948 (master)]$ [ed@594ce71a39e0 build-10948 (master)]$ [ed@594ce71a39e0 build-10948 (master)]$ buildhistory-collect-srcrevs # i586-poky-linux SRCREV_pn-lz4 = "7bb64ff2b69a9f8367de9ab483cdadf42b4c1b65" [ed@594ce71a39e0 build-10948 (master)]$ [ed@594ce71a39e0 build-10948 (master)]$ [ed@594ce71a39e0 build-10948 (master)]$ cp -fr sstate-cache sstate-cache.mirror [ed@594ce71a39e0 build-10948 (master)]$ vi conf/local.conf [ed@594ce71a39e0 build-10948 (master)]$ [ed@594ce71a39e0 build-10948 (master)]$ [ed@594ce71a39e0 build-10948 (master)]$ rm -rf buildhistory cache tmp [ed@594ce71a39e0 build-10948 (master)]$ bitbake lz4 WARNING: Host distribution "ubuntu-14.04" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution. Parsing recipes: 100% |###############################################################################################################################################| Time: 0:00:18 Parsing of 830 .bb files complete (0 cached, 830 parsed). 1299 targets, 48 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependencies Build Configuration: BB_VERSION = "1.34.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "ubuntu-14.04" TARGET_SYS = "i586-poky-linux" MACHINE = "qemux86" DISTRO = "poky" DISTRO_VERSION = "2.3" TUNE_FEATURES = "m32 i586" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "master:381897c64069ea43d595380a3ae913bcc79cf7e1" Initialising tasks: 100% |############################################################################################################################################| Time: 0:00:00 NOTE: Executing SetScene Tasks NOTE: Executing RunQueue Tasks NOTE: Tasks Summary: Attempted 470 tasks of which 469 didn't need to be rerun and all succeeded. Summary: There was 1 WARNING message shown. [ed@594ce71a39e0 build-10948 (master)]$ buildhistory-collect-srcrevs [ed@594ce71a39e0 build-10948 (master)]$ bitbake lz4 -c cleanall WARNING: Host distribution "ubuntu-14.04" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution. Parsing recipes: 100% |###############################################################################################################################################| Time: 0:00:13 Parsing of 830 .bb files complete (0 cached, 830 parsed). 1299 targets, 48 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependencies Build Configuration: BB_VERSION = "1.34.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "universal-4.8" TARGET_SYS = "i586-poky-linux" MACHINE = "qemux86" DISTRO = "poky" DISTRO_VERSION = "2.3" TUNE_FEATURES = "m32 i586" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "master:381897c64069ea43d595380a3ae913bcc79cf7e1" Initialising tasks: 100% |############################################################################################################################################| Time: 0:00:00 NOTE: Executing RunQueue Tasks NOTE: Tasks Summary: Attempted 3 tasks of which 0 didn't need to be rerun and all succeeded. Summary: There was 1 WARNING message shown. [ed@594ce71a39e0 build-10948 (master)]$ bitbake lz4 WARNING: Host distribution "ubuntu-14.04" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution. Loading cache: 100% |#################################################################################################################################################| Time: 0:00:00 Loaded 1296 entries from dependency cache. Parsing recipes: 100% |###############################################################################################################################################| Time: 0:00:00 Parsing of 830 .bb files complete (829 cached, 1 parsed). 1299 targets, 48 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependencies Build Configuration: BB_VERSION = "1.34.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "universal-4.8" TARGET_SYS = "i586-poky-linux" MACHINE = "qemux86" DISTRO = "poky" DISTRO_VERSION = "2.3" TUNE_FEATURES = "m32 i586" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "master:381897c64069ea43d595380a3ae913bcc79cf7e1" Initialising tasks: 100% |############################################################################################################################################| Time: 0:00:00 Checking sstate mirror object availability: 100% |####################################################################################################################| Time: 0:00:00 NOTE: Executing SetScene Tasks NOTE: Executing RunQueue Tasks NOTE: Tasks Summary: Attempted 470 tasks of which 456 didn't need to be rerun and all succeeded. Summary: There was 1 WARNING message shown. [ed@594ce71a39e0 build-10948 (master)]$ buildhistory-collect-srcrevs # i586-poky-linux SRCREV_pn-lz4 = "7bb64ff2b69a9f8367de9ab483cdadf42b4c1b65" Jonatan, can you try to repeat what I did? I mean to repeat exactly all steps. At this point I'm pretty sure we do different things. That's the only reason I can see why we have different results. Please, assign this bug back to me when you're done. Thanks. 73653 21 jon.richardson 2017-06-02 18:48:06 +0000 I'm not sure why we're seeing different results. I guess I need to debug it on our machines. Something is different. I tried again doing exactly what you did. The only part I can't see is what you changed SSTATE_MIRRORS to after this step: [ed@594ce71a39e0 build-10948 (master)]$ cp -fr sstate-cache sstate-cache.mirror [ed@594ce71a39e0 build-10948 (master)]$ vi conf/local.conf [ed@594ce71a39e0 build-10948 (master)]$ [ed@594ce71a39e0 build-10948 (master)]$ I switched to the master branch: jonathar@lbrmn-lnxub108:~/work/yocto/buildhistory-issue/poky$ git branch bcm/master * master jonathar@lbrmn-lnxub108:~/work/yocto/buildhistory-issue/poky$ git log commit 3dea90e8b7bfe988bc4bb70f0f0c50b4f74416bf Author: Paul Eggleton <paul.eggleton@linux.intel.com> Date: Mon May 15 09:29:56 2017 +1200 bitbake: lib/bb/fetch2: fix typo in conflicting revision message spcify -> specify (Bitbake rev: c46f183b3ea94789dd83cb280a1bf5735189c9bc) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> I built everything from scratch, it took a couple of hours. Same results. To give you a log I just used my sstate-cache from the first build and setup a new build dir: jonathar@lbrmn-lnxub108:~/work/yocto/buildhistory-issue/poky$ . oe-init-build-env build-10948 You had no conf/local.conf file. This configuration file has therefore been created for you with some default values. You may wish to edit it to, for example, select a different MACHINE (target hardware). See conf/local.conf for more information as common configuration options are commented. You had no conf/bblayers.conf file. This configuration file has therefore been created for you with some default values. To add additional metadata layers into your configuration please add entries to conf/bblayers.conf. The Yocto Project has extensive documentation about OE including a reference manual which can be found at: http://yoctoproject.org/documentation For more information about OpenEmbedded see their website: http://www.openembedded.org/ ### Shell environment set up for builds. ### You can now run 'bitbake <target>' Common targets are: core-image-minimal core-image-sato meta-toolchain meta-ide-support You can also run generated qemu images with a command like 'runqemu qemux86' jonathar@lbrmn-lnxub108:~/work/yocto/buildhistory-issue/poky/build-10948$ ll total 12 drwxrwxr-x 3 jonathar jonathar 4096 Jun 2 11:43 ./ drwxrwxr-x 19 jonathar jonathar 4096 Jun 2 11:43 ../ drwxrwxr-x 2 jonathar jonathar 4096 Jun 2 11:43 conf/ jonathar@lbrmn-lnxub108:~/work/yocto/buildhistory-issue/poky/build-10948$ bitbake lz4 WARNING: Host distribution "ubuntu-14.04" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution. Parsing recipes: 45% |######################################## | ETA: 0:00:15 Keyboard Interrupt, closing down... Parsing recipes: 100% |##########################################################################################| Time: 0:00:13 Parsing of 830 .bb files complete (0 cached, 384 parsed). 586 targets, 28 skipped, 0 masked, 0 errors. Summary: There was 1 WARNING message shown. Execution was interrupted, returning a non-zero exit code. jonathar@lbrmn-lnxub108:~/work/yocto/buildhistory-issue/poky/build-10948$ bitbake lz4 WARNING: Host distribution "ubuntu-14.04" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution. Parsing recipes: 100% |##########################################################################################| Time: 0:00:22 Parsing of 830 .bb files complete (0 cached, 830 parsed). 1299 targets, 48 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependencies Build Configuration: BB_VERSION = "1.34.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "ubuntu-14.04" TARGET_SYS = "i586-poky-linux" MACHINE = "qemux86" DISTRO = "poky" DISTRO_VERSION = "2.3" TUNE_FEATURES = "m32 i586" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "master:3dea90e8b7bfe988bc4bb70f0f0c50b4f74416bf" NOTE: Fetching uninative binary shim from http://downloads.yoctoproject.org/releases/uninative/1.6/x86_64-nativesdk-libc.tar.bz2;sha256sum=2b4fffa308d9f19e0742a1a404ff42495fb50c165e5ca0458cedca157372691a --2017-06-02 11:45:13-- http://downloads.yoctoproject.org/releases/uninative/1.6/x86_64-nativesdk-libc.tar.bz2 Resolving downloads.yoctoproject.org (downloads.yoctoproject.org)... 198.145.20.127 Connecting to downloads.yoctoproject.org (downloads.yoctoproject.org)|198.145.20.127|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 2535308 (2.4M) [application/octet-stream] Saving to: ‘/home/jonathar/work/yocto/buildhistory-issue/poky/build-10948/downloads/uninative/2b4fffa308d9f19e0742a1a404ff42495fb50c165e5ca0458cedca157372691a/x86_64-nativesdk-libc.tar.bz2’ 2017-06-02 11:45:15 (1.21 MB/s) - ‘/home/jonathar/work/yocto/buildhistory-issue/poky/build-10948/downloads/uninative/2b4fffa308d9f19e0742a1a404ff42495fb50c165e5ca0458cedca157372691a/x86_64-nativesdk-libc.tar.bz2’ saved [2535308/2535308] Initialising tasks: 100% |#######################################################################################| Time: 0:00:01 Checking sstate mirror object availability: 100% |###############################################################| Time: 0:00:00 NOTE: Executing SetScene Tasks NOTE: Executing RunQueue Tasks NOTE: Tasks Summary: Attempted 857 tasks of which 856 didn't need to be rerun and all succeeded. Summary: There was 1 WARNING message shown. jonathar@lbrmn-lnxub108:~/work/yocto/buildhistory-issue/poky/build-10948$ buildhistory-collect-srcrevs jonathar@lbrmn-lnxub108:~/work/yocto/buildhistory-issue/poky/build-10948$ jonathar@lbrmn-lnxub108:~/work/yocto/buildhistory-issue/poky/build-10948$ tail conf/local.conf #ASSUME_PROVIDED += "libsdl-native" # CONF_VERSION is increased each time build/conf/ changes incompatibly and is used to # track the version of this file when it was generated. This can safely be ignored if # this doesn't mean anything to you. CONF_VERSION = "1" SSTATE_MIRRORS="file://.* file:///home/jonathar/mirror/sstate-cache/PATH" INHERIT += "buildhistory" 73654 22 jon.richardson 2017-06-02 19:38:18 +0000 I added this first line to buildhistory.bbclass to get it to work. If it's cached and do_fetch doesn't run then how is it supposed to write the latest_srcrev file? do_populate_lic_setscene[postfuncs] += "write_srcrev" do_fetch[postfuncs] += "write_srcrev" do_fetch[vardepsexclude] += "write_srcrev" python write_srcrev() { When it's cached the do_fetch task isn't run: jonathar@lbrmn-lnxub108:~/work/yocto/buildhistory-issue/poky/build-10948$ cat tmp/work/i586-poky-linux/lz4/1_gitAUTOINC+7bb64ff2b6-r0/temp/log.task_order do_populate_lic_setscene (12787): log.do_populate_lic_setscene.12787 do_package_write_rpm_setscene (12927): log.do_package_write_rpm_setscene.12927 do_populate_sysroot_setscene (12934): log.do_populate_sysroot_setscene.12934 do_package_qa_setscene (12957): log.do_package_qa_setscene.12957 do_packagedata_setscene (13248): log.do_packagedata_setscene.13248 73669 23 eduard.bartosh 2017-06-03 21:08:40 +0000 do_fetch was rerun on all my tests mentioned above after bitbake lz4 -c cleanall. It's still a mistery for me why it's not rerun in your setup. 73797 24 eduard.bartosh 2017-06-07 14:10:27 +0000 Can you try this patch, please? diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass index 3e907fc..057a3cb 100644 --- a/meta/classes/buildhistory.bbclass +++ b/meta/classes/buildhistory.bbclass @@ -336,6 +336,7 @@ def write_recipehistory(rcpinfo, d): f.write(u"PACKAGES = %s\n" % rcpinfo.packages) f.write(u"LAYER = %s\n" % rcpinfo.layer) + write_latest_srcrev(d, pkghistdir) def write_pkghistory(pkginfo, d): bb.debug(2, "Writing package history for package %s" % pkginfo.name) @@ -853,7 +854,10 @@ def _get_srcrev_values(d): do_fetch[postfuncs] += "write_srcrev" do_fetch[vardepsexclude] += "write_srcrev" python write_srcrev() { - pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE') + write_latest_srcrev(d, d.getVar('BUILDHISTORY_DIR_PACKAGE')) +} + +def write_latest_srcrev(d, pkghistdir): srcrevfile = os.path.join(pkghistdir, 'latest_srcrev') srcrevs, tag_srcrevs = _get_srcrev_values(d) @@ -891,4 +895,3 @@ python write_srcrev() { else: if os.path.exists(srcrevfile): os.remove(srcrevfile) -} I can't reproduce the issue. Hopefully forcing of srcrev writing can fix it. 73927 25 jon.richardson 2017-06-08 22:28:49 +0000 The fix works for me. Thanks. jonathar@lbrmn-lnxub108:~/work/yocto/buildhistory-issue/poky/build-10948$ rm -rf buildhistory/ cache/ tmp/ jonathar@lbrmn-lnxub108:~/work/yocto/buildhistory-issue/poky/build-10948$ jonathar@lbrmn-lnxub108:~/work/yocto/buildhistory-issue/poky/build-10948$ bitbake lz4 WARNING: Host distribution "ubuntu-14.04" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution. Parsing recipes: 100% |##########################################################################################| Time: 0:00:28 Parsing of 830 .bb files complete (0 cached, 830 parsed). 1299 targets, 48 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependencies Build Configuration: BB_VERSION = "1.34.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "ubuntu-14.04" TARGET_SYS = "i586-poky-linux" MACHINE = "qemux86" DISTRO = "poky" DISTRO_VERSION = "2.3" TUNE_FEATURES = "m32 i586" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "master:3dea90e8b7bfe988bc4bb70f0f0c50b4f74416bf" Initialising tasks: 100% |#######################################################################################| Time: 0:00:02 Checking sstate mirror object availability: 100% |###############################################################| Time: 0:00:00 NOTE: Executing SetScene Tasks NOTE: Executing RunQueue Tasks NOTE: Tasks Summary: Attempted 857 tasks of which 856 didn't need to be rerun and all succeeded. Summary: There was 1 WARNING message shown. jonathar@lbrmn-lnxub108:~/work/yocto/buildhistory-issue/poky/build-10948$ buildhistory-collect-srcrevs # i586-poky-linux SRCREV_pn-lz4 = "7bb64ff2b69a9f8367de9ab483cdadf42b4c1b65" jonathar@lbrmn-lnxub108:~/work/yocto/buildhistory-issue/poky/build-10948$ cat tmp/work/ i586-poky-linux/ qemux86-poky-linux/ x86_64-linux/ jonathar@lbrmn-lnxub108:~/work/yocto/buildhistory-issue/poky/build-10948$ cat tmp/work/i586-poky-linux/ bash/ expat/ glibc/ lz4/ perl/ sqlite3/ bc/ flex/ glibc-locale/ m4/ ptest-runner/ xz/ bzip2/ gawk/ libffi/ make/ python/ zlib/ cryptodev-linux/ gcc-runtime/ libgcc/ ncurses/ python3/ db/ gdbm/ libtool-cross/ openssl/ readline/ diffutils/ gettext/ linux-libc-headers/ opkg-utils/ sed/ jonathar@lbrmn-lnxub108:~/work/yocto/buildhistory-issue/poky/build-10948$ cat tmp/work/i586-poky-linux/lz4/1_gitAUTOINC+7bb64ff2b6-r0/temp/log.task_order do_populate_lic_setscene (22856): log.do_populate_lic_setscene.22856 do_populate_sysroot_setscene (22857): log.do_populate_sysroot_setscene.22857 do_package_write_rpm_setscene (23062): log.do_package_write_rpm_setscene.23062 do_package_qa_setscene (23063): log.do_package_qa_setscene.23063 do_packagedata_setscene (23142): log.do_packagedata_setscene.23142 jonathar@lbrmn-lnxub108:~/work/yocto/buildhistory-issue/poky/build-10948$ tail conf/local.conf #ASSUME_PROVIDED += "libsdl-native" # CONF_VERSION is increased each time build/conf/ changes incompatibly and is used to # track the version of this file when it was generated. This can safely be ignored if # this doesn't mean anything to you. CONF_VERSION = "1" SSTATE_MIRRORS="file://.* file:///home/jonathar/mirror/sstate-cache/PATH" INHERIT += "buildhistory" jonathar@lbrmn-lnxub108:~/work/yocto/buildhistory-issue/poky/build-10948$ 73935 26 eduard.bartosh 2017-06-09 11:51:34 +0000 Patch has been sent for review: http://lists.openembedded.org/pipermail/openembedded-core/2017-June/137836.html 74041 27 eduard.bartosh 2017-06-13 08:09:44 +0000 The fix has been merged to poky master.