Bug 7563

Summary:	bitbake: printdiff looks for siginfo in incorrect locations for some tasks
Product:	[Build System, Metadata & Runtime] BitBake	Reporter:	Christopher Larson <kergoth>
Component:	bitbake	Assignee:	Christopher Larson <kergoth>
Status:	RESOLVED FIXED	QA Contact:
Severity:	major
Priority:	Undecided	CC:	poky.bs.watcher, poky.watcher
Version:	unspecified
Target Milestone:	---
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Christopher Larson 2015-04-02 21:13:49 UTC

Specifically with quilt-native (my test case due to it having no dependencies), I'm seeing siginfo written out for 'quilt' for the fetch, unpack, and patch tasks, whereas the siginfo for all the other tasks is written out for 'quilt-native' (going by the on-disk location, anyway).

I believe this is the cause of why I've seen useless -S printdiff which outputs a list of fetch tasks, but no details -- the siginfo wasn't available in the expected place.


Example:

› bitbake --no-setscene -c populate_sysroot quilt-native
Parsing recipes: 100% |##################################################################################################################################################################| Time: 00:01:22
Parsing of 884 .bb files complete (0 cached, 884 parsed). 1285 targets, 41 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION        = "1.27.0"
BUILD_SYS         = "i686-linux"
NATIVELSBSTRING   = "CentOS-6.5"
TARGET_SYS        = "i586-poky-linux"
MACHINE           = "qemux86"
DISTRO            = "poky"
DISTRO_VERSION    = "1.8+snapshot-20150402"
TUNE_FEATURES     = "m32 i586"
TARGET_FPU        = ""
meta
meta-yocto
meta-yocto-bsp    = "(nobranch):fe9c5b1ba9f95e2321093eea1211bf5655eb6966"

NOTE: Preparing RunQueue
NOTE: Executing RunQueue Tasks
NOTE: Tasks Summary: Attempted 7 tasks of which 0 didn't need to be rerun and all succeeded.
  321.32s user 3.88s system 350% cpu 1:32.91 total

› find sstate-cache
sstate-cache
sstate-cache/8c
sstate-cache/8c/sstate:quilt::0.64:r0::3:8c862fc766e8165a9149ba31d441fc58_patch.tgz.siginfo
sstate-cache/fb
sstate-cache/fb/sstate:quilt::0.64:r0::3:fb9378edacb1e2e512888cc63f789cce_fetch.tgz.siginfo
sstate-cache/c1
sstate-cache/c1/sstate:quilt::0.64:r0::3:c16c7a79a6cd95766dc26de62d769f7d_unpack.tgz.siginfo
sstate-cache/CentOS-6.5
sstate-cache/CentOS-6.5/3b
sstate-cache/CentOS-6.5/3b/sstate:quilt-native:i686-linux:0.64:r0:i686:3:3b36105dc93bf593a3496b28af5886f2_install.tgz.siginfo
sstate-cache/CentOS-6.5/78
sstate-cache/CentOS-6.5/78/sstate:quilt-native:i686-linux:0.64:r0:i686:3:78bd3c7485a2952c583b6021c4935e12_populate_sysroot.tgz.siginfo
sstate-cache/CentOS-6.5/78/sstate:quilt-native:i686-linux:0.64:r0:i686:3:78bd3c7485a2952c583b6021c4935e12_populate_sysroot.tgz
sstate-cache/CentOS-6.5/fd
sstate-cache/CentOS-6.5/fd/sstate:quilt-native:i686-linux:0.64:r0:i686:3:fd70a6d546932acf0e31af269bf1f721_compile.tgz.siginfo
sstate-cache/CentOS-6.5/f3
sstate-cache/CentOS-6.5/f3/sstate:quilt-native:i686-linux:0.64:r0:i686:3:f3987a6cc7328f9e4a7b13f7da8d1275_configure.tgz.siginfo

Comment 1 Christopher Larson 2015-04-02 21:30:42 UTC

Confirmed that what's written out and what printdiff expects don't match up for the fetch/unpack/patch tasks. If I build quilt-native, then modify its do_unpack, wipe tmp, and then run -S printdiff, it gives no useful detail:

NOTE: Preparing RunQueue
The differences between the current build and any cached tasks start at the following tasks:
/scratch/testbuild/poky2/meta/recipes-devtools/quilt/quilt-native_0.64.bb, do_unpack
NOTE: Reparsing files to collect dependency data
Writing locked sigs to /scratch/testbuild/poky2/build/locked-sigs.inc
NOTE: Tasks Summary: Attempted 0 tasks of which 0 didn't need to be rerun and all succeeded.

If I then cp -a the non-native siginfo into the NATIVELSBSTRING path, and correct their filenames to 'quilt-native:i686-linux' rather than 'quilt:', I get this, which is useful:

NOTE: Preparing RunQueue
The differences between the current build and any cached tasks start at the following tasks:
/scratch/testbuild/poky2/meta/recipes-devtools/quilt/quilt-native_0.64.bb, do_unpack
NOTE: Reparsing files to collect dependency data
Writing locked sigs to /scratch/testbuild/poky2/build/locked-sigs.inc

Task quilt-native:do_unpack couldn't be used from the cache because:
  We need hash 95848187e61c749c35950710d814039d, closest matching task was c16c7a79a6cd95766dc26de62d769f7d
  Task dependencies changed from:
['BP', 'BPN', 'PN', 'PV', 'S', 'SAVANNAH_GNU_MIRROR', 'SPECIAL_PKGSUFFIX', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_unpack', 'base_prune_suffix']
to:
['BP', 'BPN', 'PN', 'PV', 'S', 'SAVANNAH_GNU_MIRROR', 'SPECIAL_PKGSUFFIX', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_unpack', 'base_prune_suffix', 'do_something']
  basehash changed from 144921aa0aa4d2a91da897867d286d18 to 7019e9b621627c6b5001dedfe89a6eac
  Dependency on variable do_something was added
NOTE: Tasks Summary: Attempted 0 tasks of which 0 didn't need to be rerun and all succeeded.

Comment 2 Christopher Larson 2015-04-02 21:36:11 UTC

From reading sstate.bbclass, it seems this was by design for the following tasks: "fetch", "unpack", "patch", "populate_lic", "preconfigure", but at least printdiff doesn't seem to be picking those up when dumping the scene information.

Comment 3 Christopher Larson 2015-04-02 21:53:47 UTC

Working on a fix for sstatesig.py now.

Comment 4 Christopher Larson 2015-04-02 22:26:53 UTC

Currently doing more testing of https://gist.github.com/kergoth/3713d779c14dc8b98f36 before submission to the oe-core list.

Comment 5 Christopher Larson 2015-04-02 23:17:40 UTC

Bumping importance, as being able to diagnose why something is building from scratch rather than using the sstate-cache is, from my perspective at least, a pretty important piece of functionality.

Comment 6 Christopher Larson 2015-04-03 22:05:47 UTC

Patch emailed to the oe-core list.

Comment 7 Christopher Larson 2015-04-08 21:34:12 UTC

Richard merged this, so resolving.