Bug 1700

Summary: buildstats errors block all bitbake use on encypted home dir (or other non-direct disk FS)
Product: [Build System, Metadata & Runtime] Meta-yocto Reporter: Bill Mills <wmills>
Component: meta-yoctoAssignee: Beth Flanagan <elizabeth.flanagan>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium CC: elizabeth.flanagan, poky.bs.watcher, poky.watcher, richard.purdie, sgw
Version: 1.1-beta   
Target Milestone: 1.2   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: ---

Description Bill Mills 2011-10-25 04:16:27 UTC 
Using an encyptfs home dir for build dir creates this issue
I suspect any non local direct disk filesystem would cause the same (aufs, nfs, etc)

buildstat's BuildStarted handler causes the issue as it can't file a direct disk mapping for <build dir>/tmp and does not assign rdev.  It then throws an exception when rdev is accessed.  The build continues at this point.

buildstat's BuildCompleted handler then does not find what should have been created by BuildStarted and throws it own exception.

So the build of Pseudo completes all tasks but I assume won't return 0 rc due to the exceptions.  So Pseudo is never marked present and no real bitbake work gets started.

Moving the build dir to /tmp/xxx did resolve this issue.

We may decide that build dirs on encyptfs is not a good idea but this is not the way to tell people.  The buildstat code needs to get fixed at any rate.


Host OS: Ubuntu 10.04 LTS, 32 bit
Yocto 1.2/poky 6.0 from release tarball

log from subsequent runs show the error the best:
----
bill@wmills-laptop:~/w/proj/yocto/bs-test1 [rc=1] $ bitbake core-image-minimal
Pseudo is not present but is required, building this first before the main build
Loading cache: 100% |###################################################################################| ETA:  00:00:00
Loaded 1036 entries from dependency cache.
ERROR: Execution of event handler 'run_buildstats' failed
Traceback (most recent call last):
  File "run_buildstats(e)", line 18, in run_buildstats(e=<bb.event.BuildStarted object at 0x9bed6ec>)
  File "buildstats.bbclass", line 21, in set_device(e=<bb.event.BuildStarted object at 0x9bed6ec>)
UnboundLocalError: local variable 'rdev' referenced before assignment


OE Build Configuration:
BB_VERSION        = "1.13.3"
TARGET_ARCH       = "i586"
TARGET_OS         = "linux"
MACHINE           = "qemux86"
DISTRO            = "poky"
DISTRO_VERSION    = "1.1"
TUNE_FEATURES     = "m32 i586"
TARGET_FPU        = ""
meta              
meta-yocto        = "<unknown>:<unknown>"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Tasks Summary: Attempted 126 tasks of which 126 didn't need to be rerun and 0 failed.
ERROR: Execution of event handler 'run_buildstats' failed
Traceback (most recent call last):
  File "run_buildstats(e)", line 46, in run_buildstats(e=<bb.event.BuildCompleted object at 0xb5403d4c>)
IOError: [Errno 2] No such file or directory: '/home/bill/w/proj/yocto/bs-test1/tmp/buildstats/pseudo-native-qemux86/201110250652/build_stats'
Comment 1 Bill Mills 2011-10-25 13:09:07 UTC 
This is Yocto 1.1 final / poky 6.0.  Not 1.2 as marked below.
I marked this 1.1-beta in bugzilla as there is no 1.1 final right now.
Comment 2 Beth Flanagan 2011-10-26 10:10:19 UTC 
This isn't happening on nfs, at least not on the autobuilders. 

Bill, could you cat out your /proc/diskstats, noting the encyptfs device?
Comment 3 Richard Purdie 2011-10-26 13:21:14 UTC 
Let me split this into two more definite issues:

a) the first time it runs, the build will fail with the sanity check. The second time it will happily skip the sanity check. This needs to get fixed so it always fails.
b) buildstats should fail more gracefully, not a traceback but with something more human readable.
Comment 4 Beth Flanagan 2011-10-31 10:02:04 UTC 
I tried this on an encyptfs partition and couldn't reproduce. Bill, could you get me:

What your system is (Distro/version/etc)
Get me a cat out of your /proc/diskstats

I agree that buildstats should fail on this a bit more gracefully, but I'd like to also figure out why this is failing for Bill and not me.
Comment 5 Saul Wold 2011-10-31 10:53:42 UTC 
This also seems to fail using tmpfs
Comment 6 Beth Flanagan 2011-10-31 14:03:12 UTC 
The problem is that only logical devices end up with a /proc/diskstats entry. I weed out any partitions that are not on a logical drive and just avoid collecting disk statistics for them. I'll have the patch submitted tonight after testing on tmpfs.