Bug 1721

Summary: [sstate] do_populate_sysroot_setscene race condition with useradd recipes
Product: [Build System, Metadata & Runtime] Meta-yocto Reporter: Jiajun Xu <jiajun.xu>
Component: meta-yoctoAssignee: Scott Garman <scott.a.garman>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium CC: bluelightning, edwin.zhai, josh, ke.yu, msm-oss, poky.bs.watcher, poky.watcher, richard.purdie, sgw, yi.zhao
Version: unspecified   
Target Milestone: 1.2 M3   
Hardware: x86   
OS: Multiple   
Whiteboard: 2012-01-03: 0% Implemented
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: ---

Description Jiajun Xu 2011-11-02 23:59:51 UTC
MACHINE: 
qemux86/atom-pc/qemux86-64/mpc8315e-rdb/qemuppc/routerstationpro/qemumips/beagleboard/qemuarm
poky/branch:master 9b76e6a2cfc5a4d779f3b06e3acc5ff7b8275470

On latest autobuilder, I find that some recipes report failed for do_populate_sysroot_setscene. These recipes include avahi,dbus,shadow-sysroot,avahi-ui.

You could find detailed logs on autobuilder. For example, you could find qemux86 logs on http://autobuilder.pokylinux.org:8010/builders/nightly-x86/builds/204/steps/shell_12/logs/stdio.

log as below:
#####
ERROR: Task 1293 (/srv/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/meta/recipes-connectivity/avahi/avahi_0.6.30.bb, do_populate_sysroot_setscene) failed with exit code '1'
...
...
...
ERROR: Task 1429 (/srv/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/meta/recipes-core/dbus/dbus_1.4.12.bb, do_populate_sysroot_setscene) failed with exit code '1'
NOTE: Running setscene task 2127 of 2830 (/srv/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/meta/recipes-sato/pimlico/tasks_0.19.bb:do_populate_lic_setscene)
NOTE: package dbus-1.4.12-r7: task do_populate_lic_setscene: Succeeded
#####
Comment 1 Saul Wold 2011-11-03 15:39:09 UTC
My guess is this is related to useradd
Comment 2 Paul Eggleton 2011-11-17 01:51:32 UTC
Yes, it seems that way to me also.

dbus fails with:
-----------------
Running groupadd commands...
grep: /media/large/tmp2/sysroots/qemux86/etc/group: No such file or directory
pseudo: pseudo: can't run groupadd: Permission denied
-----------------

shadow-sysroot:
-----------------
DEBUG: Staging files from /media/large/tmp2/work/i586-poky-linux/shadow-sysroot-4.1.4.3-r0/sysroot-destdir/ to /media/large/tmp2/sysroots/qemux86/
install: cannot stat `/media/large/tmp2/work/i586-poky-linux/shadow-sysroot-4.1.4.3-r0/image/etc/login.defs': No such file or directory
-----------------
Comment 3 Scott Garman 2011-11-17 16:08:46 UTC
Paul: what git rev did you see this with most recently? I was under the impression that this commit should have fixed the issue:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=d0151bcd7bc8124fc9c448a01eb0e093c4f23250
Comment 4 Paul Eggleton 2011-11-18 07:29:44 UTC
Scott - that fix only affects packaging, this is at sysroot_setscene time. I'm not sure what revision I was at when I did this but it was quite recently.
Comment 5 Paul Eggleton 2011-11-22 12:37:38 UTC
I've confirmed this is still happening with current master as at revision 3ab018c6254b883a6b6a61f79405dc3f8e40ad77.
Comment 6 Scott Garman 2011-11-22 13:25:46 UTC
Thanks for the follow-ups, I'm trying to reproduce the bug now.
Comment 7 Richard Purdie 2011-11-24 10:59:57 UTC
To reproduce, you need a decent sstate cache and an empty TMPDIR. If you then build something which includes dbus/avahi, you'll see the setcenes for their do_populate_sysroot functions fail. Its due to a race, they're happening before base-passwd is installed into the sysroot.

Unfortunately there aren't dependencies between setscene tasks so we're going to have to find a more creative way to get this working without races. Perhaps we could touch the group/passwd files if missing and later call update-passwd against them in base-passwd's setscene function?
Comment 8 Matthew McClintock 2012-01-16 14:21:09 UTC
Also, are these really ERROR's? I think it recovers and just reruns the tasks in the RunQueue? Can we change this to something like 'SSTATE FAILURE:' instead of 'ERROR:'? 

-M
Comment 9 Richard Purdie 2012-01-27 01:44:26 UTC
Resolved with the patches ending with http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=75fcfde0617cf528a9506fbdc6f7f97e56298bbe
Comment 10 Matthew McClintock 2012-01-27 18:35:04 UTC
(In reply to comment #9)
> Resolved with the patches ending with
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=75fcfde0617cf528a9506fbdc6f7f97e56298bbe

These appear to have applied cleanly to Josh's edison branch. The one below had some small conflicts from the bb.data -> d rename.

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=9a58e312ef596739923f4bc44ff8ecadaa88ddd9

Completed some build tests and things appear to be working.

-M