Bug 6124

Summary: util-macros.do_package failed: 'getpwnam(): name not found: root'
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Robert Yang <liezhi.yang>
Component: coreAssignee: Robert Yang <liezhi.yang>
Status: VERIFIED FIXED QA Contact: Costin <costin.c.constantin>
Severity: normal    
Priority: Medium+ CC: alexandru.c.georgescu, jessica.zhang, kergoth, mark.hatle, meta.mr.watcher, meta.watcher, peter.kjellerstedt, richard.purdie, ross.burton, sgw
Version: 1.6   
Target Milestone: 1.7 M2   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Robert Yang 2014-04-05 08:21:23 UTC
We seldom met this error, there are some rough investigations at the end of the below:

meta/recipes-graphics/xorg-util/util-macros_1.18.0.bb:

The stack trace of python calls that resulted in this exception/failure was:
File: 'fixup_perms', lineno: 231, function: <module>
     0227:                    each_file = os.path.join(root, f)
     0228:                    fix_perms(each_file, fs_perms_table[dir].fmode, fs_perms_table[dir].fuid, fs_perms_table[dir].fgid, dir)
     0229:
     0230:
 *** 0231:fixup_perms(d)
     0232:
File: 'fixup_perms', lineno: 173, function: fixup_perms
     0169:                if len(lsplit) != 8 and not (len(lsplit) == 3 and lsplit[1].lower() == "link"):
     0170:                    msg = "Fixup perms: %s invalid line: %s" % (conf, line)
     0171:                    package_qa_handle_error("perm-line", msg, d)
     0172:                    continue
 *** 0173:                entry = fs_perms_entry(d.expand(line))
     0174:                if entry and entry.path:
     0175:                    fs_perms_table[entry.path] = entry
     0176:            f.close()
     0177:
File: 'fixup_perms', lineno: 32, function: __init__
  File "fixup_perms", line 32, in __init__

File: 'fixup_perms', lineno: 43, function: _setdir
  File "fixup_perms", line 43, in _setdir

File: 'fixup_perms', lineno: 67, function: _procuid
  File "fixup_perms", line 67, in _procuid

Exception: KeyError: 'getpwnam(): name not found: root'

DEBUG: Python function fixup_perms finished
DEBUG: Python function do_package finished
ERROR: Function failed: fixup_perms

I think that the problem is happened when etc/passwd is being created but is null before write anything into it, and the util-macros.do_package runs right at the same time, here is a hack way to reproduce it: (Use qemux86 as an example):

$ bitbake base-passwd

# Move the passwd file
$ mv tmp/sysroots/qemux86/etc/passwd tmp/sysroots/qemux86/etc/passwd.bak

# Create a passwd file which is null
$ touch tmp/sysroots/qemux86/etc/passwd

$ bitbake util-macros

Then we will see the errors.

The build is be OK if there is no tmp/sysroots/qemux86/etc/passwd, maybe
the pwd.getpwnam() reads the host's /etc/passwd when there is no
tmp/sysroots/qemux86/etc/passwd (not sure about this). but it would fail
if it exists and is null.

Maybe we can let the do_package depend on base-passwd:do_populate_sysroot
to fix the problem.
Comment 1 Robert Yang 2014-04-05 08:23:32 UTC
Here are the discussions from the mailing list:

On Fri, 2014-04-04 at 21:01 +0100, Burton, Ross wrote:
> On 4 April 2014 18:18, Robert Yang <liezhi.yang@windriver.com> wrote:
>> Maybe we can let the do_package depend on base-passwd:do_populate_sysroot
>> to fix the problem.
>
> I keep on thinking that we need to effectively have a readers/writer
> lock on the sysroot, so that e.g. do_compile() takes a read lock and
> do_populate_sysroot() takes a write lock.  This way very slim races
> such as this (and aclocal disappearing, and binaries being re-written
> whilst being executed, etc) just can't happen.  By having a
> readers/writer lock the stalls for the actual populate of the sysroot
> shouldn't have too great an impact, and would let us remove other
> workarounds we've added (such as the baroque aclocal-copy).

We've tried this before and "shouldn't" doesn't work out too well in
practice.

The aclocal-copy business is actually now an improvement since it only
accesses things it depends upon which is a good thing as far as I'm
concerned. Other ways we could possibly improve:

a) Atomically moves files into the sysroot (and I guess move and delete
for atomic removal). This means they can't "half" be there.

b) Move libs in before binaries. This way a binary can't get run before
its dependencies are there.

This all continues to hack around the fact its a shared area though and
I'd imagine regardless of what tricks you play and even if you add in
locks, there will still be determinism issues.

The conclusion you then reach is that the only way you can ultimately
solve things is a sysroot per build item. You could do something that
would perform moderately well with a directory per sysroot item, then
hardlink these into individual sysroots per workdir.

In many ways, the aclocal-copy change is an experiment for that, we
should now have the pieces where we could implement such a thing...

Cheers,

Richard
Comment 2 Robert Yang 2014-04-10 02:37:16 UTC
Comments from RP:

This is very similar to the issue:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=5612

I think I'm leaning in favour of putting in a basic passwd/group file
into the sysroot, even if we have to maintain such a copy ourselves.

Cheers,

Richard
Comment 3 Mark Hatle 2014-04-10 15:44:35 UTC
One other option that was discussed a while back.

Add a minimum set of known users and groups into pseudo.  So even if it does open the file (and find it empty) it still has the minimum set.

This minimum set could be programmed at compile time or runtime.  The minimum set I'd suggest would be simply "root:root" -- "0:0".

And Robert is correct, pseudo, when it can't find the passwd/group file simply falls back to the host system's.  If we were to provide a minimum set, that fall back should likely be disabled.
Comment 4 Robert Yang 2014-04-18 07:54:06 UTC
*** Bug 5612 has been marked as a duplicate of this bug. ***
Comment 6 Robert Yang 2014-07-11 09:59:55 UTC
(In reply to comment #5)
> Merged into master:
> 
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/
> ?id=1cb9f7b00b5ede62a9546c39d798c876775b08c0
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/
> ?id=7e6eeeefdd0a92df5203d54a9b574957ed2a25ce

Should be:
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=949bf156ca95153485c41698de4b4dd661b0f6dd
Comment 7 Costin 2014-08-04 14:30:41 UTC
Hello,

Just verified with commit 949bf156ca95153485c41698de4b4dd661b0f6dd and the issue is solved now. 

Regards,  

Costin