Bug 2779

This is happening every couple of days in our test builds. It's clearly not a good idea to just spin a number of times, and then fail. A longer timeout might make the problem go away for a while but I think that Jackie is going to implement a blocking lock.

I bumped the priority in hopes that this defect, the intended solution and it's review get somewhat faster attention.

We changed the locking for the native groupadd and useradd tools to block indefinitely rather than arbitrarily increasing the retry count or extending the timeout, and added debug messages with the patch from debian:
http://lists.alioth.debian.org/pipermail/pkg-shadow-commits/2011-July/003444.html
so that we may be able to find out the root cause, but the failure didn't happen again since then. So I think we need more info for this.

Hack work around patch:

http://gate.crashing.org/~fray/shadow-native-block-groupadd-and-useradd.patch

So I am seeing this on:

[mattsm@busy ~]$ cat /etc/issue
CentOS release 5.8 (Final)
Kernel \r on an \m

[mattsm@busy ~]$ uname -a
Linux busy.am.freescale.net 2.6.18-238.12.1.el5 #1 SMP Tue May 31 13:22:04 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

When the machine is heavily bogged down.

What type of distros/kernels are you guys seeing it on?

-M

All of our coverage builders have the same configuration:

CentOS 5.8 x86_64
Linux yow-blade1 2.6.18-308.16.1.el5 #1 SMP Tue Oct 2 22:01:43 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux

Hardware: 8-12 cores and 32 to 96GB of ram, large local disk.

Each coverage builder is running 2 builds in parallel with 20 parallel packages and 20 jobs, so the machines are often heavily loaded.

Let me know if you need any more details.

(In reply to comment #5)
> All of our coverage builders have the same configuration:
> 
> CentOS 5.8 x86_64
> Linux yow-blade1 2.6.18-308.16.1.el5 #1 SMP Tue Oct 2 22:01:43 EDT 2012
> x86_64 x86_64 x86_64 GNU/Linux
> 
> Hardware: 8-12 cores and 32 to 96GB of ram, large local disk.
> 
> Each coverage builder is running 2 builds in parallel with 20 parallel
> packages and 20 jobs, so the machines are often heavily loaded.
> 
> Let me know if you need any more details.

Very similar to our setup... I don't really have other ideas though

-M

At the time that these failures were causing problems, I thought we were using a central disk array that was shared between 4-6 builders. The guess was that we had either an iSCSI protocol timeout of some lower level occasional hardware timeout but we were never able to prove it. We had to power cycle the systems
and the problem didn't occur again in the first few days. Due to work schedules, we applied the patch so that if the hang-up occurred, we'd get some log data and the hang wouldn't break a build. So as of a couple of months ago we did see some logs indicating log delays but since the build doesn't break we have mostly ignored the problem.

I see that you have the patch but it's not clear if you need the other bits:
git show d985472e0fc5f04bac49159f697e43bb349648e4 | diffstat
 classes/useradd.bbclass                                                                   |  182 ++++++++
 recipes-extended/shadow/shadow-native_4.1.4.3.bbappend                                    |    7 
 recipes-extended/shadow/shadow/add-additional-messages-to-indicate-why-locking-fail.patch |  217 ++++++++++
 recipes-extended/shadow/shadow/shadow-native-block-groupadd-and-useradd.patch             |  196 +++++++++
 4 files changed, 602 insertions(+)

If one really wanted to understand this problem then the thing to do is to run on a kernel that has a "flight recorder" and capture the recent events when the timeout occurs.

(In reply to comment #7)
> At the time that these failures were causing problems, I thought we were
> using a central disk array that was shared between 4-6 builders. The guess
> was that we had either an iSCSI protocol timeout of some lower level
> occasional hardware timeout but we were never able to prove it. We had to
> power cycle the systems
> and the problem didn't occur again in the first few days. Due to work
> schedules, we applied the patch so that if the hang-up occurred, we'd get
> some log data and the hang wouldn't break a build. So as of a couple of
> months ago we did see some logs indicating log delays but since the build
> doesn't break we have mostly ignored the problem.
> 
> I see that you have the patch but it's not clear if you need the other bits:
> git show d985472e0fc5f04bac49159f697e43bb349648e4 | diffstat
>  classes/useradd.bbclass                                                    
> |  182 ++++++++
>  recipes-extended/shadow/shadow-native_4.1.4.3.bbappend                     
> |    7 
>  recipes-extended/shadow/shadow/add-additional-messages-to-indicate-why-
> locking-fail.patch |  217 ++++++++++
>  recipes-extended/shadow/shadow/shadow-native-block-groupadd-and-useradd.
> patch             |  196 +++++++++
>  4 files changed, 602 insertions(+)
> 
> If one really wanted to understand this problem then the thing to do is to
> run on a kernel that has a "flight recorder" and capture the recent events
> when the timeout occurs.

That alone might prevent the error from occurring.

We are using a local RAID 5 array, so it's less likely to be a iSCSI issue.

-M

(In reply to comment #8)
> (In reply to comment #7)
> > At the time that these failures were causing problems, I thought we were
> > using a central disk array that was shared between 4-6 builders. The guess
> > was that we had either an iSCSI protocol timeout of some lower level
> > occasional hardware timeout but we were never able to prove it. We had to
> > power cycle the systems
> > and the problem didn't occur again in the first few days. Due to work
> > schedules, we applied the patch so that if the hang-up occurred, we'd get
> > some log data and the hang wouldn't break a build. So as of a couple of
> > months ago we did see some logs indicating log delays but since the build
> > doesn't break we have mostly ignored the problem.

The build doesn't break anymore with the patch, and looks like it never happen in yocto's testing, I think it would be fine to close it. I will reopen it if it comes out to a problem again.

> > 
> > I see that you have the patch but it's not clear if you need the other bits:
> > git show d985472e0fc5f04bac49159f697e43bb349648e4 | diffstat
> >  classes/useradd.bbclass                                                    
> > |  182 ++++++++
> >  recipes-extended/shadow/shadow-native_4.1.4.3.bbappend                     
> > |    7 
> >  recipes-extended/shadow/shadow/add-additional-messages-to-indicate-why-
> > locking-fail.patch |  217 ++++++++++
> >  recipes-extended/shadow/shadow/shadow-native-block-groupadd-and-useradd.
> > patch             |  196 +++++++++
> >  4 files changed, 602 insertions(+)
> > 
> > If one really wanted to understand this problem then the thing to do is to
> > run on a kernel that has a "flight recorder" and capture the recent events
> > when the timeout occurs.
> 
> That alone might prevent the error from occurring.
> 
> We are using a local RAID 5 array, so it's less likely to be a iSCSI issue.
> 
> -M

(In reply to comment #9)
> (In reply to comment #8)
> > (In reply to comment #7)
> > > At the time that these failures were causing problems, I thought we were
> > > using a central disk array that was shared between 4-6 builders. The guess
> > > was that we had either an iSCSI protocol timeout of some lower level
> > > occasional hardware timeout but we were never able to prove it. We had to
> > > power cycle the systems
> > > and the problem didn't occur again in the first few days. Due to work
> > > schedules, we applied the patch so that if the hang-up occurred, we'd get
> > > some log data and the hang wouldn't break a build. So as of a couple of
> > > months ago we did see some logs indicating log delays but since the build
> > > doesn't break we have mostly ignored the problem.
> 
> The build doesn't break anymore with the patch, and looks like it never
> happen in yocto's testing, I think it would be fine to close it. I will
> reopen it if it comes out to a problem again.

But the patch is not in oe-core? I'm fine with closing it but it's not really fixed ;)

-M

Verified as per the last 2 comments. It is hard to replicate your RAID 5 and 12 core system scenario at the moment. If it is needed, we will investigate this in the future.

I'm seeing the same failure:

ERROR: tried running groupadd command 10 times without success, giving up
ERROR: Function failed: useradd_sysroot

on a 12 core RHEL-5.5 x86-64 machine, building on local disk, running a single bitbake invocation with:

BB_NUMBER_THREADS ?= "12"
PARALLEL_MAKE ?= "-j 12"

It doesn't look like the patch from comment #3 was committed upstream.  Should I just carry it locally, or can it be committed?

(In reply to comment #12)
> I'm seeing the same failure:
> 
> ERROR: tried running groupadd command 10 times without success, giving up
> ERROR: Function failed: useradd_sysroot
> 
> on a 12 core RHEL-5.5 x86-64 machine, building on local disk, running a
> single bitbake invocation with:
> 
> BB_NUMBER_THREADS ?= "12"
> PARALLEL_MAKE ?= "-j 12"
> 
> It doesn't look like the patch from comment #3 was committed upstream. 
> Should I just carry it locally, or can it be committed?

master branch?

-M

(In reply to comment #13)
> (In reply to comment #12)
> > I'm seeing the same failure:
> > 
> > ERROR: tried running groupadd command 10 times without success, giving up
> > ERROR: Function failed: useradd_sysroot
> > 
> > on a 12 core RHEL-5.5 x86-64 machine, building on local disk, running a
> > single bitbake invocation with:
> > 
> > BB_NUMBER_THREADS ?= "12"
> > PARALLEL_MAKE ?= "-j 12"
> > 
> > It doesn't look like the patch from comment #3 was committed upstream. 
> > Should I just carry it locally, or can it be committed?
> 
> master branch?

I'm seeing the error on the denzil branch of git://git.yoctoproject.org/poky (tag: denzil-7.0.1).

I checked:

git://git.yoctoproject.org/poky
master branch
meta/recipes-extended/shadow

and

git://git.openembedded.org/openembedded-core
master branch
meta/recipes-extended/shadow

and I can't see the shadow-native-block-groupadd-and-useradd.patch changes in either place.

(In reply to comment #14)
> (In reply to comment #13)
> > (In reply to comment #12)
> > > I'm seeing the same failure:
> > > 
> > > ERROR: tried running groupadd command 10 times without success, giving up
> > > ERROR: Function failed: useradd_sysroot
> > > 
> > > on a 12 core RHEL-5.5 x86-64 machine, building on local disk, running a
> > > single bitbake invocation with:
> > > 
> > > BB_NUMBER_THREADS ?= "12"
> > > PARALLEL_MAKE ?= "-j 12"
> > > 
> > > It doesn't look like the patch from comment #3 was committed upstream. 
> > > Should I just carry it locally, or can it be committed?
> > 
> > master branch?
> 
> I'm seeing the error on the denzil branch of git://git.yoctoproject.org/poky
> (tag: denzil-7.0.1).
> 
> I checked:
> 
> git://git.yoctoproject.org/poky
> master branch
> meta/recipes-extended/shadow
> 
> and
> 
> git://git.openembedded.org/openembedded-core
> master branch
> meta/recipes-extended/shadow
> 
> and I can't see the shadow-native-block-groupadd-and-useradd.patch changes
> in either place.

That's where I've seen it too, denzil is quite old by now so I don't expect new fixes.

You can workaround by re-running the bitbake command. Let us know if you see it in newer versions.

-M

(In reply to comment #15)
> (In reply to comment #14)
> > (In reply to comment #13)
> > > (In reply to comment #12)
> > > > I'm seeing the same failure:
> > > > 
> > > > ERROR: tried running groupadd command 10 times without success, giving up
> > > > ERROR: Function failed: useradd_sysroot
> > > > 
> > > > on a 12 core RHEL-5.5 x86-64 machine, building on local disk, running a
> > > > single bitbake invocation with:
> > > > 
> > > > BB_NUMBER_THREADS ?= "12"
> > > > PARALLEL_MAKE ?= "-j 12"
> > > > 
> > > > It doesn't look like the patch from comment #3 was committed upstream. 
> > > > Should I just carry it locally, or can it be committed?
> > > 
> > > master branch?
> > 
> > I'm seeing the error on the denzil branch of git://git.yoctoproject.org/poky
> > (tag: denzil-7.0.1).
> > 
> > I checked:
> > 
> > git://git.yoctoproject.org/poky
> > master branch
> > meta/recipes-extended/shadow
> > 
> > and
> > 
> > git://git.openembedded.org/openembedded-core
> > master branch
> > meta/recipes-extended/shadow
> > 
> > and I can't see the shadow-native-block-groupadd-and-useradd.patch changes
> > in either place.
> 
> That's where I've seen it too, denzil is quite old by now so I don't expect
> new fixes.
> 
> You can workaround by re-running the bitbake command. Let us know if you see
> it in newer versions.

OK, so unpatched master doesn't seem to be affected by this bug, thanks.  I couldn't tell from the closure comment whether master + shadow-native-block-groupadd-and-useradd.patch didn't show the issue, or unpatched master didn't show the issue.  I guess it must have been fixed by some other change.

I'm trying to avoid manually re-running bitbake since our build is happening unattended.  I'll apply the workaround patch while I'm still on denzil, remove it when I update to something newer, then keep an eye out for this failure in the future.  Thanks for the clarification.