Bug 8139

Created attachment 2647 [details]
results of nightly-oecore build 421

Created attachment 2648 [details]
results of nightly-arm build 422

Created attachment 2649 [details]
results of nightly-arm build 423

Created attachment 2650 [details]
results of nightly-qa-systemd build 383

As mentioned by rewitt these failures seem to be related to network dropping out

I was able to have a test scenario 100% reproducible (5/5),

I used Fedora 22 QEMU VM over OpenSUSE 13.2 kvm and ran testimage with core-image-sato.

Configuration,

MACHINE ??= "qemux86-64"
DISTRO_FEATURES_append = " systemd"
VIRTUAL-RUNTIME_init_manager = "systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
VIRTUAL-RUNTIME_initscripts = ""

INHERIT += "testimage"

(In reply to comment #6)
> I was able to have a test scenario 100% reproducible (5/5),
> 
> I used Fedora 22 QEMU VM over OpenSUSE 13.2 kvm and ran testimage with
> core-image-sato.
> 
> Configuration,
> 
> MACHINE ??= "qemux86-64"
> DISTRO_FEATURES_append = " systemd"
> VIRTUAL-RUNTIME_init_manager = "systemd"
> DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
> VIRTUAL-RUNTIME_initscripts = ""
> 
> INHERIT += "testimage"

Could you attach the qemu/ssh logs?

Created attachment 2661 [details]
smart failure fedora 22 vm

Created attachment 2662 [details]
ping failure fedora 22 vm

This looks like a different failure case to me. It seems the network is just completely dead in this scenario.

Created attachment 2666 [details]
systemd_debug patches

These patches seem to cause 100% failure for the smart tests on the centos6 autobuilder.

Just some notes with regards to investigation work:

1. Doesn't seem to be related to tap. The failure also occurs when using only user mode networking.

2. With the https://bugzilla.yoctoproject.org/attachment.cgi?id=2666 patches,
the smart tests seem to fail 100% of the time on the centos6 autobuilder.

3. I could not readily reproduce in a centos6 container.

4. It is reproducible similar to the autobuilder when using a centos6 vm.


My current theory is that the serial output thread is causing qemu to lockup for some reason. When it "hangs" two threads are blocked on a futex and one thread is blocked on a write. Since the "reader" for the log in the test code stops reading after it finds the sentinel vaule, I'm wondering if the socket buffer fills up and causes the block.

Randy all these patches seem to be for debugging, I don't see anything in there (but the upgrade perhaps), that could be forcing the tests to fail, did you do anything else?, or does it simply always fail on centos6?

(In reply to comment #13)
> Randy all these patches seem to be for debugging, I don't see anything in
> there (but the upgrade perhaps), that could be forcing the tests to fail,
> did you do anything else?, or does it simply always fail on centos6?

Nothing in these patches should cause breakage, you're correct. What these patches do is create substantially more output over the serial port. And so it is the combination of these patches and centos6(usually works fine without the debug) that causes it to always fail.

This is why my current theory based on the strace/gdb output that I've seen leads me to believe that a buffer is filling up somewhere and blocking progress.

(In reply to comment #14)
> (In reply to comment #13)
> > Randy all these patches seem to be for debugging, I don't see anything in
> > there (but the upgrade perhaps), that could be forcing the tests to fail,
> > did you do anything else?, or does it simply always fail on centos6?
> 
> Nothing in these patches should cause breakage, you're correct. What these
> patches do is create substantially more output over the serial port. And so
> it is the combination of these patches and centos6(usually works fine
> without the debug) that causes it to always fail.
> 
> This is why my current theory based on the strace/gdb output that I've seen
> leads me to believe that a buffer is filling up somewhere and blocking
> progress.

Oh I see now, have you tried these patches on other similar systems such as Fedora, or a different CentOS?

(In reply to comment #15)
> (In reply to comment #14)
> > (In reply to comment #13)
> > > Randy all these patches seem to be for debugging, I don't see anything in
> > > there (but the upgrade perhaps), that could be forcing the tests to fail,
> > > did you do anything else?, or does it simply always fail on centos6?
> > 
> > Nothing in these patches should cause breakage, you're correct. What these
> > patches do is create substantially more output over the serial port. And so
> > it is the combination of these patches and centos6(usually works fine
> > without the debug) that causes it to always fail.
> > 
> > This is why my current theory based on the strace/gdb output that I've seen
> > leads me to believe that a buffer is filling up somewhere and blocking
> > progress.
> 
> Oh I see now, have you tried these patches on other similar systems such as
> Fedora, or a different CentOS?

I looped a build on a centos6 container with the debug code overnight running the smart tests and didn't get a failure.(that isn't to say it couldn't happen)

I've also never gotten a failure on my Fedora 22 workstation. However, once again, that doesn't mean it can't happen.

So the only consistently reproducible scenario I've found is centos6 with the debug code.

Copying some of the commit message from the first patch(multiple followed to stabilize) since it gives the details.

    qemurunner: Use two serial ports and log console with a thread
    
    qemu can freeze and stop responding if the socket buffer connected to a tcp
    serial connection fills up. This happens of course when the reader of
    the serial data doesn't actually read it.
    
    This happened in the qemurunner code, because after checking for the
    "login:" sentinel, data was never again read from the serial connection.
    
    This patch solves the potential freeze by adding a thread to continuously
    read the data from the console and log it. So it also will give a full log
    of the console, rather than just up to the login prompt.
    
    To simplify this patch, another serial port was also added to use for the
    sole purpose of watching for the sentinel as well as being the interactive
    serial port. This will also prevent the possibility of lots of debug
    data on the console preventing the sentinel value from being seen due to
    interleaved text.

Should be fixed in any jethro rc.

Verified:

All recent nightly-oecore, nightly-arm and nightly-qa-systemd are successful.

They include the patches: https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=jethro&id=fd164dcc31aa6c60e9834036a7cfcea5411daa3d