Also note there are at least three different ways this is failing (different messages) but the cause would seem to be semi-common.

I'm able to reproduce this pretty easily on my local machine. Just add the following to the conf
----8<------------------------------------
INHERIT += "testimage"
MACHINE = "qemux86-64"
QEMU_USE_KVM = "True"
TEST_SUITES = "smart"
----8<------------------------------------

And run
$ bitbake core-image-sato
$ bitbake core-image-sato -c testimage


The bug is clearly caused by:
commit 1b50488c53120e70db1fb050b4ca11e05234adc9
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date:   Thu Sep 8 11:24:00 2016 +0100

    oeqa/runtime/smart: Prune feeds to save memory

I just don't yet understand why...

I just took a quick look and haven't reproduced it yet, but don't we need to keep psplash dependencies as well?

I tried to reproduce on a 16 core server and I don't see the error with configuration changes that Markus mention.

I can reproduce the error (usually but not always) with this setup
INHERIT += "testimage"
MACHINE = "qemux86"
TEST_SUITES = "ping ssh smart"
And run
$ bitbake core-image-sato-sdk
$ bitbake core-image-sato-sdk -c testimage



This default to having qemu off.

Furthermore, if I  revert RP's 1b50488c53120e70db1fb050b4ca11e05234adc9 patch I get no errors when I run bitbake core-image-sato-sdk -c testimage with -m 1024 but I do get errors on some of the smart tests in which the target process is killed when I set the memory to -m 128.

I took a closer look at the bug today. And, I have very hard time understanding what is happening behind the scenes...

I'm able to reproduce this reliably when running testimage. However, I was not able to get any errors when I tried manually running qemu and executing the same smart commands as in the tests.

Next, I tried interrupting test execution (with simple sleep()) when it fails and ssh'ing in the qemu instance. I got very confusing results:
----8<-8<-8<---------------------------------------
root@qemux86-64:~# rpm -qa|grep splash
psplash-0.1+git0+88343ad23c-r15.core2_64
psplash-default-0.1+git0+88343ad23c-r15.core2_64
root@qemux86-64:~# smart query --installed |grep splash
psplash-0.1+git0+88343ad23c-r15@core2_64
----8<-8<-8<---------------------------------------

Somehow rpm and smart have a different understanding which packages are installed in the system(!). I don't understand how this can happen - could it be that the ssh connection is closed prematurely when running smart commands which leaves the system in an incoherent state or what(?)

I verified that it is NOT an ssh connection problem or smth.

Now it looks like it's actually a problem in smart itself. I suspect (beware, I don't have much knowledge about smart internal) that smart has some worker threads or something and when smart commands are being issued too quickly things get corrupted. I was able to "fix" the problem in my setup by inserting
time.sleep(0.5)
in oeqa.runtime.smart.SmartTest.smart()

It IS a smart issue. I was able to manually reproduce this by ssh'ing into the image and issuing:
# smart remove -y psplash-default; smart install -y psplash-default; smart remove -y psplash; install -y psplash-defaul

Somehow /var/lib/smart/cache gets out-of-sync but I still don't understand why.

For M3 we've merged some delays which work around the issue. We need to look at this further in M4 and figure out if there is a better/proper fix for the issues we're seeing. It does seem like a problem within smart.

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=68e35c8e3793ed4f7f54d14a5d5b0c59598b7dda is the workaround,

Ultimately the problem seems to be that smart tracks /var/lib/rpm/Packages in order to keep a list of packages that are installed in the system. However, the rpm master db file seems to be racy.

Smart uses mtime to see if the rpm packagedb changed. Somehow the update of the Packages file is delayed/deferred (I haven't looked how (lib)rpm does this). Thus, when smart commands are fired in quick succession smart does not detect the changes done by previous command (because Packages file hasn't yet changed).

The tracking is done in smart/channels/rpm_sys.py.

I'm not sure if I managed to explain the problem above. In other words it seems that /var/lib/rpm/Packages has not been written yet when the next smart command already reads it

I think you posted a very thorough explanation of your understanding of the error condition.

And it reminds me of other issues with smart that also behave like race conditions, meaning, perhaps there will be more.

FWIW I was having trouble reproducing this but I did develop a pathological testcase: 

From 9172914c89b9f210b6354c31675ced0e85d3d23a Mon Sep 17 00:00:00 2001
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Date: Fri, 9 Sep 2016 17:48:25 +0100
Subject: smart: pathological testcase

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

diff --git a/meta/lib/oeqa/runtime/smart.py b/meta/lib/oeqa/runtime/smart.py
index 6cdb10d..1dcebbe 100644
--- a/meta/lib/oeqa/runtime/smart.py
+++ b/meta/lib/oeqa/runtime/smart.py
@@ -152,6 +152,9 @@ class SmartRepoTest(SmartTest):
         self.smart('remove -y psplash')
         self.smart('install -y psplash-default')
 
+    def test_smart_install_from_dis(self):
+        self.smart('download psplash-default; smart remove -y psplash-default; smart install -y ./psplash-default*; smart remove -y psplash-default; smart install -y ./psplash-default*; smart remove -y psplash-default; smart install -y ./psplash-default* ')
+
     @testcase(723)
     @skipUnlessPassed('test_smart_channel_add')
     def test_smart_install_from_disk(self):

which is basically making sure the commands run back to back as fast as possible to make the bug obvious.

Created attachment 3409 [details]
strace when install success

Created attachment 3410 [details]
strace when remove success

Created attachment 3411 [details]
strace when install fails

Created attachment 3412 [details]
strace when remove fails

I was able to replicate the problem using the method described by Markus. Here are the steps that I did:

1. Added to local.conf:
    INHERIT += "testimage"
    MACHINE = "qemux86-64"
    QEMU_USE_KVM = "True"
    TEST_SUITES = "smart"
2. Added a sleep of 2 hours at the beggining of test_smart_install in smart test
3. Run "bitbake core-image-sato -c testimage"
4. Wait until test_smart_channel_show pass
5. Connect to the DUT using ssh
6. Run "smart remove -y psplash-default; smart install -y psplash-default"

This only happens with sato, I couldn't reproduce the behavior with sato-sdk or without using KVM. I just tried with x86-64, haven't tried with x86.

I was able to get strace from a running target. There is a pair when it works, and the other pair is when it fails.

These results are using the patch from RP in comment 15. My results concur with what Mariano saw. Looking at core-image-sato and core-image-sato-sdk and trying x86 and x86_64 with both KVM_QEMU and non KVM QEMU, the only failure was core-image-sato,QEMU_USE_KVM="True",x86_64.  All other combinations succeeded for me.

So, I wondered what was in core-image-sato-sdk x86_64 that made it work while core-image-sato didn't.  

Turns out that the only thing you need to add to core-image-sato to make it suddenly start working is 
IMAGE_INSTALL += "kernel-devsrc"

According to the manifest file, this pulls in bc,libperl5,perl, and kernel-devsrc.

An image based on sato that adds bc,libperl5,perl still fails. Additionally adding kernel-devsrc makes it pass the RP comment 15 test. 

Why? Not sure...

I tried to recreate the bug outside testimage using the same rpm directory as testimage. I couldn't reproduce the error. The steps were:

1. Added to local.conf:
    INHERIT += "testimage"
    MACHINE = "qemux86-64"
    QEMU_USE_KVM = "True"
    TEST_SUITES = "smart"
2. Run testimage and wait for results.
3. Boot the image using "runqemu qemux86-64 core-image-sato"
4. Run a HTTP server on
    tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r0/rpm
5. Connect to the DUT using SSH
6. Setup smart channels with commands:
    smart channel -y --add all type=rpm-md baseurl=http://192.168.7.1:8000/rpm/all
    smart channel -y --add core2_64 type=rpm-md baseurl=http://192.168.7.1:8000/rpm/core2_64
    smart channel -y --add qemux86_64 type=rpm-md baseurl=http://192.168.7.1:8000/rpm/qemux86_64
7. Run commands: "smart remove -y psplash-default; smart install -y psplash-default; smart remove -y psplash-default; smart install -y psplash-default; smart remove -y psplash-default; smart install -y psplash-default; smart remove -y psplash-default; smart install -y psplash-default;"

I did some debugging and seems that this is not a race related to delayed/deferred update of rpm db. AFAIU the db update is done synchronously in the same process.

However, it seems I got fooled by mtime, I clearly do not understand it well enough. It seems that mtime does not get update after every write. If the system is under load and there are consecutive file accesses mtime seems to be updated only after the "system stress" decreases.

Thus, smart shouldn't use mtime for tracking rpm db updates but something else. I tried a simple change of changing it to md5sum, but, that somehow broke everything. Need to investigate more...

Seems my md5 approach worked. My simple patch I mentioned above failed just because of a small typo.

I sent a patch for review:
https://patchwork.openembedded.org/patch/131561/

Please test it. It should fix the issue.

I checked the patch from Markus and it solved the issue

The patch from Markus and it solved the issue for me as well.

Fixed in
http://git.openembedded.org/openembedded-core/commit/?id=e7267b4e78461e71a1175f93e2eb5e90272c2b47