Bug 9717

Missing some important links:
stdio:https://autobuilder.yoctoproject.org/main/builders/nightly-multilib/builds/815/steps/Running%20Sanity%20Tests_2/logs/stdio

failed build on AB:
https://autobuilder.yoctoproject.org/main/builders/nightly-multilib/builds/815/

Is this because the tests are run with Python3 and the Smart module is python2?

https://bugzilla.yoctoproject.org/show_bug.cgi?id=9675

(In reply to comment #2)
> Is this because the tests are run with Python3 and the Smart module is
> python2?

I should have read the error message more clearly, as we're calling the smart program rather than trying to make use of the modules from within the test suite it's unlikely to be python3 trying to call Python2 modules. May still be worth investigating that though?

Has anyone checked whether Smart works on the target since the switch to a Python3 userspace?

This is an intermittent fail, below the details :


*Public AB :

#  -> Status -> Host   
--------------------------
814 -> PASS -> ubuntu1404
815 -> Fail -> opensuse132
816 -> PASS -> ubuntu1404
818 -> Fail -> fedora23
819 -> fail -> debian8
821 -> PASS -> debian8
822 -> PASS -> centos7
823 -> fail -> opensuse421

https://autobuilder.yoctoproject.org/main/builders/nightly-multilib 

*GDC AB :

#  -> Status -> Host   
--------------------------
02 -> PASS -> ubuntu1404
03 -> PASS -> opensuse132

Tests on GDC AB were applied on commit:  2228b16bef534e337bc0f7c988b9f96dea7a3271

(In reply to comment #3)
> (In reply to comment #2)
> > Is this because the tests are run with Python3 and the Smart module is
> > python2?
> 
> I should have read the error message more clearly, as we're calling the
> smart program rather than trying to make use of the modules from within the
> test suite it's unlikely to be python3 trying to call Python2 modules. May
> still be worth investigating that though?
> 
> Has anyone checked whether Smart works on the target since the switch to a
> Python3 userspace?

Well I just checked manually and also ran the test, on master, smart works just fine for me on both cases.

I downloaded the image from the AB, and was able to reproduce it:

Just checking that smart calls python2
# head -n 2 /usr/bin/smart 
#!/usr/bin/env python

# which python
/usr/bin/python

# python
Python 2.7.11 (default, Jun  2 2016, 10:48:06) 
[GCC 6.1.1 20160511] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import smart
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named smart

As a matter of fact, the smart module isnt present:
root@qemux86-64:/usr/lib/python2.7# find | grep smart
root@qemux86-64:/usr/lib/python2.7# cd site-packages
-sh: cd: site-packages: No such file or directory


Whereas on a working image:
/usr/lib/python2.7/site-packages# ls
rpm  smart  smart-1.5-py2.7.egg-info

This is a multilib image, smart gets installed on /usr/lib64/python2.7/site-packages/smart, instead of /usr/lib/python2.7/site-packages/smart

And python isnt aware of that:

# python
Python 2.7.11 (default, Jun  2 2016, 10:48:06) 
[GCC 6.1.1 20160511] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python27.zip', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload']

If we do:

>>> sys.path.append('/usr/lib64/python2.7/site-packages')
>>> sys.path
['', '/usr/lib/python27.zip', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/lib64/python2.7', '/usr/lib64/python2.7/site-packages']
>>> import smart
>>> 


It works!

I need to check another multilib image where the test passed to see whats happening.

It was suggested on jabber that the build host path might be somehow polluting the installation path for the smart module. Here is the default python path for all the public autobuilders:

centos7.yp
['', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/gtk-2.0', '/usr/lib/python2.7/site-packages']

debian8.yp
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/pymodules/python2.7']

fedora22.yp
['', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/gtk-2.0', '/usr/lib/python2.7/site-packages']

fedora23.yp
['', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib/python2.7/site-packages']

opensuse132.yp
['', '/usr/lib/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/local/lib64/python2.7/site-packages', '/usr/local/lib/python2.7/site-packages', '/usr/lib/python2.7/site-packages']

opensuse421.yp
['', '/usr/lib/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/local/lib64/python2.7/site-packages', '/usr/local/lib/python2.7/site-packages', '/usr/lib/python2.7/site-packages']

ubuntu1404.yp
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']

ubuntu1510.yp
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']

Summarizing the list of passes and failures noted earlier, along with lib paths:

build host   pass/fail   host python lib path
=============================================
opensuse421   fail       /usr/lib64/python2.7/site-packages
opensuse421   pass       /usr/lib64/python2.7/site-packages
centos7       pass       /usr/lib64/python2.7/site-packages
debian8       pass       /usr/lib/python2.7
debian8       fail       /usr/lib/python2.7
ubuntu1404    fail       /usr/lib/python2.7/site-packages
fedora23      fail       /usr/lib64/python2.7/site-packages

There does not appear to be a direct correlation between host paths and pass/fail. In fact, in some cases, the same build host generated images that failed one time and passed another time when trying to load smart.

This is an image where the test passed, and unsurprisingly the sys.path contains the correct path:

root@qemux86-64:~# python
Python 2.7.11 (default, Jun  3 2016, 18:24:02) 
[GCC 6.1.1 20160511] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print sys.path
['', '/usr/lib/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages']

*** Bug 9754 has been marked as a duplicate of this bug. ***

Checking the working image vs the non-working one, I found a couple of things:

Checking the python libraries directory on both:

## WORKING IMAGE
# ls -l /usr/lib64/python2.7/
total 4120
-rw-r--r-- 1 root root 27746 Jun  3 18:24 ConfigParser.py
-rw-r--r-- 1 root root 25213 Jun  3 18:24 ConfigParser.pyc
-rw-r--r-- 1 root root 26537 Jun  3 18:24 Cookie.py
-rw-r--r-- 1 root root 22657 Jun  3 18:24 Cookie.pyc
-rw-r--r-- 1 root root  6482 Jun  3 18:24 MimeWriter.py
-rw-r--r-- 1 root root  7364 Jun  3 18:24 MimeWriter.pyc
-rw-r--r-- 1 root root  8577 Jun  3 18:24 Queue.py
-rw-r--r-- 1 root root  9424 Jun  3 18:24 Queue.pyc

...

## NON-WORKING IMAGE
# ls -l /usr/lib64/python2.7/
drwxr-xr-x    4 root     root          1024 Jun  2 11:51 site-packages


What this means is that, for one, this isn't a python-smartpm bug, it is likely a python-multilib bug, since on the non-working image smart is actually the one python package that's being installed correctly on /usr/lib64, while all the others are being installed on /usr/lib meaning that python is unaware that this is a multilib image.

Checking the python-smartpm recipe I don't see anything important at first look, although I see that this is installed using distutils, whereas lots of other packages use setuputils instead, it might be worth checking if there is a problem with setuptools on multilib images, this would not explain while this bug is intermitent though, only found on certain hosts.

I took a look at setuptools and distutils. Distutils has both a "purelib" and "platlib" (for platform specific libs).

The multilib patch changes the distutils install command to fixup the platform install patch, changing it from:
   $platbase/lib/python$py_version_short/site-packages'
to:
   '$platbase/'+libname+'/python$py_version_short/site-packages'

where libname is earlier set as the value returned sys.lib. On a 64-bit system with the multilib patch, sys.lib will return '${prefix}/lib64' (typically, "/usr/lib64").

Setuptools, on the other hand, has just a single installation path:
   INSTALL_SCHEMES = dict(
        posix=dict(
            install_dir='$base/lib/python$py_version_short/site-packages',
            script_dir='$base/bin',
        ),
    )

where you can see it's hard coded as (typically) under "/usr/lib".

So, that does explain why smart is installed under /usr/lib64 in some cases, but still doesn't explain why, if distutils is used for installing smart that is picks up /usr/lib for some autobuilders, but /usr/lib64 for others.

I was checking the images again, just wanted to comment which packages are being built incorreclty:

On the non-working image were missing on /usr/lib64:
< coreutils
< gconv
< libkmod.so.2
< libkmod.so.2.3.0
< locale
< ssl

and all these but coreutils are located on /usr/lib/ incorrectly

Also python2 is installed incorrectly on /usr/lib/ instead of /usr/lib64/, the folder doesexiste because python-smartpm was correctly installed on /usr/lib64.

Looking at python's sysconfigdata we can see that the working one was built for x86_64-poky and the nono-working was built for i686-pokymlib32:


Working:
 'AR': 'x86_64-poky-linux-ar',
 'CC': 'x86_64-poky-linux-gcc  -m64 -march=core2 -mtune=core2 -msse3 -mfpmath=sse '


Non-Working:
'AR': 'i686-pokymllib32-linux-ar',
'CC': 'i686-pokymllib32-linux-gcc  -m32 ',

(SWAT update) This test still fails on master:
http://errors.yoctoproject.org/Errors/Details/69498/

More information. I think I finally figured out what is going on.

The multilib build set actually builds several images in a row:
  machine: qemux86_64; multilib: True; image: lib32-core-image-minimal
  machine: qemux86_64; multilib: True; packages: ipk; image: lib32-core-image-minimal
  machine: qemux86_64; multilib: lib32; image: core-image-sato
  machine: qemux86_64; multilib: lib32; packages: ipk; image: core-image-sato
  machine: qemux86; multilib: lib64; image: lib64-core-image-sato

Build config #3 is the that is failing.

Looking at the log for creating the rootfs, we see:
  DEBUG: Executing python function do_rootfs
  NOTE: configuring RPM platform settings
  NOTE: configuring RPM system provides
  NOTE: configuring RPM DB settings
  NOTE: configuring Smart settings
  NOTE: Adding Smart channel lib32_qemux86_64 (45)
  NOTE: Adding Smart channel lib32_x86 (40)
  NOTE: Adding Smart channel all (35)
  NOTE: Adding Smart channel qemux86_64 (30)
  NOTE: Adding Smart channel core2_64 (25)
  NOTE: adding Smart RPM DB channel

Also consider that after the first build, there are rpm packages for python2.7 in build/tmp/deploy/rpm/lib32_x86 and after the third build, there are rpm packages for python in build/tmp/deploy/rpm/core2_64. Notice that the package manager appears to be looking in both locations to find packages to install in the rootfs. I do not know how it decides which ones to use when there are multiple options, but in the fail case, it is using the packages in lib32_x86 when it should be using the ones in core2_64.

On a build that had failed, I forced a rebuild of python and core-image-sato:
  $ bitbake -C install python
  $ bitbake core-image-minimal

This time, after the rebuild, the packages in core2_64 were used to create the rootfs.

I am checking for the same thing, I wasnt able to reproduce it just using the failing config since the beginning (since apparently it uses the right packages), I am at thi very moment building the same configs in order to see if rootfs gets populated incorreclty that way.

After using the buildset configs in order, I still wasn't able to reproduce it manually.

Hmmm... I still think it's a smartpm issue, since it is fed both 32-bit and 64-bit channels. A brief look at the smartpm documentation says it uses the "newest" package when there are multiple to chose from - which may or may not be the one we want it to use. I did notice you can assign priorities to different channels, so the 64-bit channel could potentially be given higher priority for the core-image-sato build.

In the morning, I'm going to look at the rootfs generation class and see if there's anyway to pass different parameters to smartpm, based on the image name and multilib setting.

Additional information: subsequent test builds (which used SSTATE) did not fail and created images with python in /usr/lib64, as it should be. Did a "very clean" build with build/tmp deleted and SSTATE_DIR commented out so everything would get built and packaged from scratch. In this case, the qemux86_64 rootfs image has the python libraries installed in /usr/lib and are 32-bit libraries. (Verified by inspecting the core-image-minimal rootfs .tar.gz file.)

Both lib32_x86 and core2_64 rpm packages were built and are present in deploy/rpm/core2_64 and deploy/rpm/lib32_x86. (In the AB config, lib32-core-image-minimal is built first, followed by core-image-sato - which should be a mostly 64-bit image.)

Verified the python rpms in lib32_x86 are using /usr/lib and the ones in core2_64 are using /usr/lib64.

It still seems like a package manager issue to me in deciding which rpm packages to use to build the rootfs.

Digging a little bit more, I've got 2 lists of packages to be installed, the Non-mulitib and the multilib ones, they're very long but I just wanted to state that python (which is an example of what is incorrectly installed) is on the Non-mulitilib packages list, which means that at least bitbake knows what its doing, and it should install the correct one, so the problem comes when smart is called and it chooses poorly:

Multilib pkgs: lib32-libgdk-pixbuf-2.0-loader-png lib32-pango-module-basic-fc lib32-connman-gnome lib32-libgdk-pixbuf-2.0-loader-jpeg lib32-connman lib32-hicolor-icon-theme lib32-dbus-glib lib32-glib-2.0 lib32-gtk+3 lib32-glibc ...

non-Multilib pkgs: locale-base-en-us locale-base-en-gb libgdk-pixbuf-2.0-loader-png packagegroup-base-extended pango-module-basic-fc packagegroup-core-x11-sato-games rpm psplash packagegroup-core-ssh-dropbear packagegroup-core-x11-base smartpm ... python-smartpm python-core python-smartpm-backend-rpm python-codecs python-textutils python-xml python-fcntl python-pickle python-crypt python-compression python-shell python-resource python-netclient python-threading ...

Ok so to debug what smart is doing I added "--explain" to the smart install command on package_manager.py (which I think should probably be enabled by default btw, it doesn't hurt anyone having more info in the log, although they're quite long), and as a result I can see which packages are being installed and where they're coming from:

After a bitbake core-image-sato -c cleansstate I get:

$ bibtake core-image-sato -c rootfs:

python-core-2.7.11-r1.0@lib32_x86 [lib32_x86]
    Requires:
      libc6-2.24-r0.0@lib32_x86 (installed)
      python-lang-2.7.11-r1.0@lib32_x86 (installed)
      libpython2.7-1.0-2.7.11-r1.0@lib32_x86 (installed)
      libpython2.7-1.0-2.7.11-r1.0@core2_64 (installed)
      libz1-1.2.8-r0.0@core2_64 (installed)
      coreutils-8.25-r0.0@lib32_x86 (installed)
      libc6-2.24-r0.0@core2_64 (installed)
      python-re-2.7.11-r1.0@lib32_x86 (installed)
      busybox-1.24.1-r0.0@lib32_x86 (installed)

Now that's obviously wrong, interestignly, if I do a (a couple of times actually, it didn't work on the first try):

$ bibtake core-image-sato -c rootfs --force 

python-core-2.7.11-r1.0@core2_64 [core2_64]
    Requires:
      libc6-2.24-r0.0@lib32_x86 (installed)
      python-lang-2.7.11-r1.0@core2_64 (installed)
      libpython2.7-1.0-2.7.11-r1.0@core2_64 (installed)
      python-re-2.7.11-r1.0@core2_64 (installed)
      libz1-1.2.8-r0.0@core2_64 (installed)
      coreutils-8.25-r0.0@core2_64 (installed)
      libc6-2.24-r0.0@core2_64 (installed)
      libz1-1.2.8-r0.0@lib32_x86 (installed)
      busybox-1.24.1-r0.0@core2_64 (installed)


Which is correct, so just wanted say that its not reproducible every time, it does however seems to be reproducible after a cleansstate

I'm attaching the (long) longs here, so we can see how they differ, its a litte difficult since most things are not in the same order though.

Created attachment 3240 [details]
Incorrect Install Log

Created attachment 3241 [details]
Correct Install Log

Found something interesting on the logs:

When adding the channels to smart we've got:

NOTE: Adding Smart channel qemux86_64 (45)
NOTE: Adding Smart channel core2_64 (40)
NOTE: Adding Smart channel all (35)
NOTE: Adding Smart channel lib32_qemux86_64 (30)
NOTE: Adding Smart channel lib32_x86 (25)

vs

NOTE: Adding Smart channel lib32_qemux86_64 (45)
NOTE: Adding Smart channel lib32_x86 (40)
NOTE: Adding Smart channel all (35)
NOTE: Adding Smart channel qemux86_64 (30)
NOTE: Adding Smart channel core2_64 (25)

Where for example in the last one core2_64 is the architecture and the (25) is the priority of it.

Also as we can see the priorities are not the same on both cases, hence why smart prefers to install python-core@core2_64 (correctly) in the first case and python-core@lib32_x86 (incorrectly) on the second case.

In the code, we substract (5) to the channel priority everytime we add another channel, so its all about the order in which they are added.

Priorities for the channel feeds weigh on the results when generating the filesystem.

If two things are considered equivalent then the priority of the feed affects which is chosen.  In a 'standard' multilib build, the result SHOULD be:

NOTE: Adding Smart channel qemux86_64 (45)
NOTE: Adding Smart channel core2_64 (40)
NOTE: Adding Smart channel all (35)
NOTE: Adding Smart channel lib32_qemux86_64 (30)
NOTE: Adding Smart channel lib32_x86 (25)

The machine has highest priority, then the native system lib, then all, THEN the multilibs.

The other listed in the previous comment will end up switching the system to a 32-bit default.  You should verify that this is not intentional in some way.

I thought at one point we had implemented that if you did something like

bitbake lib32-core-image-base  the system would end up switching the multilib order during install to make 32-bit the primary for installation.

The code was using unordered objects (dict, set) for ordered data. I've applied a patch which deals with one case of this which was causing the ordering issue Alejandro observed. Looking at the code there are other places with the issue. The "corrected" ordering isn't as Mark describes exactly but I'm not sure that matters. We need to fix some other places in this code which suffer from ordering problems too (e.g. the use of set() ).

I've done an audit on package_manager.py and I believe the only issues of concern regarding sets and dicts are in RpmPM:insert_feeds_uris(), with the arch_list, default_platform_extra and platform_extra variables.

An OrderedDict can be used for arch_list, but the others make use of set() features not part of OrderedDict(), specifically, the union operation. We could either pull in an existing OrderSet module (there are a couple of them available), or rewrite the code block to locally perform a union type operation on an OrderedDict and use OrderedDicts for those two variables.

(In reply to comment #30)
> I've done an audit on package_manager.py and I believe the only issues of
> concern regarding sets and dicts are in RpmPM:insert_feeds_uris(), with the
> arch_list, default_platform_extra and platform_extra variables.
> 
> An OrderedDict can be used for arch_list, but the others make use of set()
> features not part of OrderedDict(), specifically, the union operation. We
> could either pull in an existing OrderSet module (there are a couple of them
> available), or rewrite the code block to locally perform a union type
> operation on an OrderedDict and use OrderedDicts for those two variables.

Perhaps list comprehension operations can do the union task using the current data structure, avoiding the OrderedDict/OrderSet usage.

*** Bug 9794 has been marked as a duplicate of this bug. ***

Fixed in oe-core 61a33582dfc964d612d20d34734a787d873e312c

*** Bug 9797 has been marked as a duplicate of this bug. ***

Replaced the set add and union operations in RpmPM:insert_feeds_uris() with list and basic list operations, copying the existing code in create_configs(). This keeps the items in order, which is important because it is used later to set Smart channel priorities.

Patch sent to mailing list.

Verified : issue not present in the latest steps of AB

Jun 30 16:42	1159826bb9ca...	success	#851	Build successful
Jun 28 22:50	7e9297e5722d...	success	#850	Build successful
Jun 28 16:35	39732501e105...	success	#849	Build successful
Jun 28 10:50	786d894b6b50...	success	#848	Build successful
Jun 27 22:28	594b869946ac...	success	#847	Build successful
Jun 27 15:44	684e0de230ee...	success	#846	Build successful
Jun 22 11:18	65312e5b2771...	success	#845	Build successful

https://autobuilder.yoctoproject.org/main/builders/nightly-multilib