Bug 12148

Summary: bmap-tools does not find pkg_resources built by Yocto, and uses host pkg_resources
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Ulf Samuelsson <yocto>
Component: deploymentAssignee: Wenzong.Fan <wenzong.fan>
Status: RESOLVED FIXED QA Contact:
Severity: critical    
Priority: Medium+ CC: alejandro.hernandez, gizero, randy.macleod, richard.purdie, stephano, tim.orling
Version: unspecified   
Target Milestone: 2.4.3   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Ulf Samuelsson 2017-09-30 22:26:50 UTC
When executing "do_image_wic" 

wic create "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/../meta-emagii/wic/emagii_beaglebone.wks" --vars "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/sysroots/beaglebone/imgdata/" -e "emagii-image" -o "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/deploy-emagii-image-image-complete/emagii-image-beaglebone-20170929070314/

Note: I mention bmaptool below, but there are several apps in "bmap-tools" which could be the problem.

Then bmaptool executes the statement: "from pkg_resources import load_entry_point"
pkg_resources used to contain pkg_resources.py with "load_entry_point".
Now, "load_entry_point" is within pkg_resources/__init__.py,
but apparently this is not found, so bmaptool finds 
"/usr/lib/python3/dist-packages/pkg_resources.py" amongst the host files.
A check of bmaptool version is done, and not host bmaptool is checked.

On Ubuntu-16.04, the bmaptool version is 3.2, which does not meet the 3.4 requirement, so the build fails without creating a rootfs.
Obviously using the host version is bad, but if people are using the 3.4 bmap-toos they may not have detected the problem.

Error Log:

| DEBUG: Executing shell function do_image_wic
| wic create "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/../meta-emagii/wic/emagii_beaglebone.wks" --vars "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/sysroots/beaglebone/imgdata/" -e "emagii-image" -o "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/deploy-emagii-image-image-complete/emagii-image-beaglebone-20170929070314/"
| INFO: Creating image(s)...
|
| INFO: The new image(s) can be found here:
| /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/deploy-emagii-image-image-complete/emagii-image-beaglebone-20170929070314/emagii_beaglebone-201709290903-mmcblk.direct
|
| The following build artifacts were used to create the image(s):
|   ROOTFS_DIR: /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/deploy-emagii-image-image-complete/emagii-image-beaglebone-20170929070314/tmp.wic.slysr_64/rootfs_copy
|   BOOTIMG_DIR: /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/recipe-sysroot/usr/share
|   KERNEL_DIR: /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/deploy/images/beaglebone
|   NATIVE_SYSROOT: /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/recipe-sysroot-native
|
| INFO: The image(s) were created using OE kickstart file:
| /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/../meta-emagii/wic/emagii_beaglebone.wks
| Traceback (most recent call last):
|   File "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/recipe-sysroot-native/usr/bin/bmaptool", line 6, in <module>
|     from pkg_resources import load_entry_point
|   File "/usr/lib/python3/dist-packages/pkg_resources.py", line 2749, in <module>
|     working_set = WorkingSet._build_master()
|   File "/usr/lib/python3/dist-packages/pkg_resources.py", line 444, in _build_master
|     ws.require(__requires__)
|   File "/usr/lib/python3/dist-packages/pkg_resources.py", line 725, in require
|     needed = self.resolve(parse_requirements(requirements))
|   File "/usr/lib/python3/dist-packages/pkg_resources.py", line 628, in resolve
|     raise DistributionNotFound(req)
| pkg_resources.DistributionNotFound: bmap-tools==3.4
| WARNING: /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/temp/run.do_image_wic.19342:1 exit 1 from 'bmaptool create emagii-image-beaglebone-20170929070314.rootfs.wic -o emagii-image-beaglebone-20170929070314.rootfs.wic.bmap'
| ERROR: Function failed: do_image_wic (log file is located at /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/temp/log.do_image_wic.19342)
ERROR: Task (/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/../meta-emagii/recipes-images/images/emagii-image.bb:do_image_wic) failed with exit code '1'
NOTE: Tasks Summary: Attempted 3187 tasks of which 3186 didn't need to be rerun and 1 failed.

Summary: 1 task failed:

/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/../meta-emagii/recipes-images/images/emagii-image.bb:do_image_wic
Summary: There was 1 WARNING message shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
Comment 1 Tim Orling 2017-12-23 01:58:46 UTC
The version of python3-setuptools now available on oe-core master packages pkg_resources. So, bmap-tools now RDEPENDS on python3-setuptools. I'll be sending a patch.
Comment 2 Tim Orling 2017-12-23 02:47:47 UTC
https://patchwork.openembedded.org/patch/146884/
Comment 5 Armin Kuster 2018-01-04 17:02:43 UTC
updating to 36.5 seems safe. 

http://cgit.openembedded.org/openembedded-core/commit/meta/recipes-devtools/python?id=5b784c37f1be769a2abcc5bef5b03eb3b349138b

v36.5.0
-------

* #170: When working with Mercurial checkouts, use Windows-friendly
  syntax for suppressing output.

* Inspired by #1134, performed substantial refactoring of
  ``pkg_resources.find_on_path`` to facilitate an optimization
  for paths with many non-version entries.

v36.4.0
-------

* #1075: Add new ``Description-Content-Type`` metadata field. `See here for
  documentation on how to use this field.
  <https://packaging.python.org/specifications/#description-content-type>`_

* #1068: Sort files and directories when building eggs for
  deterministic order.

* #196: Remove caching of easy_install command in fetch_build_egg.
  Fixes issue where ``pytest-runner-N.N`` would satisfy the installation
  of ``pytest``.

* #1129: Fix working set dependencies handling when replacing conflicting
  distributions (e.g. when using ``setup_requires`` with a conflicting
  transitive dependency, fix #1124).

* #1133: Improved handling of README files extensions and added
  Markdown to the list of searched READMES.

* #1135: Improve performance of pkg_resources import by not invoking
  ``access`` or ``stat`` and using ``os.listdir`` instead.

v36.3.0
-------

* #1131: Make possible using several files within ``file:`` directive
  in metadata.long_description in ``setup.cfg``.
Comment 6 Armin Kuster 2018-01-04 17:05:38 UTC
updating to 38.2.4 adds new functions.

We change update the package and exclude the pypi part.

v38.2.4
-------

* #1220: Fix `data_files` handling when installing from wheel.

v38.2.3
-------

* fix Travis' Python 3.3 job.

v38.2.2
-------

* #1214: fix handling of namespace packages when installing
  from a wheel.

v38.2.1
-------

* #1212: fix encoding handling of metadata when installing
  from a wheel.

v38.2.0
-------

* #1200: easy_install now support installing from wheels:
  they will be installed as standalone unzipped eggs.

v38.1.0
-------

* #1208: Improve error message when failing to locate scripts
  in egg-info metadata.

v38.0.0
-------

* #458: In order to support deterministic builds, Setuptools no
  longer allows packages to declare ``install_requires`` as
  unordered sequences (sets or dicts).

v37.0.0
-------

* #878: Drop support for Python 2.6. Python 2.6 users should
  rely on 'setuptools < 37dev'.

v36.8.0
-------

* #1190: In SSL support for package index operations, use SNI
  where available.

v36.7.3
-------

* #1175: Bug fixes to ``build_meta`` module.

v36.7.2
-------

* #701: Fixed duplicate test discovery on Python 3.

v36.7.1
-------

* #1193: Avoid test failures in bdist_egg when
  PYTHONDONTWRITEBYTECODE is set.

v36.7.0
-------

* #1054: Support ``setup_requires`` in ``setup.cfg`` files.

v36.6.1
-------

* #1132: Removed redundant and costly serialization/parsing step
  in ``EntryPoint.__init__``.

* #844: ``bdist_egg --exclude-source-files`` now tested and works
  on Python 3.

v36.6.0
-------

* #1143: Added ``setuptools.build_meta`` module, an implementation
  of PEP-517 for Setuptools-defined packages.

* #1143: Added ``dist_info`` command for producing dist_info
  metadata.
Comment 7 Tim Orling 2018-01-04 17:11:52 UTC
pkg_resources was available in 36.2.7, apparently it just wasn't being packaged with the state of the recipe.

https://github.com/pypa/setuptools/tree/d548901f9911e784892e99e933d6c60016521e34/pkg_resources
Comment 8 Armin Kuster 2018-01-04 17:33:37 UTC
(In reply to comment #7)
> pkg_resources was available in 36.2.7, apparently it just wasn't being
> packaged with the state of the recipe.
> 
> https://github.com/pypa/setuptools/tree/
> d548901f9911e784892e99e933d6c60016521e34/pkg_resources

ok, so all I need to do is fix the recipe?
Comment 9 Armin Kuster 2018-01-11 16:02:19 UTC
there is a patch for rocko I need to backport.
Comment 10 Wenzong.Fan 2018-04-17 19:31:16 UTC
(In reply to comment #9)
> there is a patch for rocko I need to backport.

I checked the 2.4.2 branch and I found this commit for bmap-tools:
---------------------------------------
$ git log bmap-tools_3.4.bb
commit 8fedae4a8444394a9c00d40e41aa302d2507030b
Author: Tim Orling <timothy.t.orling@linux.intel.com>
Date:   Fri Dec 22 18:46:17 2017 -0800

    bmap-tools: fix RDEPENDS and update SRC_URI

    * RDEPENDS are now all the python3 versions
    * Upstream SRC_URI has moved to github.com/intel/bmap-tools
    * Missing RDEPENDS on python3-setuptools added
      - Provides import for pkg_resources

    Fixes [YOCTO #12148]

    (From OE-Core rev: 6cc7da6477c6995e936f241b0f6f6ec147814538)

    Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
    (cherry picked from commit 3d63452d8d8d8dc61cd26b7759b5c0d235d70451)
    Signed-off-by: Armin Kuster <akuster808@gmail.com>
    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
----------------------

And this one for python3-setuptools:

---------------------
$ git log python3-setuptools_36.2.7.bb
commit 64390f742bea1137ef56a55dbea81e99adf35b64
Author: Tim Orling <timothy.t.orling@linux.intel.com>
Date:   Thu Jan 4 14:56:44 2018 -0800

    python[3]-setuptools: inherit setuptools

    * distutils[3].bbclass was not including pkg_resources module
    * Drop uneeded DISTUTILS_INSTALL_ARGS as setuptools[3].bbclass provides it

    Fixes [YOCTO #12148]

    (From OE-Core rev: 4e09525c73348c2b47b07654ac31ffc6cbd3d5fe)

    Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
------------------------

I think it's done for the backport.
Comment 11 Wenzong.Fan 2018-04-17 21:48:49 UTC
This defect has been fixed both on master and rocko.