Bug 13217 - "bitbake -c create_manifest python" automatically overwrites upstream python-manifest.json, breaking python build
Summary: "bitbake -c create_manifest python" automatically overwrites upstream python-...
Status: RESOLVED NOTABUG
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: devtools / tool chain (show other bugs)
Version: 2.0.3
Hardware: x86 Multiple
: Low minor
Target Milestone: Future
Assignee: Unassigned
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2019-03-07 19:09 UTC by Kirk Hays
Modified: 2025-03-27 15:00 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
Generated manifest for target. (3.62 KB, text/plain)
2019-03-07 19:09 UTC, Kirk Hays
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Kirk Hays 2019-03-07 19:09:28 UTC
Created attachment 4446 [details]
Generated manifest for target.

In our embedded devices, we require a very small subset of python ("core", "json", "lang") - that's what we use, and, as we are storage constrained, it is manditory for us to trim back the target python build.

Using the "bitbake -c create_manifest python" does exactly what we require, with a replacement "python2-manifest.json" from our layer.  After the trimmed manifest is generated, we can check it into our vendor meta-layer, and life is happy.

Getting there, however, involved stepping on a rake that *might* violate the intent of poky recipes, in general:

To wit, the sumo "python2_2.7.15.bb" recipe, in task "do_create_manifest()", at line 255, copies the newly generated manifest back to "${THISDIR}", overwriting the original manifest from upstream, automatically:

"cp python2-manifest.json.new ${THISDIR}/python/python2-manifest.json"

While the original intent of "only do this when upgrading python version" is laudable, it also causes the python build to break, as it replaces the default manifest with the trimmed manifest, in the current checkout, which is not appropriate for then building native-python.

I have marked this as "minor" severity, as, once the replacement manifest is in our meta-layer, this won't happen, unless we try "create_manifest", again.  It did take me a day to figure out that creating our target manifest was breaking the native-python build, though.

Perhaps someone else will benefit from this being fixed.

To recreate:

1.  Checkout a clean copy of poky, sumo branch.

2.  Create a vendor meta-layer with a replacement "python2-manifest.json" (I've attached ours as an example).

2.  Utter "bitbake -c create_manifest python"

3.  In the poky checkout, utter "git status".

4.  Note the output:

On branch sumo
Your branch is up-to-date with 'origin/sumo'.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   meta/recipes-devtools/python/python/python2-manifest.json

no changes added to commit (use "git add" and/or "git commit -a")

5.  Utter "bitbake -c cleanall python; bitbake -c create_manifest python; bitbake -c cleanall python; bitbake python", and enjoy the resulting hilarity:

Loading cache: 100% |##########################################################################################################################################################################################################| Time: 0:00:00
Loaded 1430 entries from dependency cache.
Parsing recipes: 100% |########################################################################################################################################################################################################| Time: 0:00:21
Parsing of 931 .bb files complete (929 cached, 2 parsed). 1432 targets, 277 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "1.38.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "arm-poky-linux-gnueabi"
MACHINE              = "biamp-zynq"
DISTRO               = "poky"
DISTRO_VERSION       = "2.0"
TUNE_FEATURES        = "arm armv7a vfp thumb neon callconvention-hard cortexa9"
TARGET_FPU           = "hard"
meta                 
meta-poky            
meta-yocto-bsp       = "sumo:d3ad2438222050faf33e83598c1f6ecd25ff65b6"
meta-gplv2           = "sumo:d7687d404bbc9ba3f44ec43ea8828d9071033513"
meta-qt4             = "sumo:8e791c40140460825956430ba86b6266fdec0a93"
meta-xilinx-bsp      = "sumo:5fccc46503e468ed024185ed032891799a31db58"
meta-biamp           = "sumo:a30e3642680ddf96806658b8bb3f4e963285acf2"

Initialising tasks: 100% |#####################################################################################################################################################################################################| Time: 0:00:00
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
ERROR: python-2.7.15-r1 do_package_qa: QA Issue: non -staticdev package contains static .a library: python-misc path '/work/cortexa9hf-neon-poky-linux-gnueabi/python/2.7.15-r1/packages-split/python-misc/usr/lib/python2.7/config/libpython2.7.a' [staticdev]
ERROR: python-2.7.15-r1 do_package_qa: QA Issue: /usr/bin/pydoc contained in package python requires /usr/bin/python2.7, but no providers found in RDEPENDS_python? [file-rdeps]
ERROR: python-2.7.15-r1 do_package_qa: QA run found fatal errors. Please consider fixing them.
ERROR: python-2.7.15-r1 do_package_qa: Function failed: do_package_qa
ERROR: Logfile of failure stored in: /home/khays/source/biamp-linux-build/build-biamp-linux-zynq/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/python/2.7.15-r1/temp/log.do_package_qa.12172
ERROR: Task (/home/khays/source/biamp-linux-build/build-biamp-linux-zynq/../poky/meta/recipes-devtools/python/python_2.7.15.bb:do_package_qa) failed with exit code '1'
NOTE: Tasks Summary: Attempted 610 tasks of which 590 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /home/khays/source/biamp-linux-build/build-biamp-linux-zynq/../poky/meta/recipes-devtools/python/python_2.7.15.bb:do_package_qa
Summary: There were 4 ERROR messages shown, returning a non-zero exit code.
Comment 1 Richard Purdie 2019-03-14 14:37:34 UTC
Looks like this is working as designed. I do understand the problem, I'm wondering if the code could be smarter about figuring out where the original manifest came from. With that in mind we'll keep this open but its low priority
Comment 2 Kirk Hays 2019-03-14 16:08:09 UTC
Agreed, it may work as designed, but I would argue that it violates the "principal of least surprise" (https://en.wikipedia.org/wiki/Principle_of_least_astonishment
), in that "bitbake -c create_manifest" does not update $WORKDIR, or something under $WORKDIR, but instead updates $THISDIR, which, in the typical usage case, is a pristine layer.

I've run across no other bitbake action that does this...which may reflect on my level of experience with bitbake, as I've only been using it for five years.

Perhaps it is a documentation issue, instead.
Comment 3 Randy MacLeod 2025-03-27 15:00:07 UTC
Old bug cleanup.

This is doing what you should expect, if you'd like to a documentation change, it will be reviewed and likely merged. Thanks.