Bug 11510 - Improve the way we handle python's packaging (autopackaging)
Summary: Improve the way we handle python's packaging (autopackaging)
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: devtools / tool chain (show other bugs)
Version: 2.4
Hardware: x86 Multiple
: Medium enhancement
Target Milestone: 2.5 M2
Assignee: Ross Burton
QA Contact:
URL:
Whiteboard: LET
Depends on:
Blocks: 11694 11695
  Show dependency tree
 
Reported: 2017-05-12 18:06 UTC by Alejandro Hernandez
Modified: 2018-01-22 14:16 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alejandro Hernandez 2017-05-12 18:06:08 UTC
Improve the way we handle python packaging:

Manifest file is created by a script, on which we specify the "parts" or files from upstream python that should be installed for a certain python package.

Problems:
   - Unorganized (Hard to read)
   - Lots of new bugs
   - Impossible to know if the required files for a certain package have changed (more or less dependencies), or for new packages unless we install and test them all one by one, and even then we wouldn't know if they require less dependencies.

Goals:
   - New format for manifest file (JSON)
   - Make it work on top of “testimage” so its created correctly and automagically
Comment 1 Alejandro Hernandez 2017-06-20 23:16:45 UTC
This will broken up into three bugs:

This one:
Will create a new manifest file (JSON), on which all packages specify what modules and files they require, this is basically just a replacement of the manifest we use at the moment with a more organized one, since it will include a script to "convert" the new manifest to the old one so bitbake can still parse it the same way.

Second bug/enchancement:
This one will handle the part on which packages are parsed, this should be done via python, by reading the JSON manifest file directly, and modifying bitbake's dictionary on the go.

Up to this point, the manifest replacement will be complete, but other than it being more organized and having got rid of the script to create it (confusing), it will still have the same other problems stated above.

Third bug/enhancement:
This one will create a test which may be run on top of testimage, that will check every python package to see specifically which modules it needs, and will create a manifest file (JSON) automatically which can be used to replace the current file, this can be run either on every release or when adding a new package if one is lazy enough to not check the modules the new package needs to work.
Comment 2 Alejandro Hernandez 2017-08-28 21:45:34 UTC
Done, but moving to 2.5 per Richard's request.
Comment 3 Ross Burton 2018-01-22 14:16:50 UTC
Fixed in oe-core 3328211afdef8ffb00dd4dff1143959d5412b075.