Bug 14446 - devtool add malformed PACKAGECONFIG
Summary: devtool add malformed PACKAGECONFIG
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: Scripts and Tools (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 4.1 M2
Assignee: Thomas Roos
QA Contact:
URL:
Whiteboard: NEWCOMER
Depends on:
Blocks:
 
Reported: 2021-06-21 18:44 UTC by aatir.manzur
Modified: 2025-02-14 16:48 UTC (History)
7 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description aatir.manzur 2021-06-21 18:44:28 UTC
devtool add python3-uvicorn https://files.pythonhosted.org/packages/e9/9e/25d59f5043cf763833b2581c8027fa92342c4cf8ee523b498ecdf460c16d/uvicorn-0.14.0.tar.gz

results in a recipe with malformed PACKAGECONFIG[]
Comment 1 Ross Burton 2021-06-21 19:44:34 UTC
The malformed PACKAGECONFIG looks like this:

PACKAGECONFIG[:python_version < "3.8"] = ",,,python3-typing-extensions"
PACKAGECONFIG[standard] = ",,,python3-pyyaml python3-httptools python3-python-dotenv python3-watchgod python3-websockets"
PACKAGECONFIG[standard:sys_platform != "win32" and (sys_platform != "cygwin" and platform_python_implementation != "pypy")] = ",,,python3-uvloop!"
PACKAGECONFIG[standard:sys_platform == "win32"] = ",,,python3-colorama"


Which is taken from the extras_require parameter to setuptools.
Comment 2 Abongwa Bonalais Amahnui 2022-04-11 09:11:10 UTC
Hello Ross, I wish to ask if this bug is resolved since its been almost a year since this was put up.
Comment 3 Abongwa Bonalais Amahnui 2022-04-13 11:12:36 UTC
And also, I would like to know how to reproduce this bug.
Comment 4 Saul Wold 2022-04-13 13:24:10 UTC
Abongwa,

Yes it still reproduces with master, the way to see this is run the devtool command that is listed in the first comment.  I tested this yesterday.

The problem is that pysetuptools sees the uvicorn.egg-info/requires.txt as extra requirements.  Recipetool parses this information to generate the PACKAGECONFIG lines, specifically the python parsing function in create_buildsys_python.py (in scripts/lib/recipetool). There is a set of characters in the replacements variable and code in apply_info_replacements(). This code probably needs to be fixed to handle the "bad" lines:

[:python_version < "3.8"]
typing-extensions

[standard:sys_platform != "win32" and (sys_platform != "cygwin" and platform_pyt
hon_implementation != "PyPy")]
uvloop!=0.15.0,!=0.15.1,>=0.14.0

[standard:sys_platform == "win32"]
colorama>=0.4

Are you interested in helping out on this, if so, you can assign the bug to yourself.
Comment 5 Thomas Roos 2022-06-10 09:26:30 UTC
did pick this bug as my first to solve (NEWCOMER)
I think I found a solution, but I'm a bit unsure what the expected behavior is.
I don't see any support for expressions in PACKAGECONFIG eg. 
PACKAGECONFIG[:python_version < "3.8"] so those should not be added to the bb recipe.
I add a simple check to exclude any PACKAGECONFIG, which contains an expression.
Is this the right way to go or is something else expected?
Comment 6 Saul Wold 2022-06-10 17:14:53 UTC
Yes, that would be the general approach, it would be good if possible to provide a warning saying that recipetool was unable to parse or add those PACKAGECONFIG either as output or as a comment in the new recipe bb file.
Comment 7 Thomas Roos 2022-06-20 12:21:15 UTC
I did rework my patch (https://bpa.st/SXEQ) instead of just omitting that malformed lines my current approach is to replace the expression (parsed from requires.txt) by a parsable replacement that can be enabled manually. 
E.g.
#PACKAGECONFIG ?= "pythonversionsmaller3dot8 standard standardsysplatformnotequalswin32andsysplatformnotequalscygwinandplatformpythonimplementationnotequalspypy standardsysplatformequalsequalswin32"
PACKAGECONFIG[pythonversionsmaller3dot8] = ",,,python3-typing-extensions"
PACKAGECONFIG[standard] = ",,,python3-pyyaml python3-httptools python3-python-dotenv python3-watchgod python3-websockets"
PACKAGECONFIG[standardsysplatformnotequalswin32andsysplatformnotequalscygwinandplatformpythonimplementationnotequalspypy] = ",,,python3-uvloop!"
PACKAGECONFIG[standardsysplatformequalsequalswin32] = ",,,python3-colorama"

This will work for the uvicorn package. Is there any test to check that my modifications do not break anything?
Comment 8 Saul Wold 2022-06-22 14:19:56 UTC
(In reply to comment #7)
> I did rework my patch (https://bpa.st/SXEQ) instead of just omitting that
> malformed lines my current approach is to replace the expression (parsed
> from requires.txt) by a parsable replacement that can be enabled manually. 
> E.g.
> #PACKAGECONFIG ?= "pythonversionsmaller3dot8 standard
> standardsysplatformnotequalswin32andsysplatformnotequalscygwinandplatformpyth
> onimplementationnotequalspypy standardsysplatformequalsequalswin32"
> PACKAGECONFIG[pythonversionsmaller3dot8] = ",,,python3-typing-extensions"
> PACKAGECONFIG[standard] = ",,,python3-pyyaml python3-httptools
> python3-python-dotenv python3-watchgod python3-websockets"
> PACKAGECONFIG[standardsysplatformnotequalswin32andsysplatformnotequalscygwina
> ndplatformpythonimplementationnotequalspypy] = ",,,python3-uvloop!"
> PACKAGECONFIG[standardsysplatformequalsequalswin32] = ",,,python3-colorama"
> 
> This will work for the uvicorn package. Is there any test to check that my
> modifications do not break anything?

I would suggest adding spaces for better readability.  You could just change the replace lines to add a space for the "" or add spaces around the words your replacing.

No, but you should consider adding a test using the uvicorn package to ensure it's parsed correctly.  Take a look at meta-selftest.
Comment 9 Thomas Roos 2022-06-22 15:13:54 UTC
spaces are not allowed, but - (hyphens) are ok. Should improve the readability as lot as well:

#PACKAGECONFIG ?= "python-version-smaller-3-dot-8 standard standardsys-platform-not--equals-win32andsys-platform-not--equals-cygwinandplatform-python-implementation-not--equals-pypy standardsys-platform-equals--equals-win32"
PACKAGECONFIG[python-version-smaller-3-dot-8] = ",,,python3-typing-extensions"
PACKAGECONFIG[standard] = ",,,python3-pyyaml python3-httptools python3-python-dotenv python3-watchgod python3-websockets"
PACKAGECONFIG[standardsys-platform-not--equals-win32andsys-platform-not--equals-cygwinandplatform-python-implementation-not--equals-pypy] = ",,,python3-uvloop!"
PACKAGECONFIG[standardsys-platform-equals--equals-win32] = ",,,python3-colorama"
Comment 10 Thomas Roos 2022-07-05 07:21:37 UTC
unsure about the bugzilla workflow
I did send my patch to the mailing list. It got merged into master-next and master
https://lists.openembedded.org/g/openembedded-core/message/167334?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3ACreated%2C%2CPACKAGECONFIG%2C20%2C2%2C0%2C92039954
- so what is next? 
- Who should close the bug?
Comment 11 Ross Burton 2022-07-05 09:43:38 UTC
You can, as you sent the patch, but I'll do it now.

Fixed in oe-core a854d95a79e64f3f82abfa4cc1daec750abf4249
Comment 12 Randy MacLeod 2025-02-14 16:46:24 UTC
bulk change to add Saul's non-WR address.
Comment 13 Randy MacLeod 2025-02-14 16:48:26 UTC
Bulk change: Remove Saul's old WR address.