Bug 10677

Summary: \n in DESCRIPTION creates invalid (ipk) packages
Product: [Build System, Metadata & Runtime] OE-Core Reporter: André Draszik <byo>
Component: deploymentAssignee: Leonardo Sandoval Gonzalez <leonardo.sandoval.gonzalez>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: leonardo.sandoval.gonzalez, ross.burton
Version: 2.2.1   
Target Milestone: 2.4 M1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know

Description André Draszik 2016-11-16 09:11:30 UTC
As per
http://lists.openembedded.org/pipermail/openembedded-core/2016-November/128842.html

It looks like having explicit \n in the DESCRIPTION causes creation of ipks that can't be installed:

root@poky:~# opkg update
Package ncp-dbg version 1.2.4-r0 has no valid architecture, ignoring.
Package ncp-dev version 1.2.4-r0 has no valid architecture, ignoring.
Package ncp-doc version 1.2.4-r0 has no valid architecture, ignoring.
Package ncp version 1.2.4-r0 has no valid architecture, ignoring.

root@poky:~# opkg install ncp
Package ncp-dbg version 1.2.4-r0 has no valid architecture, ignoring.
Package ncp-dev version 1.2.4-r0 has no valid architecture, ignoring.
Package ncp-doc version 1.2.4-r0 has no valid architecture, ignoring.
Package ncp version 1.2.4-r0 has no valid architecture, ignoring.
Collected errors:
 * opkg_prepare_url_for_install: Couldn't find anything to satisfy 'ncp'.


I would think that this should either be caught early, i.e. bitbake should fail, or alternatively be fixed if possible.
Comment 1 Leonardo Sandoval Gonzalez 2016-11-22 17:25:50 UTC
Estimated time may increase, but setting to 1 to do the first analysis/setup.
Comment 2 André Draszik 2017-01-18 11:07:12 UTC
For reference, this version's DESCRIPTION caused the issue: http://lists.openembedded.org/pipermail/openembedded-devel/2016-November/109835.html

Whereas this is fine:
http://lists.openembedded.org/pipermail/openembedded-devel/2016-November/109885.html
Comment 3 André Draszik 2017-01-18 12:25:01 UTC
The comment from http://lists.openembedded.org/pipermail/openembedded-core/2016-November/128851.html is actually correct. The problem is empty lines anywhere in the DESCRIPTION, not just at the end. By making sure there are no empty lines, e.g by adding a 'dot', the resulting package can be installed without architecture mismatches (we can't use 'space' instead of 'dot', because opkg-utils' opkg.py runs textwrap.dedent() and strip() first):

> DESCRIPTION = "ncp is a utility for copying files in a LAN. It has absolutely no \n\
> security or integrity checking, no throttling, no features, except \n\
> one: you don't have to type the coordinates of your peer. \n\
> .\n\
> Please note that the DNS resolver does not use /etc/hosts (as that \n\
> would not work with IPv6 anyway), so you should have a properly \n\
> configured DNS server. \n\
> .\n\
> Basically it works like this: \n\
>    1. You and your buddy want to play Quake \n\
>    2. Your buddy has a level that you don't have \n\
>    3. He types npush filename and waits. npush sends out UDP \n\
>       announcement packets proclaiming that someone wants to send \n\
>       something \n\
>    4. You type npoll. npoll waits until it sees one of these packets \n\
>    5. The files are copied with tar over a TCP socket, so permissions \n\
>       and file dates (and if you are root, owners) are preserved. \n\
> .\n\
> There are other usage modes. You can also use it like this: \n\
>    peer1$ ncp \n\
>                            peer2$ ncp peer1 file1 file2 file3 \n\
> or like this: \n\
>    peer1$ npush file1 \n\
>                            peer2$ npoll peer1"

So I suppose the packaging could maybe be enhanced to either fail early in such a case, or to add the dots itself, or to skip the python textwrap part if pre-formatted text has been detected
Comment 4 Mariano Lopez 2017-01-23 18:22:07 UTC
(In reply to comment #3)
> So I suppose the packaging could maybe be enhanced to either fail early in
> such a case, or to add the dots itself, or to skip the python textwrap part
> if pre-formatted text has been detected

It seems the best option is to add a sanity test to check if the description empty lines.
Comment 5 Mariano Lopez 2017-01-23 18:24:07 UTC
(In reply to comment #4)
> (In reply to comment #3)
> > So I suppose the packaging could maybe be enhanced to either fail early in
> > such a case, or to add the dots itself, or to skip the python textwrap part
> > if pre-formatted text has been detected
> 
> It seems the best option is to add a sanity test to check if the description
> empty lines.

I hit the button too early, the best option would be to check if description has empty lines at build time and make the task fail, because opkg expect an empty line as terminator.
Comment 6 Mariano Lopez 2017-01-24 18:43:09 UTC
Sent patch for review:

https://patchwork.openembedded.org/series/4942/#
Comment 7 Ross Burton 2017-02-23 17:01:46 UTC
In general opkg matches dpkg, so using '.' to 'escape' blank lines seems like the best idea.
Comment 8 Mariano Lopez 2017-03-14 15:11:41 UTC
Moving to M4
Comment 9 Mariano Lopez 2017-04-05 19:00:02 UTC
Sent v2 to the mailing list, this version does what Ross suggested:

http://lists.openembedded.org/pipermail/openembedded-core/2017-April/135233.html

Patchwork:

https://patchwork.openembedded.org/series/6176/
Comment 10 Leonardo Sandoval Gonzalez 2017-04-13 15:20:47 UTC
Patch is out there, ping will be done during 2.4M1
Comment 11 Ross Burton 2017-04-18 20:33:36 UTC
Merged in oe-core 3e678d9b6a9eaeed76ce538d7f6ecf9f423864bc.
Comment 12 Ross Burton 2017-05-03 16:28:42 UTC
So I was wrong...

The format for empty lines in description is " .".

package_deb was just patched to fix the same problem there:

-                         # We don't limit the width when manually indent, but we do
-                         # need the textwrap.fill() to set the initial_indent and
-                         # subsequent_indent, so set a large width
-                         ctrlfile.write('%s\n' % textwrap.fill(t, width=100000, initial_indent=' ', subsequent_indent=' '))
+                         ctrlfile.write(' %s\n' % (t.strip() or '.'))
Comment 13 Leonardo Sandoval Gonzalez 2017-05-03 19:27:36 UTC
(In reply to comment #12)
> So I was wrong...
> 
> The format for empty lines in description is " .".
> 
> package_deb was just patched to fix the same problem there:
> 
> -                         # We don't limit the width when manually indent,
> but we do
> -                         # need the textwrap.fill() to set the
> initial_indent and
> -                         # subsequent_indent, so set a large width
> -                         ctrlfile.write('%s\n' % textwrap.fill(t,
> width=100000, initial_indent=' ', subsequent_indent=' '))
> +                         ctrlfile.write(' %s\n' % (t.strip() or '.'))

Patch sent with the correction:

http://lists.openembedded.org/pipermail/openembedded-core/2017-May/136287.html
Comment 14 Leonardo Sandoval Gonzalez 2017-05-08 14:38:09 UTC
This is suggested (by Ross) way to fix this problem

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=ross/mut&id=a428745d1594219413032d89065b522ff84a6ad3

So I will send another revision before M1.
Comment 15 Leonardo Sandoval Gonzalez 2017-05-22 17:34:14 UTC
V2 revision sent for review:

http://lists.openembedded.org/pipermail/openembedded-core/2017-May/136974.html
Comment 16 Leonardo Sandoval Gonzalez 2017-05-29 17:41:25 UTC
Sent v3

http://lists.openembedded.org/pipermail/openembedded-core/2017-May/137286.html