Bug 8180

Summary: extensible SDK: silent error when using ipk packaging
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Paul Eggleton <bluelightning>
Component: devtools / tool chainAssignee: Benjamin Esquivel <benjamin.esquivel>
Status: VERIFIED FIXED QA Contact: Alexandru Georgescu <alexandru.c.georgescu>
Severity: normal    
Priority: Medium+ CC: bogdanx.a.voiculescu, meta.mr.watcher, meta.watcher
Version: 1.8   
Target Milestone: 1.9 M3   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
python stack trace if the exception is raised none

Description Paul Eggleton 2015-08-18 14:19:16 UTC
If you use ipk packaging and run bitbake core-image-minimal -c populate_sdk_ext and then look at log.do_populate_sdk_ext, you can see there are a couple of errors that don't cause the task to fail:

('movefile: Failed to move', '/home/paul/poky/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/var/lib/opkg/status', 'to', '/home/paul/poky/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/var/lib/opkg', OSError(21, 'Is a directory'))
('movefile: Failed to move', '/home/paul/poky/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/var/lib/opkg/lists', 'to', '/home/paul/poky/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/var/lib/opkg', OSError(39, 'Directory not empty'))

We're not checking the return of our calls to movefile() which is why the task doesn't fail. However, I'm not entirely sure why we're even getting these errors in the first place. Ultimately we don't care about the opkg database in the SDK anyway but it would be worth investigating.
Comment 1 Benjamin Esquivel 2015-08-20 16:46:10 UTC
I got this error reproduced:

('movefile: Failed to move', '/home/besquive/poky-ipk/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/var/lib/opkg/info', 'to', '/home/besquive/poky-ipk/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/var/lib/opkg', OSError(17, 'File exists'))
('movefile: Failed to move', '/home/besquive/poky-ipk/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/var/lib/opkg/status', 'to', '/home/besquive/poky-ipk/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/var/lib/opkg', OSError(21, 'Is a directory'))
Comment 2 Benjamin Esquivel 2015-08-20 19:50:56 UTC
It seems like meta/lib/oe/sdk.py, where movefile is invoked, checks for no return codes of anything it calls. At least it would be good to put some try checks around bb.utils.* calls (bb.utils.movefile is the object of this bug)
Comment 3 Benjamin Esquivel 2015-08-20 20:15:31 UTC
sadly, the way that bb.utils.movefile deals with exceptions removes the ability to re-raise or let un-handled exceptions continue their own raise path. Will check the cost for having bb.utils.movefile let go exceptions. I bet this will be a fun test.
Comment 4 Benjamin Esquivel 2015-08-20 20:33:03 UTC
Created attachment 2671 [details]
python stack trace if the exception is raised

Attached a python stack trace that follows the path of the failure if the exception continues it's raise path.
Comment 5 Benjamin Esquivel 2015-08-20 21:36:58 UTC
the movefile function in bb.utils is doing something like:

os.rename(path/to/file, new/path/)

and the os.rename function specifies in the manual:

'...If dst is a directory, OSError will be raised.'

which is the same error we're getting here.

I did a quick test to do instead:

os.rename(path/to/file, new/path/file)

and the errors are gone.

I'm questioning whether this works for rpm/deb or if this is something ipk is doing wrong (pass of the src/dest paths to the movefile function)
Comment 6 Benjamin Esquivel 2015-08-20 21:49:06 UTC
This error also happens in deb/rpm
Comment 7 Benjamin Esquivel 2015-08-20 22:04:58 UTC
a first approach to fix this:

http://lists.openembedded.org/pipermail/openembedded-core/2015-August/109117.html
Comment 8 Benjamin Esquivel 2015-08-21 18:12:18 UTC
A second and better approach to fix it:

http://lists.openembedded.org/pipermail/openembedded-core/2015-August/109190.html

now checks for the path join to do it to a dir, avoid doing it to a file name at the end which would be bogus
Comment 9 Benjamin Esquivel 2015-08-26 21:36:34 UTC
created a related bug:  the sdk.py script does not check for return codes

https://bugzilla.yoctoproject.org/show_bug.cgi?id=8213
Comment 10 Benjamin Esquivel 2015-09-10 17:57:36 UTC
Now it shows up in master:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=7c079463c96891f0dbf6d7744c0d6bfe7af1e0ca
Comment 11 Alexandru Georgescu 2015-11-02 16:50:05 UTC
(In reply to comment #10)
> Now it shows up in master:
> 
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/
> ?id=7c079463c96891f0dbf6d7744c0d6bfe7af1e0ca

Verified with jethro:370a2916c4559c17952392b872322e8e490b124c

Steps:
1. modify conf/local.conf to use ipk packaging:

PACKAGE_CLASSES ?= "package_ipk"
added SSTATE_DIR and DL_DIR

2. build the sdk_ext: 

autobuilder@sstate1:~/tmp/poky/build-jethro-7629-sdk-ext$ bitbake core-image-minimal -c populate_sdk_ext


I will try again without using SSTATE_DIR / DL_DIR, than set it to verified.
Comment 12 Alexandru Georgescu 2015-11-03 06:44:34 UTC
(In reply to comment #11)
> (In reply to comment #10)
> > Now it shows up in master:
> > 
> > http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/
> > ?id=7c079463c96891f0dbf6d7744c0d6bfe7af1e0ca
> 
> Verified with jethro:370a2916c4559c17952392b872322e8e490b124c
> 
> Steps:
> 1. modify conf/local.conf to use ipk packaging:
> 
> PACKAGE_CLASSES ?= "package_ipk"
> added SSTATE_DIR and DL_DIR
> 
> 2. build the sdk_ext: 
> 
> autobuilder@sstate1:~/tmp/poky/build-jethro-7629-sdk-ext$ bitbake
> core-image-minimal -c populate_sdk_ext
> 
> 
> I will try again without using SSTATE_DIR / DL_DIR, than set it to verified.

Verified without SSTATE_DIR and DL_DIR