Bug 2638

Summary: errors while running license_create_manifest - searching for files in ${TMPDIR}/pkgdata/*/runtime
Product: [Build System, Metadata & Runtime] Meta-yocto Reporter: Andrei Gherzan <andrei>
Component: meta-yoctoAssignee: Andrei Gherzan <andrei>
Status: VERIFIED FIXED QA Contact:
Severity: normal    
Priority: Medium CC: bluelightning, elizabeth.flanagan, florin.sarbu, laurentiu.serban, poky.bs.watcher, poky.watcher, ross.burton, sgw
Version: unspecified   
Target Milestone: 1.3 M2   
Hardware: x86   
OS: Multiple   
Whiteboard: (autobuilder)
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: ---

Description Andrei Gherzan 2012-06-20 18:17:03 UTC
While creating an image the do_rootfs log contains a lot of errors like warnings like these:

[...]
ls: cannot access /media/HDD-WRS/Projects/Yocto/2012-06-19-14-46-vea9-excalibur/tmp-eglibc-eglibc/pkgdata/*/runtime/libasound2: No such file or directory                                                                                    
ls: cannot access /media/HDD-WRS/Projects/Yocto/2012-06-19-14-46-vea9-excalibur/tmp-eglibc-eglibc/pkgdata/*/runtime/libattr1: No such file or directory                                                                                      
ls: cannot access /media/HDD-WRS/Projects/Yocto/2012-06-19-14-46-vea9-excalibur/tmp-eglibc-eglibc/pkgdata/*/runtime/libblkid1: No such file or directory                                                                                     
ls: cannot access /media/HDD-WRS/Projects/Yocto/2012-06-19-14-46-vea9-excalibur/tmp-eglibc-eglibc/pkgdata/*/runtime/libc6: No such file or directory                                                                                         
ls: cannot access /media/HDD-WRS/Projects/Yocto/2012-06-19-14-46-vea9-excalibur/tmp-eglibc-eglibc/pkgdata/*/runtime/libcap2: No such file or directory                                                                                       
[...]

This is generated by the fact that while generating license.manifest package information is searched in:
filename=`ls ${TMPDIR}/pkgdata/*/runtime/${pkg}| head -1`

Well, this is ok as long as package name is the same as the package name after packaging - for example dbus is transformed in dbus-1. So, searching ls ${TMPDIR}/pkgdata/*/runtime/dbus-1 will fail because the searched values is the PKG_dbus value. 

I have two fixes for this.

1. One which won't impact the build time:
Patch package.bbclass to create symlinks for all packages which are packaged differently as provided in PACKAGES. For example dbus-1 will be a symlink to dbus. In this way the above line will find the correct file for every INSTALLED_PKGS.

2. One which will impact a little the do_rootfs time:
Patch license.bbclass - don't search for a specific file in license_create_manifest but search for the file name which contains the PKG_* value as iterated in INSTALLED_PACKAGES.

I have patches for both scenarios. Just need an advice.

One more thing. This issue was introduced by this commit:

commit 885e1f7e6886c783ae481afc7c08bac04843cae9
Author: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
Date:   Wed May 30 12:01:15 2012 -0700
    license.bbclass: optimize pkg runtime data
    This fixes an ugly way I was trying to find pkg runtime
    data for package and license manifest creation.

At that time packages similar to the one exampled above were just excluded form license manifest.

I can submit patch anytime.

@g
Comment 1 Beth Flanagan 2012-06-20 18:31:04 UTC
Can you link me to the patches?

I'm planning on rewriting license_create_manifest to be event driven and to just utilize the datastore values it has during do_install which will remove the need to search for the package name. ETA on this is by the end of M2.
Comment 2 Andrei Gherzan 2012-06-20 18:36:24 UTC
(In reply to comment #1)
> Can you link me to the patches?
> 
> I'm planning on rewriting license_create_manifest to be event driven and to
> just utilize the datastore values it has during do_install which will remove
> the need to search for the package name. ETA on this is by the end of M2.

Patches are as simple as this:

1 solution:
Add in emit_pkgdata from packages.bbappend while generation pkgdata:
pkgval = d.getVar('PKG_%s' % (pkg), True)                                                                                                                                                                                    
<------><------>if pkgval != pkg:                                                                                                                                                                                                            
<------><------><------>subdata_sym = pkgdatadir + "/runtime/%s" % pkgval                                                                                                                                                                    
<------><------><------>os.symlink(pkg,subdata_sym)

2 solution:
Replace
filename=`ls ${TMPDIR}/pkgdata/*/runtime/${pkg}| head -1`
with
filename=`grep -lr "^PKG_.*${pkg}$" ${TMPDIR}/pkgdata/*/runtime/*`

This could be a good idea until your work is finished.

@g
Comment 3 Beth Flanagan 2012-06-20 18:47:53 UTC
(In reply to comment #2)
> (In reply to comment #1)
> > Can you link me to the patches?
> > 
> > I'm planning on rewriting license_create_manifest to be event driven and to
> > just utilize the datastore values it has during do_install which will remove
> > the need to search for the package name. ETA on this is by the end of M2.
> 
> Patches are as simple as this:
> 
> 1 solution:
> Add in emit_pkgdata from packages.bbappend while generation pkgdata:
> pkgval = d.getVar('PKG_%s' % (pkg), True)                                   
> 
> <------><------>if pkgval != pkg:                                           
> 
> <------><------><------>subdata_sym = pkgdatadir + "/runtime/%s" % pkgval   
> 
> <------><------><------>os.symlink(pkg,subdata_sym)
> 
> 2 solution:
> Replace
> filename=`ls ${TMPDIR}/pkgdata/*/runtime/${pkg}| head -1`
> with
> filename=`grep -lr "^PKG_.*${pkg}$" ${TMPDIR}/pkgdata/*/runtime/*`
> 
> This could be a good idea until your work is finished.
> 
> @g

Yeah, I can see the utility here for 1. Number 2 I'm less enthusiastic about as I'd like this to effect do_rootfs time as little as possible. Let me test out 1 later this week so I can see what the implications are. It should be ok, but I like treading lightly around package.bbclass.
Comment 4 Andrei Gherzan 2012-06-20 18:49:58 UTC
Ok. Waiting for feedback and i will submit patch.

@g
Comment 5 Beth Flanagan 2012-06-25 15:27:39 UTC
Ok, I looked at this a bit and the more I think about it, the more I think #1 is the right answer here. So, yes, ACK.
Comment 6 Andrei Gherzan 2012-06-28 08:20:56 UTC
Patch submitted.
http://lists.linuxtogo.org/pipermail/openembedded-core/2012-June/024551.html

@g
Comment 7 Andrei Gherzan 2012-07-06 10:05:11 UTC
@Saul Ping on this. :)
Comment 8 Saul Wold 2012-07-19 14:37:45 UTC
*** Bug 2784 has been marked as a duplicate of this bug. ***
Comment 9 Paul Eggleton 2012-07-23 21:47:47 UTC
I've posted a new version of Andrei's patch with the feedback addressed as part of the rootfs rework patchset. Sorry to rush on this but I needed the fix so I could do my own reverse pkgdata lookups.
Comment 10 Andrei Gherzan 2012-07-24 20:39:48 UTC
Sorry for my late reply on this but i was on a mini-vacation.

No worries Paul. I will try to find your fix in my 400 emails... Could you pass me a copy please?

ag
Comment 11 Paul Eggleton 2012-07-24 20:42:58 UTC
Sure, here it is:

http://patchwork.openembedded.org/patch/32871/
Comment 12 Andrei Gherzan 2012-07-25 18:12:06 UTC
Acked.
Comment 13 Paul Eggleton 2012-07-26 15:09:01 UTC
Merged to master in revision 5334a93d261598757ccce48a78b8ba2b7e5806f3.
Comment 14 Laurentiu Serban 2012-07-31 08:58:53 UTC
I could not reproduce this issue with commit fb53d69d9c726e6a663ea276cc440676e20f8326 so setting to Verified