Bug 7232

Summary: file: magic.mgc wrong architecture in x86_64 sdk
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Jack Mitchell <jack>
Component: coreAssignee: Jack Mitchell <jack>
Status: RESOLVED WORKSFORME QA Contact:
Severity: normal    
Priority: Medium CC: belen.barros.pena, fabien.proriol, meta.mr.watcher, meta.watcher, richard.purdie
Version: unspecified   
Target Milestone: 1.8   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
file generate with Yocto 1.7.0 (Good)
none
Same file generate with 1.7.1 (BAD) none

Description Jack Mitchell 2015-01-22 12:41:01 UTC
in recent x86_64 sdks file fails with the following errors. I believe this is due to the magic.mgc file being the wrong architecture for the SDK. Pointing file at the hosts magic.mgc make file behave as expected.

/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1883: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1885: Warning: offset `' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1885: Warning: type `' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1886: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1886: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1888: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1888: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1890: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1890: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1892: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1892: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1894: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1894: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1896: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1896: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1904: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1904: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1906: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1906: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1908: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1908: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1911: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1911: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1913: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1913: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1915: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1915: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1917: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1917: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1919: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1919: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1921: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1921: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1923: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1923: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1925: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1925: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1927: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1927: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1929: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1929: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1931: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1931: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1933: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1933: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1936: Warning: offset `-' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1936: Warning: type `-' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1944: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1944: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1948: Warning: offset `-' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1948: Warning: type `-' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1980: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/magic.mgc, 1980: Warning: type `.' invalid
Comment 1 Saul Wold 2015-01-26 19:09:23 UTC
I tried to reproduce this with a bitbake core-image-minimal -c populate_sdk, 
this produced a proper SDK installer and correctly installed file.  I did not 
see the issue mentioned in this report.

$ which file
/opt/poky/1.7/sysroots/x86_64-pokysdk-linux/usr/bin/file
$ file --version
file-5.22
magic file from 
/opt/poky/1.7/sysroots/x86_64-pokysdk-linux/usr/share/misc/magic.mgc

Can you please confirm which file and use --version to show which magic.mgc, 
although it appears to be the correct file.

Any additional information about how you built your SDK install would be 
helpful.

Thanks
Comment 2 Jack Mitchell 2015-01-27 13:49:27 UTC
Ok more details below, in your test case did you actually try to use file, on a file? My target build is an i686 core2 with the SDK build for x86_64.

Working with modified magic file:

[jack@jackArch ~]$ whereis file
file: /usr/bin/file /usr/share/file /mnt/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/bin/file /mnt/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/bin/file.real /usr/share/man/man1/file.1p.gz /usr/share/man/man1/file.1.gz /usr/share/man/mann/file.n.gz
[jack@jackArch ~]$ which file
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/bin/file
[jack@jackArch ~]$ file --version
file-5.22
magic file from /usr/share/file
[jack@jackArch ~]$ file ~/test/version.c 
/home/jack/test/version.c: ASCII text

Non-working with oe-core magic file

[jack@jackArch ~]$ file --magic-file /eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc/ ~/test/version.c 
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc//magic.mgc, 1: Warning: offset `�
     ' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc//magic.mgc, 1: Warning: type `�
   ' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc//magic.mgc, 6: Warning: offset `Firmware v' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc//magic.mgc, 6: Warning: type `Firmware v' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc//magic.mgc, 7: Warning: offset `' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc//magic.mgc, 7: Warning: type `' invalid

.....

/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc//magic.mgc, 1936: Warning: type `-' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc//magic.mgc, 1944: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc//magic.mgc, 1944: Warning: type `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc//magic.mgc, 1948: Warning: offset `-' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc//magic.mgc, 1948: Warning: type `-' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc//magic.mgc, 1980: Warning: offset `.' invalid
/eng/rc/sdk/oe-core/6/x86_64/sysroots/x86_64-oecore-linux/usr/share/misc//magic.mgc, 1980: Warning: type `.' invalid
file: could not find any valid magic files!
[jack@jackArch ~]$
Comment 3 condo4 2015-02-19 09:31:19 UTC
I can see the same problem on the dizzy branch.


If I call 
> file foo.jpeg
I have errors:
/opt/ste/tools/Yocto-my-1.7.1/sysroots/x86_64-my-linux/usr/share/misc/magic.mgc, 1840: Warning: type `.' invalid

file: Size of `/opt/ste/tools/Yocto-my-1.7.1/sysroots/x86_64-my-linux/usr/share/misc/magic.mgc' 2847788 is not a multiple of 248

I think this bug was appeared between 1.7.0 and 1.7.1; I need to verify this, but, I follow the head of the branch dizzy and this bug appeared 1 or 2 months ago...
Comment 4 condo4 2015-02-19 09:38:01 UTC
Attached 2 magic.mgc:
magic.mgc was generate with Yocto 1.7.0 some months ago and works
magic.mgc.corrupt was the same file generate with 1.7.1 and corrupts

My host is x86_64
Comment 5 condo4 2015-02-19 09:39:28 UTC
Created attachment 2416 [details]
file generate with Yocto 1.7.0 (Good)
Comment 6 condo4 2015-02-19 09:40:07 UTC
Created attachment 2417 [details]
Same file generate with 1.7.1 (BAD)
Comment 7 Richard Purdie 2015-03-28 10:27:54 UTC
I tried to reproduce this both on master and on dizzy branch head. I couldn't do so in either case:

$ /tmp/3/sysroots/x86_64-pokysdk-linux/usr/bin/file /bin/bash
/bin/bash: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=cb63ec0718f2022619814c04a5b6cd8a36752a83, stripped

Is this bug still occurring for people?
Comment 9 Richard Purdie 2015-03-28 10:46:25 UTC
Looking further, I think this is the same magic.h issue as reported on the list. I'm going t tweak the CVE patches to only change magic.h.in, not magic.h and push to the daisy/dizzy branches. It would be helpful if people could test and see if this resolves the problem for them.
Comment 10 Richard Purdie 2015-03-28 11:10:54 UTC
Its a BUILT_SOURCES file so that doesn't explain things. At this point I'm not at all sure what the issue here is or why 1.7.1 would introduce it, unless it is the wrapper script doing something unexpected.
Comment 11 Richard Purdie 2015-03-28 14:21:42 UTC
The nativesdk-file configure/compile logs would be interesting for a broken toolchain...
Comment 12 Jack Mitchell 2015-03-28 20:20:07 UTC
I'm afraid I've moved on from my last position and as such don't have access to the build machine that this bug originally arose on.

However, what may have been causing it is that I was doing x86_64 and i686 SDK builds, so it could be that an artifact from the 32 bit build was being transfered over to the 64 bit build. It might be worth cleaning the build, then performing a 32 bit sdk build, followed by a 64 bit sdk build and see if that changes anything.
Comment 13 Richard Purdie 2015-05-21 15:12:44 UTC
We really need more information about how to reproduce this. Jack doesn't have access and I can't reproduce. If anyone does reproduce this and has more info please reopen.