Bug 6484

Target link is sato-sdk.

Will test tomorrow for other BSPs.

Did not find bug on other BSPs. Seems only NUC is affected.

Bug not found on qemu.

Nitin, can you see if you can reproduce this behavior? I suspect this is NOT a BSP issue, but since it didn't reproduce on qemu for Mihail, we should at least try to reproduce before handing off to the core team.

ok, now qemu data is out. I am planning to reproduce this myself, and see what is going on.

Thanks,
Nitin

I rebuilt the nuc sato sdk image with exact same commits here. And there are no file system permission issues as mentioned in this bug. I could not reproduce the issue. While I tried downloading the nuc core sdk image from the AB, and it does have the permission issues.
  So this points the issue back to the AB environment.

The permissions inside the rootfs dir of the core sdk image recipe are good. Means something went wrong in packaging.


/home/pokybuild/yocto-autobuilder/yocto-slave/nuc/build/build/tmp/work/nuc-poky-linux/core-image-sato-sdk/1.0-r0/rootfs/usr/bin> ls -l *g++*
lrwxrwxrwx 1 pokybuild users     21 Jun 25 05:15 g++ -> x86_64-poky-linux-g++
-rwxr-xr-x 1 pokybuild users 809168 Jun 25 04:23 x86_64-poky-linux-g++

/home/pokybuild/yocto-autobuilder/yocto-slave/nuc/build/build/tmp/work/nuc-poky-linux/core-image-sato-sdk/1.0-r0/rootfs/usr/bin> ls -l *python*
lrwxrwxrwx 1 pokybuild users    7 Jun 25 05:15 python -> python2
lrwxrwxrwx 1 pokybuild users    9 Jun 25 05:15 python2 -> python2.7
-rwxr-xr-x 1 pokybuild users 5048 Jun 25 04:06 python2.7
-rwxr-xr-x 1 pokybuild users 1618 Jun 25 04:05 python2.7-config
lrwxrwxrwx 1 pokybuild users   16 Jun 25 05:15 python2-config -> python2.7-config
lrwxrwxrwx 1 pokybuild users   14 Jun 25 05:15 python-config -> python2-config

inside the rootfs.img file g++ has right permissions, while python script is not executable.

So the python2.7 script's permissions got messed up in the rootfs.img creation process. And the x86_64-poky-linux-g++ script's permissions got messed up afterwords.


nitin@yocto-hm1:/tmp> sudo mount rootfs.img  mnt
nitin@yocto-hm1:/tmp> cd mnt/usr/bin/

nitin@yocto-hm1:/tmp/mnt/usr/bin> ls -l *g++*
lrwxrwxrwx 1 root root     21 Jun 25 05:15 g++ -> x86_64-poky-linux-g++*
-rwxr-xr-x 1 root root 809168 Jun 25 04:23 x86_64-poky-linux-g++*

nitin@yocto-hm1:/tmp/mnt/usr/bin> ls *python* -l
lrwxrwxrwx 1 1006 users    7 Jun 25 05:15 python -> python2
lrwxrwxrwx 1 root root     9 Jun 25 05:15 python2 -> python2.7
-rw-r--r-- 1 root users 5048 Jun 25 04:06 python2.7
-rwxr-xr-x 1 root root  1618 Jun 25 04:05 python2.7-config*
lrwxrwxrwx 1 root root    16 Jun 25 05:15 python2-config -> python2.7-config*
lrwxrwxrwx 1 root root    14 Jun 25 05:15 python-config -> python2-config*

This is not a meta-intel layer related issue, and looks like issue is in packaging of rootfs or post-install of python & gcc packages on the autobuilder environment, assigning the bug owner as Saul, to delegate or look further.

Adding Mark Hatle to the CC list.

not sure, but might be related to bug 6500

I believe this issue occurred during a very small window when pseudo was being updated. 

There was a random permissions issue that was created for about day or so.. I suspect this is the problem.

I'm going to close this and if it reoccurs we should open it again.

Mihai will verify.
Also, Mihai please see if a TC is needed for this types of behavior and if so, add it to our runtime testing.

Thanks!

Verified on master: 2d1660112e54653f7bb763939d0416472c49fe01
TC isn't needed. If this problem reappears our existing TCs will pick it up.

I just managed to replicate this on the autobuilder:

https://autobuilder.yoctoproject.org/main/builders/nightly-qa-systemd/builds/41/steps/Running%20Sanity%20Tests_1/logs/stdio

As previously, the rootfs directory in tmp/work has the right permissions, but the image doesn't.

So I downloaded the file system and looked at it:

$ ls *python* -l
lrwxrwxrwx 1 root root    7 Sep 16 01:27 python -> python2
lrwxrwxrwx 1 root root    9 Sep 16 01:27 python2 -> python2.7
-rwxr-xr-x 1 root root 5064 Sep 15 23:31 python2.7
-rwxr-xr-x 1 root root 1618 Sep 15 23:30 python2.7-config
lrwxrwxrwx 1 root root   16 Sep 16 01:27 python2-config -> python2.7-config
lrwxrwxrwx 1 root root   14 Sep 16 01:27 python-config -> python2-config

Not funny.

Issue has not reappeared in any of the images we test normally.
This seems to point to something going wrong during AB operations, as Nitin suggested.

Ross,

Is there anything the QA team can do to help you?

Lets close it again as worksforme until some can replicate again, ideally outside the autobuilder.

Verified. See previous comment.

Closing.

Happened again https://autobuilder.yoctoproject.org/main/builders/nightly-ipk/builds/52/steps/Running%20Sanity%20Tests_2/logs/stdio.

The permissions are corrupt in the work/rootfs directory:

[ross@centos7 bin]$ ls -l pytho*
lrwxrwxrwx. 1 pokybuild users     7 Oct  1 00:20 python -> python2
lrwxrwxrwx. 1 pokybuild users     9 Oct  1 00:20 python2 -> python2.7
-rw-r--r--. 1 pokybuild users 11215 Sep 30 22:39 python2.7
-rw-r--r--. 1 pokybuild users  1618 Sep 30 22:39 python2.7-config
lrwxrwxrwx. 1 pokybuild users    16 Oct  1 00:20 python2-config -> python2.7-config
lrwxrwxrwx. 1 pokybuild users    14 Oct  1 00:20 python-config -> python2-config

[ross@centos7 package]$ ls -l usr/bin/
total 48
-rw-r--r--. 2 pokybuild users    96 Apr  9  2012 2to3
-rw-r--r--. 2 pokybuild users    95 Apr  9  2012 idle
-rw-r--r--. 2 pokybuild users    79 Apr  9  2012 pydoc
lrwxrwxrwx. 1 pokybuild users     7 Sep 30 22:39 python -> python2
lrwxrwxrwx. 1 pokybuild users     9 Sep 30 22:39 python2 -> python2.7
-rw-r--r--. 2 pokybuild users 11215 Sep 30 22:39 python2.7
-rw-r--r--. 2 pokybuild users  1618 Sep 30 22:39 python2.7-config
lrwxrwxrwx. 1 pokybuild users    16 Sep 30 22:39 python2-config -> python2.7-config
lrwxrwxrwx. 1 pokybuild users    14 Sep 30 22:39 python-config -> python2-config
-rw-r--r--. 2 pokybuild users 18543 Apr  9  2012 smtpd.py


[ross@centos7 image]$ ls -l usr/bin/
total 48
-rwxr-xr-x. 1 pokybuild users    96 Apr  9  2012 2to3
-rwxr-xr-x. 1 pokybuild users    95 Apr  9  2012 idle
-rwxr-xr-x. 1 pokybuild users    79 Apr  9  2012 pydoc
lrwxrwxrwx. 1 pokybuild users     7 Sep 30 22:39 python -> python2
lrwxrwxrwx. 1 pokybuild users     9 Sep 30 22:39 python2 -> python2.7
-rwxr-xr-x. 1 pokybuild users 11215 Sep 30 22:39 python2.7
-rwxr-xr-x. 1 pokybuild users  1618 Sep 30 22:39 python2.7-config
lrwxrwxrwx. 1 pokybuild users    16 Sep 30 22:39 python2-config -> python2.7-config
lrwxrwxrwx. 1 pokybuild users    14 Sep 30 22:39 python-config -> python2-config
-rwxr-xr-x. 1 pokybuild users 18543 Apr  9  2012 smtpd.py

is this build still available?

Can you get me the full contents of the affected directories?

I'm looking specifically at the run scripts, pseudo database and pseudo logs for failure conditions.

The perms on the actual filesystem may be different (but usually '+x' is preserved there.)  But it's really the permissions on the pseudo side (captured in the db) that are interesting.

The build directory is still in place if you have ssh access to that builder.  If not, I've archived the python build directory already, where is the pseudo db kept?

Pseudo DB is located in the tmp/work/.../${pn}/${pv}/pseudo

So if you have two whole work directory, we should have everything we need to investigate.

The ../pseudo/*.log files would be the first place to investigate and see if there is an entry for the files with the wrong information.

Is this useful:

sqlite> select path, mode from files where path like "%usr/bin/python2.7";
.../work/i586-poky-linux/python/2.7.3-r0.3/image/usr/bin/python2.7|33261

.../work/i586-poky-linux/python/2.7.3-r0.3/packages-split/python-core/usr/bin/python2.7|33188

That's interesting. The one in image/ is 755, the one in packages-split is 644.

I am not entirely sure what permissions I expect to really have on the filesystem, but I think normally I'd expect to get reasonable permissions for things; pseudo masks out write bits in some cases, but I don't think it's masking out execute bits, in general.

It seems very odd that this is only sometimes happening.

Is there a "pseudo.log" file in the build directory? If so, any contents?

Created attachment 2157 [details]
pseudo.log

pseudo.log attached.

Oh-hoh.

creat for '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-ipk/build/build/tmp/work/i586-poky-linux/python/2.7.3-r0.3/package/usr/lib/python2.7/encodings/hz.pyc' replaces existing 4869771627 ['/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-ipk/build/build/tmp/work/i586-poky-linux/python/2.7.3-r0.3/package/usr/bin/pydoc'].

I can tell you what this means, and the fact that it is happening tells us roughly what's going wrong.

What it means is, at some point in the past, pseudo became aware of inode 4869771627, which was usr/bin/pydoc. Then a request came in for encodings/hz.pyc, using the same inode number. But pseudo never saw the old file removed, so it reports a mismatch.

What usually causes this:
* Files being removed by a process which isn't running under pseudo.
* Files being removed or recreated by some unrelated thing, like if objcopy were running outside of pseudo, that could result in replacing a file with another file with a new inode number.

So the question, basically, is what happened to package/usr/bin/pydoc between when it was created and when this message showed up.

Suspected cause is 64-bit inodes going into a 32-bit data type in sqlite.  pseudo has been fixed in git.

pseudo 1.6.2 is in master-next.

And was merged in b2c6a032d6e5deb07e76ed75fcd0931fad6a748c.

Marking as closed, as there is no reliable way to verify. Will revisit if problem resurfaces.