Bug 169

Summary: [LTP] LTP default test suite cannot fully pass
Product: [Build System, Metadata & Runtime] Meta-yocto Reporter: Jiajun Xu <jiajun.xu>
Component: meta-yoctoAssignee: Kevin Tian <kevin.tian>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Low CC: jessica.zhang, kevin.tian, kunwei.tian, poky.bs.watcher, poky.watcher, richard.purdie, sgw, xianchao.zhang, yi.zhao
Version: 0.9   
Target Milestone: 1.0 M1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: ---
Attachments:
Description Flags
M3 LTP test result
none
Yocto 1.0 M2 LTP result none

Description Jiajun Xu 2010-07-29 20:33:29 UTC
Testing Env:    
Hardware: HP Mini Netbook
poky-image-sato-live image from milestone build (20100725) on autobuilder,
http://yoctobuild01.jf.intel.com/milestone/20100725-1/.
poky commit info: fbf2022c9a082edfa86d6acd7078dd6ad6968f71 from poky master
tree
LTP from http://sourceforge.net/projects/ltp/files/LTP%20Source/ltp-20100430/ltp-full-20100430.tgz/download

Some LTP default test suite cannot fully pass on poky. On netbook, following test components are performend: 
syscalls
fs
fsx
dio
io
mm
ipc
sched
math
nptl
pyt
admin_tools
timers
commands

And there are 34 cases failed with totally 1183 cases.

The failure case list:
gethostid01                    FAIL       1
lchown02                       FAIL       2
lchown02_16                    FAIL       2
sysctl03                       FAIL       1
syslog01                       FAIL       1
syslog02                       FAIL       1
syslog03                       FAIL       1
syslog04                       FAIL       1
syslog05                       FAIL       1
syslog07                       FAIL       1
syslog08                       FAIL       1
syslog09                       FAIL       1
syslog10                       FAIL       1
su01                           FAIL       1
cron02                         FAIL       1
cron_deny01                    FAIL       1
cron_allow01                   FAIL       1
cron_dirs_checks01             FAIL       1
at_deny01                      FAIL       1
at_allow01                     FAIL       1
clock_gettime03                FAIL       1
timer_create04                 FAIL       1
ar                             FAIL       1
ld                             FAIL       1
ldd                            FAIL       1
nm                             FAIL       1
file                           FAIL       9
tar                            FAIL       2
cron                           FAIL       10
logrotate                      FAIL       1
cpio                           FAIL       1
unzip01                        FAIL       127
gzip01                         FAIL       2
ln_tests01                     FAIL       1
Comment 1 Jiajun Xu 2010-09-17 05:56:02 UTC
Created attachment 47 [details]
M3 LTP test result
Comment 2 Jiajun Xu 2010-09-17 05:58:39 UTC
Attachment is M3 LTP test result for all targets.
Except for these failure cases, there are some cases no response when running in our testing. They are:

proc01
setfsgid03
setfsgid03_16
setfsuid03
setfsuid03_16
msgctl10
shm_test
Comment 3 Jiajun Xu 2010-12-27 18:11:11 UTC
Created attachment 83 [details]
Yocto 1.0 M2 LTP result
Comment 4 Jiajun Xu 2010-12-27 18:13:20 UTC
(In reply to comment #3)
> Created attachment 83 [details]
> Yocto 1.0 M2 LTP result

Attachment is Yocto 1.0 M2 LTP test result.
Comment 5 Kevin Tian 2010-12-27 18:34:51 UTC
Could you also attach a metric which components are included in this test for each target (qemu, boards, ...)? 

BTW, I took a quick lookd at qemux86 result. some test cases are not run this time, like at_deny01, ... Could you double confirm what have changed since last run?

I'd like to collect below info for each target:

LTP version:
Image Profile:
run command:
components:

Based on those information I can then update https://wiki.yoctoproject.org/wiki/LTP_result accordingly.
Comment 6 Jiajun Xu 2010-12-27 18:52:13 UTC
(In reply to comment #5)
> Could you also attach a metric which components are included in this test for
> each target (qemu, boards, ...)? 
> BTW, I took a quick lookd at qemux86 result. some test cases are not run this
> time, like at_deny01, ... Could you double confirm what have changed since last
> run?
> I'd like to collect below info for each target:
> LTP version:
> Image Profile:
> run command:
> components:
> Based on those information I can then update
> https://wiki.yoctoproject.org/wiki/LTP_result accordingly.

Oh, sorry, Kevin. I forgot to mention that we used a latest version of LTP this time. Detail information as below:

LTP version: http://sourceforge.net/projects/ltp/files/LTP%20Source/ltp-20101031/
============
Image Profile: For different targets, we use their SDK images accordingly.
============
run command: "./runltp -p -q -l result-M2-20101218.log -C  result-M2-20101218.fail -d /opt/tmp"
============
components:
For qemu targets, run following components
syscalls
mm
ipc
sched
math
nptl
pty
admin_tools
commands

For real hardware, we will run following components:
syscalls
fs
fsx
dio
io
mm
ipc
sched
math
nptl
pty
admin_tools
timers
commands

We will provide such information for each target in next time testing.
Comment 7 Kevin Tian 2010-12-27 18:59:46 UTC
Thanks Jiajun. those info are very useful. :-)
Comment 8 Kevin Tian 2011-01-30 00:46:58 UTC
I've finished the audit for all LTP failures in M2 except routerstation. Totally 6 bugs are created:

http://bugzilla.pokylinux.org/show_bug.cgi?id=672 [LTP]PATH contain ending colon for root user 
http://bugzilla.pokylinux.org/show_bug.cgi?id=673 [LTP]normal user can't use cron 
http://bugzilla.pokylinux.org/show_bug.cgi?id=674 [LTP]bash is not default /bin/sh in lsb profile
http://bugzilla.pokylinux.org/show_bug.cgi?id=675 [LTP]normal user can't use at
http://bugzilla.pokylinux.org/show_bug.cgi?id=676 [LTP]no 'file' and 'ldd' in sdk profile
http://bugzilla.pokylinux.org/show_bug.cgi?id=678 [LTP] fallocate always gets -EFBIG on qemuarm/qemumips
Comment 9 Kevin Tian 2011-02-17 23:33:44 UTC
we have moved all the information to https://wiki.yoctoproject.org/wiki/LTP_result, and thus should be able to close this bug