Bug 909

Summary: [LTP] mem02 failed on qemuarm sato-sdk/lsb-sdk image
Product: [Yocto Project Subprojects] Kernel Reporter: Jiajun Xu <jiajun.xu>
Component: kernel-toolingAssignee: Bruce Ashfield <bruce.ashfield>
Status: RESOLVED NOTABUG QA Contact:
Severity: normal    
Priority: Medium CC: dvhart, kevin.tian, liming.wang, richard.purdie, sgw, xudong.hao, yi.zhao, yp.kernel.watcher, yp.watcher
Version: unspecified   
Target Milestone: 1.0 PointRelease   
Hardware: x86   
OS: arm   
Whiteboard: passes tests on the latest builds. will close soon, needs more testing (Aug 12, 2011)
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: ---
Attachments:
Description Flags
the source file of mem02 none

Description Jiajun Xu 2011-03-18 06:49:58 UTC
Created attachment 121 [details]
the source file of mem02

Image location: http://autobuilder.pokylinux.org/nightly/20110305-4/machines/qemu/qemuarm/
Tree/Branch: Distro/Bernard
Commit: commit c192fefd868be53c20dfd8bc44e3e815cf53a10a
Author: Saul Wold <sgw@linux.intel.com>
Date:   Fri Mar 4 18:13:50 2011 -0800

    lsbsetup: Fix LIC_FILE_CHKSUM
    
    Signed-off-by: Saul Wold <sgw@linux.intel.com>



LTP mem02 case could pass in v1.0 M2 on qemuarm and could pass on other platforms. The failed log is as below:

####
tag=mem02 stime=1299575651
cmdline="mem02"
contacts=""
analysis=exit
<<<test_output>>>
mem02       0  TINFO  :  pm1=0x400c0008 pm2=0x400e8008 
mem02       1  TFAIL  :  free did not dealloc memory 
####
Comment 1 Bruce Ashfield 2011-03-21 13:19:20 UTC
very strange.

If the M2 kernel runs against the M4 userspace, does the result
change ?
Comment 2 Jiajun Xu 2011-03-23 07:36:30 UTC
cc Yi Zhao from Windriver, who may help to have a try.
Comment 3 Yi Zhao 2011-03-23 23:12:02 UTC
I just checked the result about this case on RC3 and gold build qumuarm images and found it can *pass* both on sato-sdk and lsb-sdk images.

<<<test_start>>>
tag=mem02 stime=1300870891
cmdline="mem02"
contacts=""
analysis=exit
<<<test_output>>>
mem02       1  TPASS  :  calloc - calloc of 64MB of memory succeeded
mem02       2  TPASS  :  malloc - malloc of 64MB of memory succeeded
mem02       3  TPASS  :  realloc - realloc of 5 bytes succeeded
mem02       4  TPASS  :  realloc - realloc of 15 bytes succeeded
<<<execution_status>>>
initiation_status="ok"
duration=7 termination_type=exited termination_id=0 corefile=no
cutime=501 cstime=217
<<<test_end>>>


Image location:
RC3 build: http://autobuilder.yoctoproject.org/nightly/20110313-2/machines/qemu/qemuarm/
commit a33a2cc024f9c9147c92c5da70d8fbb631b968f9
Author: Nitin A Kamble <nitin.a.kamble@intel.com>
Date:   Fri Mar 11 10:08:01 2011 -0800

    perl: another set of parallel build fixes
    
    [YOCTO #784]
    
    Imported more commits from the perl upstream tree
    
    (From OE-Core rev: c3b74b0c3833541ab5e89a7f9597f1ef8a413a70)
    
    Signed-off-by: Nitin A Kamble <nitin.a.kamble@intel.com>
    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>


Gold build: http://autobuilder.yoctoproject.org/nightly/20110318-3/machines/qemu/qemuarm/

commit 82af8b9fb6d6d1439943fd15b8a75c0e663e4a45
Author: Paul Eggleton <paul.eggleton@linux.intel.com>
Date:   Mon Feb 28 13:29:15 2011 +0000

    sstate.bbclass: always delete stamp files in sstate_clean
    
    For safety, always delete the stamp files in sstate_clean regardless of
    whether the manifest file exists or not.
    
    (From OE-Core rev: f781c35da9a11eefdb06bda72ca89753df863efa)
    
    Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Comment 4 Bruce Ashfield 2011-03-24 06:18:03 UTC
Thanks for the re-test. We'll wait on this and see if the problem is also shown to be fixed in further testing. But it does sound like this is either intermittent or resolved.
Comment 5 Bruce Ashfield 2011-04-28 19:25:18 UTC
I can't reproduce this, can we do a re-test ?
Comment 6 Yi Zhao 2011-05-04 04:33:31 UTC
I re-test it on 1.0 final release image and 20110423 build image (git commit 653cd1423793a3fd66da5c5682021d5c5b729cfe).

Due to bug 433, the qemuarm can not support > 128MB.

I wrote a while loop to run the 'mm' testcase for many times this afternoon.
For 1.0 quemarm sato-sdk image, the test ran 9 times and they all passed.
For 20110423 build qemuarm sato-sdk image, the test ran 9 times and the issue happened in the last 3 times.
Comment 7 Bruce Ashfield 2011-05-05 10:56:43 UTC
Hmm. This is very strange. There's been no changes to the kernel
between the report of success and failure. 

If there are re-boots between the test runs, do they all succeed ?
Comment 8 Yi Zhao 2011-05-25 04:05:22 UTC
I re-test it on 1.0 final release image, 1.0.1 image(git commit 1b159ff35d9f484e72afb8b19a9e69a2770d8379)and 20110423 build image (git commit
653cd1423793a3fd66da5c5682021d5c5b729cfe).

I run 'mm' testcase 10 times on each target. Between every test runs, I reboot the target.
For 1.0 quemuarm sato-sdk image, they all passed.
For 1.0.1 qemuarm sato-sdk image, they failed 2 times.
For 20110423 build qemuarm sato-sdk image, they failed 1 times.
Comment 9 Darren Hart 2011-05-25 22:22:44 UTC
Given the failure rates for the latter run, I don't think a sample size of 10 is sufficient to rule out the problem doesn't exist on 1.0. How long does the test take to run?
Comment 10 Yi Zhao 2011-06-13 02:25:49 UTC
I wrote a while loop to run the 'mm' testcase for 100 times on 1.0 final release image and 1.0.1 image(git commit 1b159ff35d9f484e72afb8b19a9e69a2770d8379). 

Between every test there is no reboot. Running 1 'mm' testcase  would take about 35 minutes, so 100 times would take about 60 hours. 
 
For 1.0 quemuarm sato-sdk image, they all passed. The failure rate is 0%.
For 1.0.1 qemuarm sato-sdk image, they failed 25 times. The failure rate is 25%.

And I checked the latest result on 1.1 M1 build. it passed both on sato-sdk and lsb-sdk.
Comment 11 Darren Hart 2011-06-15 09:50:02 UTC
Those numbers are pretty convincing, looks like a regressino.
Comment 12 Bruce Ashfield 2011-06-15 11:13:46 UTC
what changed in 1.0.1 that could have impacted this thought ? there were virtually no kernel changes. If this is working in 1.1, then it really should be the same as 1.0.1, since the kernel is again the same.

bottom line, we need to see non-kernel changes that went into 1.0.1, since I don't see this as a kernel issue. Who can get us that list ?
Comment 13 Yi Zhao 2011-08-17 23:01:52 UTC
I tested them again on qemuarm and beagleboard with 1.1 M3 build (git commit abdec68876ac7ce1a6d5c223d1757b7e8a14b85f)

Run the 'mm' testcase for 100 times with a loop script. Between every test there is no reboot. 

For quemuarm sato-sdk image, they failed 21 times.
For beagleboard sato-sdk image, they failed 18 times.
Comment 14 Liming Wang 2011-08-26 02:47:46 UTC
This bug shouldn't be a real bug, such link has disscussed what mem02 does:

http://www.mail-archive.com/ltp-list@lists.sourceforge.net/msg00543.html
http://www.mail-archive.com/ltp-list@lists.sourceforge.net/msg00546.html

Above mailing said:
On Tue, 2007-09-18 at 08:48 -0400, Jeff Burke wrote:
> Folks,
>       I have attached a patch that provides additional INFO when the test
> fails in the calloc portion of the test. The second part of the patch

>       The issue I have is that after the free(pm1), when it does the calloc
> the second time there is no guaranty that pm1 will equal pm2. It does
> 99% of the time. But I have seen this fail on ppc64 intermittently.
> 
> Thanks,
> Jeff

You are correct. There is no guaranty that pm1=pm2 when it mallocs
again. But, your patch for finding out the values of pm1 and pm2 should
be useful. Patching the same.
--Subrata--
Comment 15 Liming Wang 2011-08-26 03:01:46 UTC
One question is that why mem02 test only fails on arm architecture? That's to say, why the first calloc and the second calloc return different address on arm.

It depends on how the architecture implements "arch_get_unmapped_area" function.

We know calloc calls mmap to get memory, which calls kernel function as follows:
sys_mmap2()
  -> mmap_pgoff()
      ->do_mmap_pgoff()
          -> get_unmapped_area()
             -> current->mm->get_unmapped_area() 
                ->arch_get_unmapped_area()

And arch_get_unmapped_area is to get unmmaped area.
All architectures' implementation of arch_get_unmapped_area, except arm, don't randomize the returned address of unmmaped area.

For arm arch, it imports a commit to randomize the address of mmap:

commit cc92c28b2db5b406657ecc05235d4ca4e222ae34
Author: Nicolas Pitre <nico@fluxnic.net>
Date:   Mon Jun 14 21:16:19 2010 -0400

    [ARM] add address randomization to mmap()
    
    Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org>

It's imported in:
cc92c28b2db5b406657ecc05235d4ca4e222ae34 tags/v2.6.36-rc1~591^2^2~6^2~2

So we can see this case failure on arm after v2.6.36-rc1. If we revert this commit, this case works well.
I suggest we can reject this bug and mark mem02 as known issue for arm arch.
Comment 16 Bruce Ashfield 2011-08-26 05:59:46 UTC
Liming: good analysis. I double checked the code and the links and I agree. This
isn't a bug, but expected behaviour. So I'm closing this case.

I'm not sure who's tracking known ltp/posix failures, but I'll find out and get this added.
Comment 17 Saul Wold 2011-08-26 08:19:56 UTC
(In reply to comment #16)
> Liming: good analysis. I double checked the code and the links and I agree.
> This
> isn't a bug, but expected behaviour. So I'm closing this case.
> 
> I'm not sure who's tracking known ltp/posix failures, but I'll find out and get
> this added.

We did have someone tracking this on the wiki at:

https://wiki.yoctoproject.org/wiki/LTP_result

But he is no longer working in the group, please go ahead and add this info to the wiki.
Comment 18 Jiajun Xu 2011-08-26 08:43:37 UTC
(In reply to comment #16)
> Liming: good analysis. I double checked the code and the links and I agree.
> This
> isn't a bug, but expected behaviour. So I'm closing this case.
> I'm not sure who's tracking known ltp/posix failures, but I'll find out and get
> this added.

Thanks, Liming, Bruce, Saul,
Yi Zhao and I will add it to the ltp/posix wiki.