Bug 8304 - smart sanity test failed to find filelist.xml.gz
Summary: smart sanity test failed to find filelist.xml.gz
Status: VERIFIED WORKSFORME
Alias: None
Product: Package Management Issues
Classification: Runtime
Component: RPM (show other bugs)
Version: 1.9
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 2.0
Assignee: Mariano Lopez
QA Contact: Lucian Musat
URL: http://errors.yoctoproject.org/Errors...
Whiteboard:
Depends on:
Blocks: 8249
  Show dependency tree
 
Reported: 2015-09-15 18:31 UTC by Saul Wold
Modified: 2015-10-28 15:59 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: Regression (Found by QA test)
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Saul Wold 2015-09-15 18:31:59 UTC
During the SmartRepoTest, the filelists.xml.gz is not available, this appears to be a transient issue and may have resulted from another build attempting to write that file at the same time this test was running.


======================================================================
FAIL: test_smart_channel_add (oeqa.runtime.smart.SmartRepoTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/meta/lib/oeqa/utils/decorators.py", line 102, in wrapped_f
    return func(*args)
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/meta/lib/oeqa/runtime/smart.py", line 80, in test_smart_channel_add
    self.smart('update')
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/meta/lib/oeqa/utils/decorators.py", line 90, in wrapped_f
    return f(*args)
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/meta/lib/oeqa/runtime/smart.py", line 22, in smart
    self.assertEqual(status, expected, message)
AssertionError: 1 != 0 : smart update
Loading cache...
Updating cache...               ######################################## [100%]

                                                                               
Fetching information for 'all'...
                                                                               
-> http://192.168.7.7:33381/rpm/all/repodata/repomd.xml
repomd.xml                      ######################################## [ 16%]
                                                                               
-> http://192.168.7.7:33381/rpm/all/repodata/filelists.xml.gz
filelists.xml.gz                ######################################## [ 25%]
                                                                               
-> http://192.168.7.7:33381/rpm/all/repodata/primary.xml.gz
primary.xml.gz                  ######################################## [ 33%]
                                                                               
Fetching information for 'qemux86_64'...
                                                                               
-> http://192.168.7.7:33381/rpm/qemux86_64/repodata/repomd.xml
repomd.xml                      ######################################## [ 50%]
error: Failed acquiring information for 'qemux86_64':
error: http://192.168.7.7:33381/rpm/qemux86_64/repodata/filelists.xml.gz: None
                                                                               
Fetching information for 'core2_64'...
                                                                               
-> http://192.168.7.7:33381/rpm/qemux86_64/repodata/primary.xml.gz
primary.xml.gz                  ######################################## [ 66%]
                                                                               
-> http://192.168.7.7:33381/rpm/qemux86_64/repodata/filelists.xml.gz
filelists.xml.gz                ######################################## [ 75%]
                                                                               
-> http://192.168.7.7:33381/rpm/core2_64/repodata/repomd.xml
repomd.xml                      ######################################## [ 83%]
                                                                               
-> http://192.168.7.7:33381/rpm/core2_64/repodata/primary.xml.gz
primary.xml.gz                  ######################################## [ 91%]
                                                                               
-> http://192.168.7.7:33381/rpm/core2_64/repodata/filelists.xml.gz
filelists.xml.gz                ######################################## [100%]

Updating cache...               ######################################## [100%]

Channels have 4667 new packages.
Saving cache...

----------------------------------------------------------------------
Ran 46 tests in 1170.443s

FAILED (failures=1, skipped=15)
NOTE: Stopping logging thread
NOTE: Stop event received
NOTE: Tearing down logging thread
NOTE: Sending SIGTERM to runqemu
DEBUG: Python function do_testimage finished
ERROR: Function failed: core-image-lsb - FAILED - check the task log and the ssh log
Comment 1 Benjamin Esquivel 2015-09-21 19:36:44 UTC
Seems to be the same or very similar report:

http://errors.yoctoproject.org/Errors/Details/17546/
Comment 2 Benjamin Esquivel 2015-09-22 22:36:49 UTC
I left this test running iterated to see if I can run into the failure.

So far, 8 iterations and no failures.
Comment 3 Richard Purdie 2015-10-16 15:44:51 UTC
Could be due to qemu networking glitch but how/why/when?
Comment 4 Benjamin Esquivel 2015-10-16 21:58:58 UTC
Looking for more occurrences of the error for analysis
Comment 5 Mariano Lopez 2015-10-19 19:37:07 UTC
Unfortunately the logs on the autobuilder are gone (qemu, ssh, tests, and dumps)
Comment 6 Mariano Lopez 2015-10-20 02:10:26 UTC
Tried to reproduce the bug simulating a network error between the host and the target (I disable eth0 in the target for 30 seconds and the I enabled it again). Here is the log:

| ======================================================================
| FAIL: test_smart_channel_add (oeqa.runtime.smart.SmartRepoTest)
| ----------------------------------------------------------------------
| Traceback (most recent call last):
|   File "/home/mlopezva/repo/poky/meta/lib/oeqa/utils/decorators.py", line 106, in wrapped_f
|     return func(*args)
|   File "/home/mlopezva/repo/poky/meta/lib/oeqa/runtime/smart.py", line 80, in test_smart_channel_add
|     self.smart('update')
|   File "/home/mlopezva/repo/poky/meta/lib/oeqa/utils/decorators.py", line 94, in wrapped_f
|     return f(*args)
|   File "/home/mlopezva/repo/poky/meta/lib/oeqa/runtime/smart.py", line 22, in smart
|     self.assertEqual(status, expected, message)
| AssertionError: 1 != 0 : smart update
| Loading cache...
| Updating cache...               ######################################## [100%]
| 
Fetching information for 'all'...                                                
-> http://192.168.7.1:38460/rpm/all/repodata/repomd.xml                          
| repomd.xml                      ######################################## [ 16%]
-> http://192.168.7.1:38460/rpm/all/repodata/filelists.xml.gz                    
| filelists.xml.gz                ######################################## [ 25%]
-> http://192.168.7.1:38460/rpm/all/repodata/primary.xml.gz                      
| primary.xml.gz                  ######################################## [ 33%]
| 
| Fetching information for 'qemux86_64'...
-> http://192.168.7.1:38460/rpm/qemux86_64/repodata/repomd.xml                   
| repomd.xml                      ######################################## [ 50%]
-> http://192.168.7.1:38460/rpm/qemux86_64/repodata/filelists.xml.gz             
| filelists.xml.gz                ######################################## [ 58%]
-> http://192.168.7.1:38460/rpm/qemux86_64/repodata/primary.xml.gz               
| primary.xml.gz                  ######################################## [ 66%]
| 
| Fetching information for 'core2_64'...
-> http://192.168.7.1:38460/rpm/core2_64/repodata/repomd.xml                     
| repomd.xml                                                               [ 75%]
| error: Failed acquiring release file for 'core2_64':
| error: http://192.168.7.1:38460/rpm/core2_64/repodata/repomd.xml: [Errno 101] Network is unreachable
| 
| Updating cache...               ######################################## [100%]
| 
| Channels have 476 new packages.
| Saving cache...
| 
| ======================================================================
| FAIL: test_smart_local_dir (oeqa.runtime.smart.SmartRepoTest)
| ----------------------------------------------------------------------
| Traceback (most recent call last):
|   File "/home/mlopezva/repo/poky/meta/lib/oeqa/utils/decorators.py", line 106, in wrapped_f
|     return func(*args)
|   File "/home/mlopezva/repo/poky/meta/lib/oeqa/runtime/smart.py", line 145, in test_smart_local_dir
|     self.smart('download psplash')
|   File "/home/mlopezva/repo/poky/meta/lib/oeqa/utils/decorators.py", line 94, in wrapped_f
|     return f(*args)
|   File "/home/mlopezva/repo/poky/meta/lib/oeqa/runtime/smart.py", line 22, in smart
|     self.assertEqual(status, expected, message)
| AssertionError: 1 != 0 : smart download psplash
| Loading cache...
| Updating cache...               ######################################## [100%]
| error: psplash-0.1+git0+14c8f7b705-r15@core2_64 is not available for downloading
| 
| Saving cache...
| 
| ----------------------------------------------------------------------
| Ran 46 tests in 1461.044s
| 
| FAILED (failures=2, skipped=22)
Comment 7 Mariano Lopez 2015-10-20 14:22:10 UTC
Here is the log when removing the file from the tmp/deploy/rpm/repodata directory.


======================================================================
FAIL: test_smart_channel_add (oeqa.runtime.smart.SmartRepoTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mlopezva/repo/poky/meta/lib/oeqa/utils/decorators.py", line 106, in wrapped_f
    return func(*args)
  File "/home/mlopezva/repo/poky/meta/lib/oeqa/runtime/smart.py", line 80, in test_smart_channel_add
    self.smart('update')
  File "/home/mlopezva/repo/poky/meta/lib/oeqa/utils/decorators.py", line 94, in wrapped_f
    return f(*args)
  File "/home/mlopezva/repo/poky/meta/lib/oeqa/runtime/smart.py", line 22, in smart
    self.assertEqual(status, expected, message)
AssertionError: 1 != 0 : smart update
Loading cache...
Updating cache...               ######################################## [100%]

                                                                               Fetching information for 'all'...
                                                                               -> http://192.168.7.1:41442/rpm/all/repodata/repomd.xml
repomd.xml                      ######################################## [ 16%]
                                                                               -> http://192.168.7.1:41442/rpm/all/repodata/primary.xml.gz
primary.xml.gz                  ######################################## [ 25%]
                                                                               -> http://192.168.7.1:41442/rpm/all/repodata/filelists.xml.gz
filelists.xml.gz                ######################################## [ 33%]
                                                                               
Fetching information for 'qemux86_64'...
                                                                               -> http://192.168.7.1:41442/rpm/qemux86_64/repodata/repomd.xml
repomd.xml                                                               [ 41%]
error: Failed acquiring release file for 'qemux86_64':
error: http://192.168.7.1:41442/rpm/qemux86_64/repodata/repomd.xml: File not found
                                                                               
Fetching information for 'core2_64'...
                                                                               -> http://192.168.7.1:41442/rpm/core2_64/repodata/repomd.xml
repomd.xml                      ######################################## [ 83%]
                                                                               -> http://192.168.7.1:41442/rpm/core2_64/repodata/filelists.xml.gz
filelists.xml.gz                ######################################## [ 91%]
                                                                               -> http://192.168.7.1:41442/rpm/core2_64/repodata/primary.xml.gz
primary.xml.gz                  ######################################## [100%]

Updating cache...               ######################################## [100%]

Channels have 4982 new packages.
Saving cache...

----------------------------------------------------------------------
Ran 46 tests in 783.945s

FAILED (failures=1, skipped=22)
Comment 8 Mariano Lopez 2015-10-20 14:32:06 UTC
I did another test with a timeout artificially low to check if it would be the load in the autobuilders but it seems unlikely. Here is the log:

======================================================================
FAIL: test_smart_channel_add (oeqa.runtime.smart.SmartRepoTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mlopezva/repo/poky/meta/lib/oeqa/utils/decorators.py", line 106, in wrapped_f
    return func(*args)
  File "/home/mlopezva/repo/poky/meta/lib/oeqa/runtime/smart.py", line 78, in test_smart_channel_add
    self.smart('channel -y --add {a} type=rpm-md baseurl={u}/{a}'.format(a=arch, u=deploy_url))
  File "/home/mlopezva/repo/poky/meta/lib/oeqa/utils/decorators.py", line 94, in wrapped_f
    return f(*args)
  File "/home/mlopezva/repo/poky/meta/lib/oeqa/runtime/smart.py", line 22, in smart
    self.assertEqual(status, expected, message)
AssertionError: 255 != 0 : smart channel -y --add qemux86_64 type=rpm-md baseurl=http://192.168.7.1:58044/rpm/qemux86_64

Process killed - no output for 15 seconds. Total running time: 15 seconds.
Comment 9 Mariano Lopez 2015-10-20 14:33:33 UTC
With the previous three logs it seems the problem was not a network error, file not found, or reaching the timeout. Here is a summary of the logs for different failures:

ORIGINAL FAILURE
Log: error: http://192.168.7.7:33381/rpm/qemux86_64/repodata/filelists.xml.gz: None

NETWORK ERROR
Log: error: http://192.168.7.1:38460/rpm/core2_64/repodata/repomd.xml: [Errno 101] Network is unreachable

REMOVE FILE
Log: error: http://192.168.7.1:41442/rpm/qemux86_64/repodata/repomd.xml: File not found

TIMEOUT
Log: Process killed - no output for X seconds. Total running time: X seconds. (unitest log)
Comment 10 Mariano Lopez 2015-10-27 16:40:45 UTC
This is the only occurrence of this bug and I wasn't able to reproduce. Closing and if the error happens again it will be reopened.
Comment 11 Lucian Musat 2015-10-28 15:59:44 UTC
Setting VERIFIED as per last comment.