Bug 12114 - oe-selftest on Ubuntu1704 hangs due to a resource warning of unclosed file.
Summary: oe-selftest on Ubuntu1704 hangs due to a resource warning of unclosed file.
Status: VERIFIED FIXED
Alias: None
Product: Build Testing
Classification: QA/Testing
Component: general (show other bugs)
Version: 2.3.2
Hardware: Other x86_64
: Medium+ normal
Target Milestone: 2.5 M1
Assignee: Sonia Leon
QA Contact:
URL:
Whiteboard: Backport 2.3.3?
Depends on:
Blocks:
 
Reported: 2017-09-21 23:26 UTC by oscar lopez arandas
Modified: 2017-12-07 15:07 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
fulllog (28.36 KB, text/x-log)
2017-09-21 23:27 UTC, oscar lopez arandas
no flags Details
full_log (98.28 KB, application/octet-stream)
2017-10-31 18:19 UTC, oscar lopez arandas
no flags Details
all_tescases_pass (92.64 KB, text/plain)
2017-12-07 15:07 UTC, oscar lopez arandas
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description oscar lopez arandas 2017-09-21 23:26:41 UTC
Ubuntu 17.04 seems to hang when running the oe-selftest for release 2.3.2 rc1. With the following WARNING: 
/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-ubuntu1704/build/bitbake/lib/bb/server/__init__.py:39: ResourceWarning: unclosed file <_io.FileIO name=32 mode='wb' closefd=True>
  self._idlefuns = {} 

Initial research points this to be a resource issue in the GDC AB worker
. This bug is filed in so that this could be investigated further more.

STATUS: FAILED
BUILD: 2.3.2.rc1:ce26a57e04ad65c02087629701d96448a44e73d5
ENVIRONMENT: Ubuntu 17.04 x86-64

STEPS TO REPRODUCE: 

1 Clone poky 
2 source environment
3 git checkout
4 oe-selftest --run-all 

EXPECTED OUTCOME: 
  All tests cases should be executed correctly 

ACTUAL OUTCOME:
was reproduced 3 times and resulted the same fault
==============================================================================
WARNING: /home/ab/autobuilder/yocto-worker/nightly-oe-selftest-ubuntu1704/build/bitbake/lib/bb/server/__init__.py:39: ResourceWarning: unclosed file <_io.FileIO name=32 mode='wb' closefd=True>
  self._idlefuns = {}


================================================================================
Full log attached
Comment 1 oscar lopez arandas 2017-09-21 23:27:41 UTC
Created attachment 4020 [details]
fulllog
Comment 2 Jose Perez C 2017-09-22 12:02:48 UTC
Assigning to Sonia who is GDC AB maintainer for further investigation
Comment 3 Leonardo Sandoval Gonzalez 2017-09-22 16:04:37 UTC
(In reply to comment #0)
> Ubuntu 17.04 seems to hang when running the oe-selftest for release 2.3.2
> rc1. With the following WARNING: 
> /home/ab/autobuilder/yocto-worker/nightly-oe-selftest-ubuntu1704/build/
> bitbake/lib/bb/server/__init__.py:39: ResourceWarning: unclosed file
> <_io.FileIO name=32 mode='wb' closefd=True>
>   self._idlefuns = {} 
> 
> Initial research points this to be a resource issue in the GDC AB worker
> . This bug is filed in so that this could be investigated further more.
> 
> STATUS: FAILED
> BUILD: 2.3.2.rc1:ce26a57e04ad65c02087629701d96448a44e73d5
> ENVIRONMENT: Ubuntu 17.04 x86-64
> 
> STEPS TO REPRODUCE: 
> 
> 1 Clone poky 
> 2 source environment
> 3 git checkout
> 4 oe-selftest --run-all 
> 
> EXPECTED OUTCOME: 
>   All tests cases should be executed correctly 
> 
> ACTUAL OUTCOME:
> was reproduced 3 times and resulted the same fault
> =============================================================================
> =
> WARNING:
> /home/ab/autobuilder/yocto-worker/nightly-oe-selftest-ubuntu1704/build/
> bitbake/lib/bb/server/__init__.py:39: ResourceWarning: unclosed file
> <_io.FileIO name=32 mode='wb' closefd=True>
>   self._idlefuns = {}
> 
> 
> =============================================================================
> ===
> Full log attached

I have seen this log which  indeed requires further investigation.

Is the latter causing test failure? python has GC (garbage collector) and I believe this is coming from the latter, so not really sure how to tackle this.
Comment 4 oscar lopez arandas 2017-09-22 16:40:11 UTC
Checking more in depth the logs of the release 2.3.2, there in common three warnings that were not appearing before, are present in fedora24, fedora 25, ubuntu1604 an centos7a, the tests of oe-selftest happen.
=================================================================================
 WARNING: /home/ab/autobuilder/yocto-worker/nightly-oe-selftest-centos7a/build/bitbake/lib/bb/server/__init__.py:39: ResourceWarning: unclosed file <_io.FileIO name=33 mode='wb'>
  self._idlefuns = {}
=================================================================================
WARNING: /home/ab/autobuilder/yocto-worker/nightly-oe-selftest-centos7a/build/bitbake/lib/bb/cache.py:449: ResourceWarning: unclosed file <_io.TextIOWrapper name=20 mode='w' encoding='UTF-8'>
  value = pickled.load()
=================================================================================

WARNING: /home/ab/autobuilder/yocto-worker/nightly-oe-selftest-centos7a/build/bitbake/lib/bb/data_smart.py:1036: ResourceWarning: unclosed file <_io.TextIOWrapper name=20 mode='w' encoding='UTF-8'>
  data_str = str([(k, data[k]) for k in sorted(data.keys())])
=================================================================================
WARNING: /home/ab/autobuilder/yocto-worker/nightly-oe-selftest-fedora24/build/bitbake/lib/bb/event.py:129: ResourceWarning: unclosed file <_io.TextIOWrapper name=23 mode='w' encoding='UTF-8'>
  for name, handler in list(_handlers.items()):
==================================================================================
Comment 5 Sonia Leon 2017-10-30 17:17:26 UTC
After an update and refresh problem is not present anymore.
Comment 6 oscar lopez arandas 2017-10-31 18:19:06 UTC
i cant put in  verify why ubuntu1604 and fedora25 the warnings is present only ubuntu17.04 is not present the warnings.

BUILD: 2.4 rc3 :65d23bd7986615fdfb0f1717b615534a2a14ab80
ENVIRONMENT: Ubuntu 16.04 x86-64 and fedora25
============================================================================

WARNING: /home/ab/autobuilder/yocto-worker/nightly-oe-selftest-ubuntu1604/build/bitbake/lib/bb/codeparser.py:88: ResourceWarning: unclosed file <_io.TextIOWrapper name=47 mode='w' encoding='UTF-8'>

WARNING: /home/ab/autobuilder/yocto-worker/nightly-oe-selftest-ubuntu1604/build/bitbake/lib/bb/codeparser.py:88: ResourceWarning: unclosed file <_io.TextIOWrapper name=47 mode='w' encoding='UTF-8'>
  for i in items:
===============================================================================
full log's in attachment
Comment 7 oscar lopez arandas 2017-10-31 18:19:38 UTC
Created attachment 4110 [details]
full_log
Comment 8 Richard Purdie 2017-11-02 14:57:44 UTC
Is this bug about the warning or the hang?

We see the warnings in builds which don't hang so I believe these are two separate issues. We do see the warnings on the mail AB too.
Comment 9 oscar lopez arandas 2017-11-03 14:44:37 UTC
(In reply to comment #8)
> Is this bug about the warning or the hang?
> 
> We see the warnings in builds which don't hang so I believe these are two
> separate issues. We do see the warnings on the mail AB too.

the main error was that he hang up and sent warnings that never came out and failed. now no workers are hanging but the warnings are still present ...
Comment 10 Joshua Lock 2017-12-07 14:35:26 UTC
Last comment is that oe-selftest isn't hanging, therefore resolving the bug.
Comment 11 oscar lopez arandas 2017-12-07 15:07:08 UTC
the problem is solved or-self test isn't hanging, therefore resolving the bug.

Build Configuration:
BB_VERSION        = "1.35.0"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "ubuntu-17.04"
TARGET_SYS        = "x86_64-poky-linux"
MACHINE           = "qemux86-64"
DISTRO            = "poky"
DISTRO_VERSION    = "2.4"
TUNE_FEATURES     = "m64 core2"         
meta-yocto-bsp    = "heads/master:65d23bd7986615fdfb0f1717b615534a2a14ab80"
================================================================================
2017-10-17 21:41:35,544 - oe-selftest - INFO - RESULTS - systemd_boot.Systemdboot.test_image_efi_file - Testcase 1528: PASSED
2017-10-17 21:41:35,544 - oe-selftest - INFO - SUMMARY:
2017-10-17 21:41:35,544 - oe-selftest - INFO - oe-selftest () - Ran 316 tests in 19684.423s
2017-10-17 21:41:35,544 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (skipped=1)
program finished with exit code 0
elapsedTime=19700.524417
================================================================================
full-log in attachment
Comment 12 oscar lopez arandas 2017-12-07 15:07:59 UTC
Created attachment 4152 [details]
all_tescases_pass