Bug 14023

Summary: oe-selftest doesn't work with BB_SERVER_TIMEOUT=60
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Richard Purdie <richard.purdie>
Component: coreAssignee: Pavel Zhukov <pavel>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: meta.mr.watcher, meta.watcher, pavel, randy.macleod
Version: 3.2   
Target Milestone: 4.0 M4   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Bug Depends on:    
Bug Blocks: 13023    

Comment 1 Randy MacLeod 2020-08-27 08:09:07 UTC
Depends on memory resident bitbake
Comment 5 Pavel Zhukov 2022-03-23 07:59:01 UTC
Progress report:

Steps to reproduce: 
1) To save time: Delete all tests but devtool.DevtoolUpgradeTests.test_devtool_upgrade and devtool.DevtoolUpgradeTests.test_devtool_upgrade_git from the devtool.DevtoolUpgradeTests testsuite. 

2) Set BB_SERVER_TIMEOUT to 60 (any value from 10 to 60 actually).

3)  rm -rf ../build-st/ && BB_SERVER_TIMEOUT=10 oe-selftest -K   --run-tests devtool.DevtoolUpgradeTests 


Some findings:
Only second test fails. For example:

2022-03-22 19:01:00,563 - oe-selftest - INFO - RESULTS - devtool.DevtoolUpgradeTests.test_devtool_upgrade: PASSED (236.70s)
2022-03-22 19:01:00,563 - oe-selftest - INFO - RESULTS - devtool.DevtoolUpgradeTests.test_devtool_upgrade_git: FAILED (8.75s)


2022-03-22 19:39:02,856 - oe-selftest - INFO - RESULTS - devtool.DevtoolUpgradeTests.test_devtool_upgrade_git: PASSED (227.88s)
2022-03-22 19:39:02,856 - oe-selftest - INFO - RESULTS - devtool.DevtoolUpgradeTests.test_devtool_xupgrade: FAILED (8.95s)  <== NOTE renamed to xupgrade to change the order of execution

I've tracked it down to this line: https://github.com/yoctoproject/poky/blob/master/scripts/lib/devtool/standard.py#L603

build_targets is called with "devtool-upgrade-test[12]" "do_patch" parameters but actual do_fetch -> do_unpack -> do_patch is not called (properly cached somewhere) down in bitbake server.

WARNING: ['buildTargets', ['devtool-upgrade-test1'], 'do_patch'] # <=             commandline debug output tinfoil.py result = self.server_connection.connection.runCommand(commandline)

NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "1.53.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "x86_64-poky-linux"
MACHINE              = "qemux86-64"
DISTRO               = "poky"
DISTRO_VERSION       = "3.4+snapshot-ae5f87700713e184fcca7d6bcdc5b86267cdce3b"
TUNE_FEATURES        = "m64 core2"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "master:ae5f87700713e184fcca7d6bcdc5b86267cdce3b"
meta-selftest        = "master:db2fb814b7f3cd5ae095d05246455df5a1b004e3"
workspace            = "<unknown>:<unknown>"

Initialising tasks...done.
Sstate summary: Wanted 0 Local 0 Mirrors 0 Missed 0 Current 20 (0% match, 100% complete)
NOTE: Executing Tasks
NOTE: Tasks Summary: Attempted 93 tasks of which 90 didn't need to be rerun and all succeeded.
ERROR: Something went wrong with source extraction - the devtool-source class was not active or did not function correctly:
[Errno 2] No such file or directory: '/mnt/builds/yocto/build-st/tmp/work/core2-64-poky-linux/devtool-upgrade-test1/1.5.3-r5/devtooltmp-ke8zieoj/initial_rev'
Comment 6 Richard Purdie 2022-03-25 15:16:53 UTC
Thanks for this, narrowing it down to a test case like this is incredibly helpful.

I did some work on trying to find the problem and initially I also thought it was some mtime cache issue. It turns out it was an inotify issue (which we use to monitor whether files have changed).

The issue is the removal of the workspace, then the addition of it back. Since we lost the inotify monitoring when the file was deleted and didn't add it back, chages would go undetected causing the bug as things didn't reparse.

I've sent out a patch which fixes this. I'm very happy to have this fixed as it should help a class of bugs where the server is slow to shutdown and is reused instead of restarted.

Whether it is the cause of all the issues in selftest I'm not sure but we can test and see.
Comment 7 Richard Purdie 2022-03-25 22:44:49 UTC
After this is applied there are still failures but fewer of them. We'll probably need to identify further smaller test cases to analyse and keep going:

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3350/steps/14/logs/stdio
Comment 8 Richard Purdie 2022-03-25 23:02:24 UTC
FWIW the next reproduder I found which fails is oe-selftest -r bblayers.BitbakeLayers.test_bitbakelayers_showrecipes devtool.DevtoolModifyTests.test_devtool_buildclean  -j 1 -K
Comment 9 Richard Purdie 2022-03-30 21:08:32 UTC
I figured out some further issues with my previous inotify fix. That brings us to:

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3377/steps/14/logs/stdio

which at least doesn't have devtool failures...
Comment 10 Richard Purdie 2022-03-31 16:59:37 UTC
With proposed fixes to sys.modules and sys.path, this was reduced to:

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3382/steps/15/logs/stdio
Comment 11 Richard Purdie 2022-03-31 20:49:13 UTC
First successful build: https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3386 :)
Comment 12 Pavel Zhukov 2022-04-06 06:48:52 UTC
from the IRC RP:  there is at least one intermittent issue which could be from memres: https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3400/steps/14/logs/stdio
Comment 13 Richard Purdie 2022-04-17 21:45:59 UTC
The PRServ issues is now in https://bugzilla.yoctoproject.org/show_bug.cgi?id=14786 and we've not seen any other issues since the inotify and sys.modules and sys.path fixes were merged. I think we can close this one, thanks!