| Summary: | oe-selftest doesn't work with BB_SERVER_TIMEOUT=60 | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] OE-Core | Reporter: | Richard Purdie <richard.purdie> |
| Component: | core | Assignee: | Pavel Zhukov <pavel> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium+ | CC: | meta.mr.watcher, meta.watcher, pavel, randy.macleod |
| Version: | 3.2 | ||
| Target Milestone: | 4.0 M4 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
| Bug Depends on: | |||
| Bug Blocks: | 13023 | ||
|
Description
Richard Purdie
2020-08-25 15:17:44 UTC
Depends on memory resident bitbake https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3215 https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3178 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3221 Re-run and updated failures from master: https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3240 https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3199 https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/3211 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3242 Progress report: Steps to reproduce: 1) To save time: Delete all tests but devtool.DevtoolUpgradeTests.test_devtool_upgrade and devtool.DevtoolUpgradeTests.test_devtool_upgrade_git from the devtool.DevtoolUpgradeTests testsuite. 2) Set BB_SERVER_TIMEOUT to 60 (any value from 10 to 60 actually). 3) rm -rf ../build-st/ && BB_SERVER_TIMEOUT=10 oe-selftest -K --run-tests devtool.DevtoolUpgradeTests Some findings: Only second test fails. For example: 2022-03-22 19:01:00,563 - oe-selftest - INFO - RESULTS - devtool.DevtoolUpgradeTests.test_devtool_upgrade: PASSED (236.70s) 2022-03-22 19:01:00,563 - oe-selftest - INFO - RESULTS - devtool.DevtoolUpgradeTests.test_devtool_upgrade_git: FAILED (8.75s) 2022-03-22 19:39:02,856 - oe-selftest - INFO - RESULTS - devtool.DevtoolUpgradeTests.test_devtool_upgrade_git: PASSED (227.88s) 2022-03-22 19:39:02,856 - oe-selftest - INFO - RESULTS - devtool.DevtoolUpgradeTests.test_devtool_xupgrade: FAILED (8.95s) <== NOTE renamed to xupgrade to change the order of execution I've tracked it down to this line: https://github.com/yoctoproject/poky/blob/master/scripts/lib/devtool/standard.py#L603 build_targets is called with "devtool-upgrade-test[12]" "do_patch" parameters but actual do_fetch -> do_unpack -> do_patch is not called (properly cached somewhere) down in bitbake server. WARNING: ['buildTargets', ['devtool-upgrade-test1'], 'do_patch'] # <= commandline debug output tinfoil.py result = self.server_connection.connection.runCommand(commandline) NOTE: Resolving any missing task queue dependencies Build Configuration: BB_VERSION = "1.53.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "universal" TARGET_SYS = "x86_64-poky-linux" MACHINE = "qemux86-64" DISTRO = "poky" DISTRO_VERSION = "3.4+snapshot-ae5f87700713e184fcca7d6bcdc5b86267cdce3b" TUNE_FEATURES = "m64 core2" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "master:ae5f87700713e184fcca7d6bcdc5b86267cdce3b" meta-selftest = "master:db2fb814b7f3cd5ae095d05246455df5a1b004e3" workspace = "<unknown>:<unknown>" Initialising tasks...done. Sstate summary: Wanted 0 Local 0 Mirrors 0 Missed 0 Current 20 (0% match, 100% complete) NOTE: Executing Tasks NOTE: Tasks Summary: Attempted 93 tasks of which 90 didn't need to be rerun and all succeeded. ERROR: Something went wrong with source extraction - the devtool-source class was not active or did not function correctly: [Errno 2] No such file or directory: '/mnt/builds/yocto/build-st/tmp/work/core2-64-poky-linux/devtool-upgrade-test1/1.5.3-r5/devtooltmp-ke8zieoj/initial_rev' Thanks for this, narrowing it down to a test case like this is incredibly helpful. I did some work on trying to find the problem and initially I also thought it was some mtime cache issue. It turns out it was an inotify issue (which we use to monitor whether files have changed). The issue is the removal of the workspace, then the addition of it back. Since we lost the inotify monitoring when the file was deleted and didn't add it back, chages would go undetected causing the bug as things didn't reparse. I've sent out a patch which fixes this. I'm very happy to have this fixed as it should help a class of bugs where the server is slow to shutdown and is reused instead of restarted. Whether it is the cause of all the issues in selftest I'm not sure but we can test and see. After this is applied there are still failures but fewer of them. We'll probably need to identify further smaller test cases to analyse and keep going: https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3350/steps/14/logs/stdio FWIW the next reproduder I found which fails is oe-selftest -r bblayers.BitbakeLayers.test_bitbakelayers_showrecipes devtool.DevtoolModifyTests.test_devtool_buildclean -j 1 -K I figured out some further issues with my previous inotify fix. That brings us to: https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3377/steps/14/logs/stdio which at least doesn't have devtool failures... With proposed fixes to sys.modules and sys.path, this was reduced to: https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3382/steps/15/logs/stdio First successful build: https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3386 :) from the IRC RP: there is at least one intermittent issue which could be from memres: https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3400/steps/14/logs/stdio The PRServ issues is now in https://bugzilla.yoctoproject.org/show_bug.cgi?id=14786 and we've not seen any other issues since the inotify and sys.modules and sys.path fixes were merged. I think we can close this one, thanks! |