Bug 15241 - "oe-selftest --newbuilddir" fails when BB_LOGCONFIG contains the old builddir
Summary: "oe-selftest --newbuilddir" fails when BB_LOGCONFIG contains the old builddir
Status: RESOLVED FIXED
Alias: None
Product: Functional (self) Testing
Classification: QA/Testing
Component: oe-selftest (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 5.1 M4
Assignee: Yoann Congal
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2023-10-18 22:15 UTC by Yoann Congal
Modified: 2024-09-12 22:09 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Yoann Congal 2023-10-18 22:15:04 UTC 
When BB_LOGCONFIG contains a path relative to the build directory (this is the case on the autobuilder), oe-selftest with the --newbuilddir option fails.

This fails:
 $ rm -rf /home/yocon/Documents/projets/yocto/poky/build/build-test && BB_LOGCONFIG=/home/yocon/Documents/projets/yocto/poky/build/../bitbake/contrib/autobuilderlog.json oe-selftest --newbuilddir /home/yocon/Documents/projets/yocto/poky/build/build-test -r runcmd.RunCmdTests.test_log
2023-10-19 00:01:58,629 - oe-selftest - INFO - Changing cwd to /home/yocon/Documents/projets/yocto/poky/build
2023-10-19 00:01:58,630 - oe-selftest - INFO - Adding layer libraries:
2023-10-19 00:01:58,630 - oe-selftest - INFO - 	/home/yocon/Documents/projets/yocto/poky/meta/lib
2023-10-19 00:01:58,630 - oe-selftest - INFO - 	/home/yocon/Documents/projets/yocto/poky/meta-yocto-bsp/lib
2023-10-19 00:01:58,630 - oe-selftest - INFO - 	/home/yocon/Documents/projets/yocto/poky/meta-openembedded/meta-oe/lib
2023-10-19 00:01:58,631 - oe-selftest - INFO - 	/home/yocon/Documents/projets/yocto/poky/meta-selftest/lib
2023-10-19 00:01:58,633 - oe-selftest - INFO - Checking base configuration is valid/parsable
NOTE: Starting bitbake server...
2023-10-19 00:02:03,692 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/yocon/Documents/projets/yocto/poky/build/build-test/build-st/conf/local.conf
2023-10-19 00:02:03,693 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2023-10-19 00:02:03,694 - oe-selftest - INFO - test_log (runcmd.RunCmdTests)
2023-10-19 00:02:03,708 - oe-selftest - INFO -  ... ok
2023-10-19 00:02:08,770 - oe-selftest - INFO - ----------------------------------------------------------------------
2023-10-19 00:02:08,770 - oe-selftest - INFO - Ran 1 test in 8.142s
2023-10-19 00:02:08,771 - oe-selftest - INFO - OK
Traceback (most recent call last):
  File "/home/yocon/Documents/projets/yocto/poky/scripts/oe-selftest", line 60, in <module>
    ret = main()
  File "/home/yocon/Documents/projets/yocto/poky/scripts/oe-selftest", line 47, in main
    results = args.func(logger, args)
  File "/home/yocon/Documents/projets/yocto/poky/meta/lib/oeqa/selftest/context.py", line 440, in run
    rc = self._internal_run(logger, args)
  File "/home/yocon/Documents/projets/yocto/poky/meta/lib/oeqa/selftest/context.py", line 404, in _internal_run
    configuration = self.get_configuration(args)
  File "/home/yocon/Documents/projets/yocto/poky/meta/lib/oeqa/selftest/context.py", line 374, in get_configuration
    metadata = metadata_from_bb()
  File "/home/yocon/Documents/projets/yocto/poky/meta/lib/oeqa/utils/metadata.py", line 27, in metadata_from_bb
    data_dict = get_bb_vars()
  File "/home/yocon/Documents/projets/yocto/poky/meta/lib/oeqa/utils/commands.py", line 248, in get_bb_vars
    bbenv = get_bb_env(target, postconfig=postconfig)
  File "/home/yocon/Documents/projets/yocto/poky/meta/lib/oeqa/utils/commands.py", line 244, in get_bb_env
    return bitbake("-e", postconfig=postconfig).output
  File "/home/yocon/Documents/projets/yocto/poky/meta/lib/oeqa/utils/commands.py", line 234, in bitbake
    return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options)
  File "/home/yocon/Documents/projets/yocto/poky/meta/lib/oeqa/utils/commands.py", line 212, in runCmd
    raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'bitbake  -e' returned non-zero exit status 1:
Traceback (most recent call last):
  File "/home/yocon/Documents/projets/yocto/poky/bitbake/bin/bitbake", line 36, in <module>
    sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv),
  File "/home/yocon/Documents/projets/yocto/poky/bitbake/lib/bb/main.py", line 404, in bitbake_main
    return ui_module.main(server_connection.connection, server_connection.events,
  File "/home/yocon/Documents/projets/yocto/poky/bitbake/lib/bb/ui/knotty.py", line 568, in main
    conf = bb.msg.setLoggingConfig(logconfig, logconfigfile)
  File "/home/yocon/Documents/projets/yocto/poky/bitbake/lib/bb/msg.py", line 301, in setLoggingConfig
    with open(os.path.normpath(userconfigfile), 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/yocon/Documents/projets/yocto/poky/build/build-test/bitbake/contrib/autobuilderlog.json'

This succeed:
 $ rm -rf /home/yocon/Documents/projets/yocto/poky/build/build-test && BB_LOGCONFIG=/home/yocon/Documents/projets/yocto/poky/bitbake/contrib/autobuilderlog.json oe-selftest --newbuilddir /home/yocon/Documents/projets/yocto/poky/build/build-test -r runcmd.RunCmdTests.test_log
2023-10-19 00:02:45,342 - oe-selftest - INFO - Changing cwd to /home/yocon/Documents/projets/yocto/poky/build
2023-10-19 00:02:45,343 - oe-selftest - INFO - Adding layer libraries:
2023-10-19 00:02:45,343 - oe-selftest - INFO - 	/home/yocon/Documents/projets/yocto/poky/meta/lib
2023-10-19 00:02:45,343 - oe-selftest - INFO - 	/home/yocon/Documents/projets/yocto/poky/meta-yocto-bsp/lib
2023-10-19 00:02:45,343 - oe-selftest - INFO - 	/home/yocon/Documents/projets/yocto/poky/meta-openembedded/meta-oe/lib
2023-10-19 00:02:45,344 - oe-selftest - INFO - 	/home/yocon/Documents/projets/yocto/poky/meta-selftest/lib
2023-10-19 00:02:45,346 - oe-selftest - INFO - Checking base configuration is valid/parsable
NOTE: Starting bitbake server...
2023-10-19 00:02:50,610 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/yocon/Documents/projets/yocto/poky/build/build-test/build-st/conf/local.conf
2023-10-19 00:02:50,610 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2023-10-19 00:02:50,611 - oe-selftest - INFO - test_log (runcmd.RunCmdTests)
2023-10-19 00:02:50,626 - oe-selftest - INFO -  ... ok
2023-10-19 00:02:55,693 - oe-selftest - INFO - ----------------------------------------------------------------------
2023-10-19 00:02:55,694 - oe-selftest - INFO - Ran 1 test in 8.191s
2023-10-19 00:02:55,694 - oe-selftest - INFO - OK
2023-10-19 00:03:09,852 - oe-selftest - INFO - RESULTS:
2023-10-19 00:03:09,853 - oe-selftest - INFO - RESULTS - runcmd.RunCmdTests.test_log: PASSED (0.02s)
2023-10-19 00:03:09,857 - oe-selftest - INFO - SUMMARY:
2023-10-19 00:03:09,857 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 8.193s
2023-10-19 00:03:09,857 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0)


Note the difference in BB_LOGCONFIG:
FAIL: /home/yocon/Documents/projets/yocto/poky/build/../bitbake/contrib/autobuilderlog.json
PASS: /home/yocon/Documents/projets/yocto/poky/bitbake/contrib/autobuilderlog.json

My hypothesis is that oe-selftest --newbuilddir tries to rewrite paths from the old builddir to the newly created one, and, doing so rewrite the path for the autobuilderlog.json file to a non-existing path.
One workaround in the autobuilder case: Use "readlink -f" to make the path of BB_LOGCONFIG absolute.
Comment 1 Jose Quaresma 2023-10-18 23:18:10 UTC 
I have found the same for BBLAYERS that was fixed [1] expanding any relative path that could exist.

[1] https://git.yoctoproject.org/poky/commit/?id=f5f465ff5777eb99941db3dda84e65d4699d97f7
Comment 2 Jose Quaresma 2023-10-18 23:22:59 UTC 
Maybe it would be better to also expand BB_LOGCONFIG when the newbuilddir argument is present.
Comment 3 Yoann Congal 2023-10-19 14:40:36 UTC 
NB: https://git.yoctoproject.org/yocto-autobuilder-helper/commit/?id=6f48f65c682c2ec076f926891a98913fb09647d4 does contain a workaround this bug (but should not be fully reversed, I did put 2 things in my patch -_-')
=> Once this bug is fixed, the workaround should be removed
Comment 4 Yoann Congal 2023-12-19 07:00:58 UTC 
Bumping target milestone to 5.0 M2
Comment 5 Yoann Congal 2024-01-23 14:39:29 UTC 
Bulk move to Milestone 5.0 M3
Comment 6 Yoann Congal 2024-02-22 16:23:32 UTC 
Bulk move to 5.0 M4
Comment 7 Yoann Congal 2024-09-10 20:51:17 UTC 
Patch sent : https://lists.openembedded.org/g/openembedded-core/message/204370
Comment 9 Yoann Congal 2024-09-12 17:02:54 UTC 
(In reply to Yoann Congal from comment #3)
> NB:
> https://git.yoctoproject.org/yocto-autobuilder-helper/commit/
> ?id=6f48f65c682c2ec076f926891a98913fb09647d4 does contain a workaround this
> bug (but should not be fully reversed, I did put 2 things in my patch -_-')
> => Once this bug is fixed, the workaround should be removed

Workaround removal sent: https://lists.yoctoproject.org/g/yocto-patches/message/650