Bug 15546

Summary: nettoe/autotools: devtool -f does not remove recipe from workspacelayer anymore, but FileNotFoundError
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Robert Berger <pokylinux>
Component: devtools / tool chainAssignee: Unassigned <unassigned>
Status: NEW --- QA Contact:
Severity: normal    
Priority: Medium CC: meta.mr.watcher, meta.watcher, michael.opdenacker, randy.macleod, tim.orling, yoann.congal
Version: 5.1   
Target Milestone: 6.1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know
Bug Depends on: 15545    
Bug Blocks:    
Attachments:
Description Flags
Copy of pastebin link for the paranoid. none

Description Robert Berger 2024-07-08 21:22:09 UTC
I am on poky master:
5d88faa0f35f0205c1475893d8589d1e6533dcc0

I use the extensible SDK in a container, create with devtool a recipe, create a layer and want to move the recipe from the workspace layer to my new layer.

Up to now it worked like this:

devtool finish -f nettoe /workdir/sources/meta-nettoe


...

this happens now:

https://pastebin.com/sEYywRnk
Comment 1 Robert Berger 2024-07-08 21:25:33 UTC
Please note, that the project is an autotooled project.
Comment 2 Randy MacLeod 2024-07-11 14:37:42 UTC
Copy of the pastebin log:

+ devtool finish -f nettoe /workdir/sources/meta-nettoe
NOTE: Starting bitbake server...
NOTE: Started PRServer with DBfile: /opt/ext-sdk/resy/5.0.66/multi-v7-ml/imx6q-phytec-mira-rdk-nand/cache/prserv.sqlite3, Address: 127.0.0.1:46009, PID: 3944
WARNING: Source tree is not clean, continuing as requested by -f/--force
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#1)... (21:10:54.098402)
NOTE: Started PRServer with DBfile: /opt/ext-sdk/resy/5.0.66/multi-v7-ml/imx6q-phytec-mira-rdk-nand/cache/prserv.sqlite3, Address: 127.0.0.1:35489, PID: 3983
Loading cache: 100% |##########################################################################################################################################################| Time: 0:00:01
Loaded 4889 entries from dependency cache.
Parsing recipes: 100% |########################################################################################################################################################| Time: 0:00:00
Parsing of 2955 .bb files complete (2953 cached, 2 parsed). 4890 targets, 561 skipped, 0 masked, 0 errors.
INFO: Updating SRCREV in recipe nettoe_git.bb
Traceback (most recent call last):
  File "/opt/ext-sdk/resy/5.0.66/multi-v7-ml/imx6q-phytec-mira-rdk-nand/layers/poky-training-master/scripts/esdk-tools/devtool", line 350, in <module>
    ret = main()
          ^^^^^^
  File "/opt/ext-sdk/resy/5.0.66/multi-v7-ml/imx6q-phytec-mira-rdk-nand/layers/poky-training-master/scripts/esdk-tools/devtool", line 337, in main
    ret = args.func(args, config, basepath, workspace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ext-sdk/resy/5.0.66/multi-v7-ml/imx6q-phytec-mira-rdk-nand/layers/poky-training-master/scripts/lib/devtool/standard.py", line 2177, in finish
    updated, appendfile, removed = _update_recipe(args.recipename, workspace, rd, args.mode, appendlayerdir, wildcard_version=True, no_remove=False, no_report_remove=removing_original, initial_rev=args.initial_rev, dry_run_outdir=dry_run_outdir, no_overrides=args.no_overrides, force_patch_refresh=args.force_patch_refresh)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ext-sdk/resy/5.0.66/multi-v7-ml/imx6q-phytec-mira-rdk-nand/layers/poky-training-master/scripts/lib/devtool/standard.py", line 1871, in _update_recipe
    updated, appendf, removed = _update_recipe_srcrev(recipename, workspace, srctree, crd, appendlayerdir, wildcard_version, no_remove, no_report_remove, dry_run_outdir)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ext-sdk/resy/5.0.66/multi-v7-ml/imx6q-phytec-mira-rdk-nand/layers/poky-training-master/scripts/lib/devtool/standard.py", line 1554, in _update_recipe_srcrev
    upd_f, new_f, del_f = _export_local_files(srctree, rd, local_files_dir, srctreebase)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ext-sdk/resy/5.0.66/multi-v7-ml/imx6q-phytec-mira-rdk-nand/layers/poky-training-master/scripts/lib/devtool/standard.py", line 1462, in _export_local_files
    shutil.copy2(fullfile, os.path.join(destdir, f))
  File "/opt/ext-sdk/resy/5.0.66/multi-v7-ml/imx6q-phytec-mira-rdk-nand/buildtools/sysroots/x86_64-resysdk-linux/usr/lib/python3.12/shutil.py", line 475, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/opt/ext-sdk/resy/5.0.66/multi-v7-ml/imx6q-phytec-mira-rdk-nand/buildtools/sysroots/x86_64-resysdk-linux/usr/lib/python3.12/shutil.py", line 262, in copyfile
    with open(dst, 'wb') as fdst:
         ^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/devtooln1zuo8r9/tmpj8128iec/build-aux/config.guess'
+ rm -rf /workdir/sources/meta-nettoe/recipes-example
+ rm -rf workspace/sources/nettoe
+ layers/poky-training-master/bitbake/bin/bitbake-layers show-layers
NOTE: Starting bitbake server...
NOTE: Started PRServer with DBfile: /opt/ext-sdk/resy/5.0.66/multi-v7-ml/imx6q-phytec-mira-rdk-nand/cache/prserv.sqlite3, Address: 127.0.0.1:33395, PID: 4080
Comment 3 Randy MacLeod 2024-07-11 14:38:53 UTC
Likely related to:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15545
and we should fix that one to see if this problem goes away.
Comment 4 Michael Opdenacker 2024-11-28 16:13:30 UTC
Same problem on master with nano:

$ devtool add https://nano-editor.org/dist/v5/nano-5.8.tar.xz
$ devtool build nano
$ devtool finish nano ../meta-foo

ERROR: Source tree is not clean:

 M INSTALL
 M Makefile.in
 M aclocal.m4
 M compile
 M config.guess
 M config.h.in
 M config.rpath
 M config.sub
 M configure
 M depcomp
 M doc/Makefile.in
 M doc/texinfo.tex
 M install-sh
 M lib/Makefile.in
 M m4/Makefile.in
 D m4/gettext.m4
 D m4/iconv.m4
 D m4/lib-ld.m4
 D m4/lib-link.m4
 D m4/lib-prefix.m4
 D m4/nls.m4
 D m4/po.m4
 D m4/progtest.m4
 M missing
 M po/Makefile.in.in
 M src/Makefile.in
 M syntax/Makefile.in
?? autom4te.cache/
?? config.guess~
?? config.h.in~
?? config.sub~

Ensure you have committed your changes or use -f/--force if you are sure there's nothing that needs to be committed

$ devtool finish -f nano ../meta-foo
Traceback (most recent call last):
  File "/home/mike/work/yocto/poky/scripts/devtool", line 350, in <module>
    ret = main()
          ^^^^^^
  File "/home/mike/work/yocto/poky/scripts/devtool", line 337, in main
    ret = args.func(args, config, basepath, workspace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mike/work/yocto/poky/scripts/lib/devtool/standard.py", line 2167, in finish
    updated, appendfile, removed = _update_recipe(args.recipename, workspace, rd, args.mode, appendlayerdir, wildcard_version=True, no_remove=False, no_report_remove=removing_original, initial_rev=args.initial_rev, dry_run_outdir=dry_run_outdir, no_overrides=args.no_overrides, force_patch_refresh=args.force_patch_refresh)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mike/work/yocto/poky/scripts/lib/devtool/standard.py", line 1863, in _update_recipe
    updated, appendf, removed = _update_recipe_patch(recipename, workspace, srctree, crd, appendlayerdir, wildcard_version, no_remove, no_report_remove, initial_rev, dry_run_outdir, force_patch_refresh)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mike/work/yocto/poky/scripts/lib/devtool/standard.py", line 1646, in _update_recipe_patch
    upd_f, new_f, del_f = _export_local_files(srctree, rd, local_files_dir, srctreebase)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mike/work/yocto/poky/scripts/lib/devtool/standard.py", line 1452, in _export_local_files
    shutil.copy2(fullfile, os.path.join(destdir, f))
  File "/usr/lib/python3.12/shutil.py", line 475, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.12/shutil.py", line 262, in copyfile
    with open(dst, 'wb') as fdst:
         ^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/devtool75c4i768/tmp1371ayq0/doc/Makefile.in'
Comment 5 Robert Berger 2024-12-03 17:13:24 UTC
Surprisingly enough I was able to do the full nano 5.8 stuff and devtool finish worked.

When I did the upgrade to 5.9 and tried the devtool finish there, it failed.

https://pastebin.com/1AW5Xp9f
Comment 6 Randy MacLeod 2025-05-01 14:07:05 UTC
Bulk move of all unassigned 5.2 medium importance bugs to 5.3.
Comment 7 Michael Opdenacker 2025-11-26 15:43:38 UTC
Still facing the same issue with the current openembedded-core master branch.
Comment 8 Robert Berger 2025-12-02 13:47:04 UTC
Michael has a workaround in his lectures!

something like:

pushd workspace/sources/nano/
git restore .
git clean -fdx
popd
Comment 9 Randy MacLeod 2026-04-01 18:24:06 UTC
Created attachment 5206 [details]
Copy of  pastebin link for the paranoid.