14207 2021-01-29 07:58:34 +0000 pseudo abort with wic images 2021-05-21 08:10:31 +0000 1 1 6 Yocto Project Subprojects Pseudo pseudo 3.2.1 NUC x86_64 RESOLVED NOTABUG Medium+ major 3.4 M1 1 rameshkrishnanx.geddy.sekar chee.yang.lee chee.yang.lee meta.mr.watcher meta.watcher randy.macleod richard.purdie tim.orling yp.pseudo.watcher yp.watcher --- --- Don't know oldest_to_newest 89078 0 4765 rameshkrishnanx.geddy.sekar 2021-01-29 07:58:34 +0000 Created attachment 4765 On the attached file i have both persistent.bbclass and persistent-data.wks.in contents. please add them in meta/classes and scripts/lib/wic/canned-wks/ directories We are getting a pseudo abort while building wic images using gatesgarth branch. I have attached a simpler version of wks and bbclass that we use. This issue isn't present in dunfell. Current master also has this issue. Console Error: | output: mke2fs 1.45.6 (20-Mar-2020) | Discarding device blocks: done | Creating filesystem with 157892 1k blocks and 19840 inodes | Filesystem UUID: 2b06a58d-ba0c-410f-8e0b-84d285bdfb3f | Superblock backups stored on blocks: | 8193, 24577, 40961, 57345, 73729 | | Allocating group tables: done | Writing inode tables: done | Creating journal (4096 blocks): done | Copying files into the device: abort()ing pseudo client by server request. See https://wiki.yoctoproject.org/wiki/Pseudo_Abort for more details on this. | Check logfile: /infrastructure/rgeddysx/OE-CORE/build/tmp-glibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/tmp-wic/pseudo1/pseudo.log | Aborted | | WARNING: exit code 1 from a shell command. | ERROR: Task (/infrastructure/rgeddysx/OE-CORE/oe-core/meta/recipes-core/images/core-image-minimal.bb:do_image_wic) failed with exit code '1' NOTE: Tasks Summary: Attempted 2736 tasks of which 1 didn't need to be rerun and 1 failed. Below are the steps to reproduce the issue: #git clone git://git.openembedded.org/openembedded-core-contrib -b stable/gatesgarth-next oe-core #cd oe-core #git clone git://git.openembedded.org/bitbake -b 1.48 #cd .. #mkdir build #cd build #. ../oe-core/oe-init-build-env . ## Added persistent.bbclass and "persistent-data.wks.in" in the respective dir ## Add the below lines in local.conf PACKAGE_CLASSES ?= "package_rpm" (should be rpm . by default it is ipk) IMAGE_FSTYPES = "wic wic.bmap" EFI_PREFIX = "/boot/efi" WKS_FILE = "persistent-data.wks.in" IMGCLASSES_append = " persistent" IMAGE_ROOTFS_EXCLUDE_PATH_append = " etc/" IMAGE_FEATURES += " package-management" Run the target #bitbake core-image-minimal 89079 1 rameshkrishnanx.geddy.sekar 2021-01-29 08:01:49 +0000 Changed the Product from OE-Core to Pseudo since the bug is more related to Pseudo 89113 2 chee.yang.lee 2021-02-02 00:51:54 +0000 happen on oe-core master too meta = "HEAD:3a4fed4ae0e8a0d1bd62ea5fa1ef12925e1f20f5" 89116 3 richard.purdie 2021-02-02 16:25:06 +0000 Could you share the contents of the log file as mentioned in the error: /infrastructure/rgeddysx/OE-CORE/build/tmp-glibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/tmp-wic/pseudo1/pseudo.log 89118 4 rameshkrishnanx.geddy.sekar 2021-02-02 16:50:40 +0000 Though the console says, Check logfile: /infrastructure/rgeddysx/OE-CORE/build/tmp-glibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/tmp-wic/pseudo1/pseudo.log But the same tmp-wic folder is missing in the build directory. It might have got cleanedup otherwise with do_image_wic task on failure. 89124 5 4769 chee.yang.lee 2021-02-03 02:41:13 +0000 Created attachment 4769 Pseudo log file attached pseudo log. i comment out this line so wic keeps the tmp and log files. +++ b/scripts/lib/wic/plugins/imager/direct.py @@ -275,7 +275,7 @@ class DirectPlugin(ImagerPlugin): shutil.move(path, os.path.join(self.outdir, fname)) # remove work directory - shutil.rmtree(self.workdir, ignore_errors=True) + #shutil.rmtree(self.workdir, ignore_errors=True) # Overhead of the MBR partitioning scheme (just one sector) MBR_OVERHEAD = 1 89131 6 richard.purdie 2021-02-04 00:02:14 +0000 To summarise, you're splitting out part of the rootfs into a separate data directory which is in its own wic partition. The pseudo log is helpful as it is saying: path mismatch [2 links]: ino 258881762 db '/var/lib/rpm/.dbenv.lock' req '/data/persistent/var/lib/rpm/.dbenv.lock'. i.e. that a file is in the pseudo database as being in one location but on disk its been moved. I notice this path looks the same as the symlink trick that the attached class is adding. The issue is likely somewhere in and around how pseudo is monitoring these different paths and this rpm trick not being something used elsewhere. It does sound more like a misconfiguration/misuse of pseudo rather than a bug in pseudo itself but it will need further debugging. 89180 7 chee.yang.lee 2021-02-09 09:53:25 +0000 current workaround is to switch the task (do_persistent_fixup) from IMAGE_PREPROCESS_COMMAND to "after do_rootfs before do_flush_pseudodb". IMAGE_PREPROCESS_COMMAND are part of do_image. Gatesgarth after "classes/image_types_wic: Reorder do_flush_pseudodb"[1], the pseudodb flushed before do_image. While in dunfell(3.1.5), pseudodb are flush after do_image. i think this is why it error in gatesgarth but not in dunfell. task_order on dunfell(3.1.5): do_rootfs (68770): log.do_rootfs.68770 do_image_qa (76656): log.do_image_qa.76656 do_write_qemuboot_conf (76660): log.do_write_qemuboot_conf.76660 do_image (76667): log.do_image.76667 do_write_wks_template (77027): log.do_write_wks_template.77027 do_rootfs_wicenv (77028): log.do_rootfs_wicenv.77028 do_flush_pseudodb (77029): log.do_flush_pseudodb.77029 do_image_ext4 (77032): log.do_image_ext4.77032 do_image_tar (77039): log.do_image_tar.77039 do_image_wic (77045): log.do_image_wic.77045 [1] https://git.openembedded.org/openembedded-core/commit/meta/classes?h=gatesgarth&id=445b0a9544b55735496bbb23dbff3399b3b9e9a4 90451 8 chee.yang.lee 2021-05-21 08:10:31 +0000 Close this as not a bug, not issue with pseudo tools. This is more to problem with configuration for task sequence. the activity moving files in rootfs should be done in ROOTFS command like ROOTFS_POSTPROCESS_COMMAND, ROOTFS_POSTINSTALL_COMMAND, add to do_rootfs or set the task in between do_rootfs and do_flush_pseudodb. in wks, use --rootfs-dir when the rootfs is build/prepared at other path with its own pseudo.db. use --change-directory for a partition to take a subdirectory within current rootfs (/data in this case) while keeping the correct uid/gid. it should use --change-directory in this case. part /data --source rootfs --fstype=ext4 --label data --align 4096 --use-uuid --change-directory=data --fsoptions rw,noatime 4765 2021-01-29 07:58:34 +0000 2021-01-29 07:58:34 +0000 On the attached file i have both persistent.bbclass and persistent-data.wks.in contents. please add them in meta/classes and scripts/lib/wic/canned-wks/ directories bbclass and wks file.txt text/plain 2814 rameshkrishnanx.geddy.sekar PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0NCkJCQ0xB U1MgZmlsZTogcGVyc2lzdGVudC5iYmNsYXNzDQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PQ0KZmFrZXJvb3QgcHl0aG9uIGRvX3BlcnNpc3RlbnRfZml4 dXAoKXsNCiAgICBpbXBvcnQgb3MsIHNodXRpbCwgc3RhdCwgc3VicHJvY2Vzcw0KDQogICAgcm9v dGZzID0gZC5nZXRWYXIoIklNQUdFX1JPT1RGUyIpDQogICAgZXRjID0gb3MucGF0aC5qb2luKHJv b3RmcywgImV0YyIpDQogICAgdmFyID0gb3MucGF0aC5qb2luKHJvb3RmcywgInZhciIpDQogICAg ZGF0YSA9IG9zLnBhdGguam9pbihyb290ZnMsICJkYXRhIikNCiAgICBob21lID0gb3MucGF0aC5q b2luKHJvb3RmcywgImhvbWUiKQ0KICAgIHNoYXJlID0gb3MucGF0aC5qb2luKHJvb3RmcywgInVz ci9zaGFyZSIpDQogICAgcGVyc2lzdGVudCA9IG9zLnBhdGguam9pbihkYXRhLCAicGVyc2lzdGVu dCIpDQogICAgbW9kZSA9IHN0YXQuU19JUldYVSB8IHN0YXQuU19JUkdSUCB8IHN0YXQuU19JWEdS UCB8IHN0YXQuU19JUk9USCB8IHN0YXQuU19JWE9USA0KDQogICAgIyBzYW5pdHkgY2hlY2sNCiAg ICBpZiBub3QgKHJvb3RmcyBhbmQgZXRjIGFuZCB2YXIgYW5kIGRhdGEgYW5kIGhvbWUgYW5kIHBl cnNpc3RlbnQgYW5kIHNoYXJlKToNCiAgICAgICAgYmIuZXJyb3IoIlslc10gcGF0aHMgYXJlIGJh ZFxuIiAlIGQuZ2V0VmFyKCJQTiIpKQ0KDQogICAgIyBnZW5lcmF0ZSBsZC5zby5jYWNoZQ0KICAg IHN1YnByb2Nlc3MuY2FsbChbImxkY29uZmlnIiwgIi1yIiwgcm9vdGZzLCAiLWMiLCAibmV3Iiwg Ii12IiBdKQ0KDQogICAgIyBjcmVhdGUgcGVyc2lzdGVudCBkYXRhIGFyZWENCiAgICBpZiBub3Qg b3MucGF0aC5pc2RpcihkYXRhKToNCiAgICAgICAgb3MubWtkaXIoZGF0YSwgbW9kZSkNCg0KICAg IGlmIG5vdCBvcy5wYXRoLmlzZGlyKHBlcnNpc3RlbnQpOg0KICAgICAgICBvcy5ta2RpcihwZXJz aXN0ZW50LCBtb2RlKQ0KDQogICAgIyBtb3ZlIGV2ZXJ5dGhpbmcgZWxzZSB0byBwZXJzaXN0ZW50 IC9kYXRhDQogICAgc2h1dGlsLm1vdmUoZXRjLCBwZXJzaXN0ZW50KQ0KICAgIHNodXRpbC5tb3Zl KHZhciwgcGVyc2lzdGVudCkNCiAgICBzaHV0aWwubW92ZShob21lLCBwZXJzaXN0ZW50KQ0KDQog ICAgIyBjcmVhdGUgZW1wdHkgZHVtbWllcw0KICAgIG9zLm1rZGlyKGV0YywgbW9kZSkNCiAgICBv cy5ta2Rpcih2YXIsIG1vZGUpDQogICAgb3MubWtkaXIoaG9tZSwgbW9kZSkNCg0KICAgICMgZml4 IC5tYW5pZmVzdCBnZW5lcmF0aW9uLCBhdCBsZWFzdCBmb3IgcnBtDQogICAgIyBUT0RPOiBvdGhl ciBwYWNrYWdlIG1hbmFnZXJzDQogICAgcnBtbGliID0gInZhci9saWIvcnBtIg0KICAgIHJwbV9w ZXJzaXN0ID0gb3MucGF0aC5qb2luKHBlcnNpc3RlbnQscnBtbGliKQ0KICAgIGlmIG9zLnBhdGgu aXNkaXIocnBtX3BlcnNpc3QpOg0KICAgICAgICBvcy5tYWtlZGlycyhvcy5wYXRoLmpvaW4ocm9v dGZzLCAidmFyL2xpYiIpLCBtb2RlKQ0KICAgICAgICBvcy5zeW1saW5rKCIuLi8uLi9kYXRhL3Bl cnNpc3RlbnQvdmFyL2xpYi9ycG0iLCBvcy5wYXRoLmpvaW4ocm9vdGZzLHJwbWxpYikpDQp9DQoN CklNQUdFX1BSRVBST0NFU1NfQ09NTUFORCArPSAiZG9fcGVyc2lzdGVudF9maXh1cDsiDQoNCj09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0gDQpXS1Nf RklMRSA6IHBlcnNpc3RlbnQtZGF0YS53a3MuaW4NCj09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT0NCiMgc2hvcnQtZGVzY3JpcHRpb246IENyZWF0ZSBh biBFRkkgZGlzayBpbWFnZSB3aXRoIHN5c3RlbWQtYm9vdA0KIyBsb25nLWRlc2NyaXB0aW9uOiBD cmVhdGVzIGEgcGFydGl0aW9uZWQgRUZJIGRpc2sgaW1hZ2UgdGhhdCB0aGUgdXNlcg0KIyBjYW4g ZGlyZWN0bHkgZGQgdG8gYm9vdCBtZWRpYS4gVGhlIHNlbGVjdGVkIGJvb3Rsb2FkZXIgaXMgZ3J1 Yi1lZmkuDQoNCnBhcnQgLyAtLXNvdXJjZSByb290ZnMgLS1mc3R5cGU9ZXh0NCAtLWxhYmVsIHBy aW1hcnkgLS1hbGlnbiA0MDk2IC0tdXNlLXV1aWQgLS1leGNsdWRlLXBhdGggJHtJTUFHRV9ST09U RlNfRVhDTFVERV9QQVRIfSAtLWZzb3B0aW9ucyBub2F0aW1lIC0tYWN0aXZlDQojcGFydCAtLWZz dHlwZT1leHQ0IC0tbGFiZWwgc2Vjb25kYXJ5IC0tYWxpZ24gNDA5NiAtLXVzZS11dWlkDQpwYXJ0 IC9kYXRhIC0tc291cmNlIHJvb3RmcyAtLWZzdHlwZT1leHQ0IC0tbGFiZWwgZGF0YSAtLWFsaWdu IDQwOTYgLS11c2UtdXVpZCAtLXJvb3Rmcy1kaXI9JHtJTUFHRV9ST09URlN9L2RhdGEgLS1mc29w dGlvbnMgcncsbm9hdGltZQ0KI3BhcnQgc3dhcCAtLXBhcnQtdHlwZT0iMDY1N0ZENkQtQTRBQi00 M0M0LTg0RTUtMDkzM0M4NEI0RjRGIiAtLXVzZS11dWlkIC0tYWxpZ24gNDA5NiAtLWxhYmVsIHN3 YXAgLS1mc3R5cGUgc3dhcA0KYm9vdGxvYWRlciAtLXB0YWJsZSBncHQgLS10aW1lb3V0PTUgLS1h cHBlbmQ9IiR7QVBQRU5EfSINCg0KPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PQ0K 4769 2021-02-03 02:41:13 +0000 2021-02-03 02:41:13 +0000 Pseudo log file pseudo.log application/octet-stream 589 chee.yang.lee ZGVidWdfbG9nZmlsZTogZmQgMgpwaWQgMTUwMzYgW3BhcmVudCAxNTAzNV0sIGRvaW5nIG5ldyBw aWQgc2V0dXAgYW5kIHNlcnZlciBzdGFydApTZXR1cCBjb21wbGV0ZSwgc2VuZGluZyBTSUdVU1Ix IHRvIHBpZCAxNTAzNS4KcGF0aCBtaXNtYXRjaCBbMiBsaW5rc106IGlubyAyNTg4ODE3NjIgZGIg Jy9kYXRhLzRUL29wZW5lbWJlZGRlZC1jb3JlL2J1aWxkL3RtcC1nbGliYy93b3JrL3FlbXV4ODZf NjQtb2UtbGludXgvY29yZS1pbWFnZS1taW5pbWFsLzEuMC1yMC90bXAtd2ljL3Jvb3RmczEvdmFy L2xpYi9ycG0vLmRiZW52LmxvY2snIHJlcSAnL2RhdGEvNFQvb3BlbmVtYmVkZGVkLWNvcmUvYnVp bGQvdG1wLWdsaWJjL3dvcmsvcWVtdXg4Nl82NC1vZS1saW51eC9jb3JlLWltYWdlLW1pbmltYWwv MS4wLXIwL3RtcC13aWMvcm9vdGZzMS9kYXRhL3BlcnNpc3RlbnQvdmFyL2xpYi9ycG0vLmRiZW52 LmxvY2snLgpkYiBjbGVhbnVwIGZvciBzZXJ2ZXIgc2h1dGRvd24sIDAxOjQ5OjQxLjU1NwptZW1v cnktdG8tZmlsZSBiYWNrdXAgY29tcGxldGUsIDAxOjQ5OjQxLjU1OC4KZGIgY2xlYW51cCBmaW5p c2hlZCwgMDE6NDk6NDEuNTU4Cg==