Bug 10815

To sum up: wic doesn't seem to be obeying ROOTFS_SIZE/IMAGE_ROOTFS_EXTRA_SPACE when the rootfs partition command in the ks file doesn't pass --size=.

Correction, the analysis of the cause was incorrect. It's when --rootfs-dir= isn't passed on the partition command of the ks file, the rootfs dir passed to wic itself isn't relevant.

The logic which falls back to ROOTFS_SIZE, due to where it's done, seems to go based on rootfs_dir as that's the only way for it to know if the partition in question is a rootfs, but it'd be better to just do the fallback in prepare_rootfs(), as we know that's done for a rootfs, and ensures that the source prep method is run after part.size is set. Testing that now.

Confirmed. This patch changes the behavior so it obeys the overhead factor and extra space as defined in the metadata when there's no --size= or --rootfs-dir= in the ks:

diff --git i/scripts/lib/wic/partition.py w/scripts/lib/wic/partition.py
index f383533..55ec4c0 100644
--- i/scripts/lib/wic/partition.py
+++ w/scripts/lib/wic/partition.py
@@ -28,7 +28,7 @@ import os
 import tempfile
                                                                                          
 from wic.utils.oe.misc import msger, parse_sourceparams
-from wic.utils.oe.misc import exec_cmd, exec_native_cmd
+from wic.utils.oe.misc import exec_cmd, exec_native_cmd, get_bitbake_var
 from wic.plugin import pluginmgr

 partition_methods = {
@@ -190,6 +190,16 @@ class Partition():
         if os.path.isfile(rootfs):
             os.remove(rootfs)

+        # Get rootfs size from bitbake variable if it's not set in .ks file
+        if not self.size:
+            # Bitbake variable ROOTFS_SIZE is calculated in
+            # Image._get_rootfs_size method from meta/lib/oe/image.py
+            # using IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT,
+            # IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE
+            rsize_bb = get_bitbake_var('ROOTFS_SIZE')
+            if rsize_bb:
+                self.size = int(round(float(rsize_bb)))
+
         for prefix in ("ext", "btrfs", "vfat", "squashfs"):
             if self.fstype.startswith(prefix):
                 method = getattr(self, "prepare_rootfs_" + prefix)

Log fragment from before the change (note the 8192, that's from --extra-size 8M in the ks):

Debug: Requested partition size for /: 0
Debug: current_blocks: 558876
Debug: Added 8192 extra blocks to / to get to 567068 total blocks

Log fragment from after the change:

Debug: Requested partition size for /: 770048
Debug: current_blocks: 558876
Debug: Requested blocks 770048, current_blocks 558876
Debug: Added 211172 extra blocks to / to get to 770048 total blocks

The patch looks ok to me. Can you send it for review to the mailing list?

assigning to Christopher as he did all work.

Please reassign back to me if anything is still required from me.

One downside to that particular patch is, the overhead-factor as specified in the ks applies in addition to any defined overhead factor in the metadata in this case. Thoughts on that?

Adding Ed to CC to get his thoughts on comment #6 :)

I think it's enough to warn user if overhead-factor is specified in .wks file , but --size is not and we're using bitbake variables to determine rootfs size.

Does it make sense?

Patch submitted to oe-core.

My fix for the bug 10816 uncovered test regression brought by your changes. Here is a full test log:

$ oe-selftest -r wic
2016-12-17 13:50:16,979 - selftest - INFO - Running bitbake -e to get BBPATH
2016-12-17 13:50:19,120 - selftest - INFO - Checking that everything is in order before running the tests
2016-12-17 13:50:21,269 - selftest - INFO - Running bitbake -p
2016-12-17 13:50:24,173 - selftest - INFO - test runner init'ed like unittest
2016-12-17 13:50:26,174 - selftest - INFO - Loading tests from: oeqa.selftest.wic
2016-12-17 13:51:28,596 - selftest - INFO - Adding: "include selftest.inc" in local.conf
2016-12-17 13:51:28,596 - selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2016-12-17 13:51:28 - test_bmap (oeqa.selftest.wic.Wic)
Test generation of .bmap file ... ok
2016-12-17 13:51:52 - test_build_artifacts (oeqa.selftest.wic.Wic)
Test wic create directdisk providing all artifacts. ... ok
2016-12-17 13:52:16 - test_build_image_name (oeqa.selftest.wic.Wic)
Test wic create directdisk --image-name core-image-minimal ... ok
2016-12-17 13:52:22 - test_compress_bzip2 (oeqa.selftest.wic.Wic)
Test compressing an image with bzip2 ... ok
2016-12-17 13:52:30 - test_compress_gzip (oeqa.selftest.wic.Wic)
Test compressing an image with gzip ... ok
2016-12-17 13:52:37 - test_compress_xz (oeqa.selftest.wic.Wic)
Test compressing an image with xz ... ok
2016-12-17 13:52:49 - test_createhelp (oeqa.selftest.wic.Wic)
Test wic create --help ... ok
2016-12-17 13:52:49 - test_directdisk_bootloader_config (oeqa.selftest.wic.Wic)
Test creation of directdisk-bootloader-config image ... ok
2016-12-17 13:52:55 - test_gpt_image (oeqa.selftest.wic.Wic)
Test creation of core-image-minimal with gpt table and UUID boot ... ok
2016-12-17 13:53:03 - test_help (oeqa.selftest.wic.Wic)
Test wic --help ... ok
2016-12-17 13:53:03 - test_help_kickstart (oeqa.selftest.wic.Wic)
Test wic help kickstart ... ok
2016-12-17 13:53:03 - test_help_overview (oeqa.selftest.wic.Wic)
Test wic help overview ... ok
2016-12-17 13:53:04 - test_help_plugins (oeqa.selftest.wic.Wic)
Test wic help plugins ... ok
2016-12-17 13:53:06 - test_image_env (oeqa.selftest.wic.Wic)
Test generation of <image>.env files. ... ok
2016-12-17 13:53:30 - test_iso_image (oeqa.selftest.wic.Wic)
Test creation of hybrid iso image with legacy and EFI boot ... ok
2016-12-17 13:53:42 - test_listhelp (oeqa.selftest.wic.Wic)
Test wic list --help ... ok
2016-12-17 13:53:42 - test_mkefidisk (oeqa.selftest.wic.Wic)
Test creation of mkefidisk image ... ok
2016-12-17 13:53:49 - test_mkgummidisk (oeqa.selftest.wic.Wic)
Test creation of mkgummidisk image ... ok
2016-12-17 13:53:55 - test_no_command (oeqa.selftest.wic.Wic)
Test wic without command ... ok
2016-12-17 13:53:55 - test_qemu (oeqa.selftest.wic.Wic)
Test wic-image-minimal under qemu ... Loading cache:   0% |                                                                                                                          Loading cache: 100% |#################################################################################################################################################| Time: 0:00:00
NOTE: rootfs file: /home/ed/git/yocto/poky/build/tmp/deploy/images/qemux86-64/wic-image-minimal-qemux86-64.wic
NOTE: Qemu log file: /home/ed/git/yocto/poky/build/tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/testimage/qemu_boot_log.20161217115406
NOTE: SSH log file: /home/ed/git/yocto/poky/build/tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/testimage/ssh_target_log.20161217115406
Created listening socket for qemu serial console on: 127.0.0.1:60614
Created listening socket for qemu serial console on: 127.0.0.1:44022
Not using kvm for runqemu
launchcmd=runqemu snapshot tcpserial=44022 qemux86-64 /home/ed/git/yocto/poky/build/tmp/deploy/images/qemux86-64/wic-image-minimal-qemux86-64.wic nographic bootparams="console=tty1 console=ttyS0,115200n8 printk.time=1" qemuparams="-serial tcp:127.0.0.1:60614"
runqemu started, pid is 3379
waiting at most 60 seconds for qemu pid
qemu started - qemu procces pid is 3391
Starting logging thread
Starting thread event loop
Connection request received
Setting connection established event
Output from runqemu:
runqemu - INFO - MACHINE: qemux86-64
runqemu - INFO - DEPLOY_DIR_IMAGE: /home/ed/git/yocto/poky/build/tmp/deploy/images/qemux86-64
runqemu - INFO - CONFFILE: /home/ed/git/yocto/poky/build/tmp/deploy/images/qemux86-64/wic-image-minimal-qemux86-64.qemuboot.conf
runqemu - INFO - Continuing with the following parameters:

stty: standard input: Inappropriate ioctl for device
runqemu - INFO - Running /bin/ip link...
runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap0.lock...
runqemu - INFO - Using preconfigured tap device tap0
runqemu - INFO - If this is not intended, touch /tmp/qemu-tap-locks/tap0.skip to make runqemu skip tap0.
runqemu - WARNING - Can't detect drive type /home/ed/git/yocto/poky/build/tmp/deploy/images/qemux86-64/wic-image-minimal-qemux86-64.wic
runqemu - WARNING - Tring to use virtio block drive
runqemu - INFO - Running ldd /home/ed/git/yocto/poky/build/tmp/sysroots/x86_64-linux/usr/bin/qemu-system-x86_64...
runqemu - INFO - Running /home/ed/git/yocto/poky/build/tmp/sysroots/x86_64-linux/usr/bin/qemu-system-x86_64 -device virtio-net-pci,netdev=net0,mac=52:54:00:12:34:02 -netdev tap,id=net0,ifname=tap0,script=no,downscript=no  -nographic -serial tcp:127.0.0.1:60614  -cpu core2duo -m 256 -serial tcp:127.0.0.1:44022 -drive if=virtio,file=/home/ed/git/yocto/poky/build/tmp/deploy/images/qemux86-64/wic-image-minimal-qemux86-64.wic,format=raw -no-reboot -vga vmware -show-cursor -usb -usbdevice tablet -device virtio-rng-pci -snapshot 
QEMU 2.7.0 monitor - type 'help' for more information
(qemu) warning: TCG doesn't support requested feature: CPUID.01H:EDX.vme [bit 1]

Waiting at most 1000 seconds for login banner
Connection from 127.0.0.1:44079
Reached login banner
Logged as root in serial console
Stopping logging thread
Stop event received
Tearing down logging thread
Sending SIGTERM to runqemu
ok
2016-12-17 13:54:27 - test_qemux86_directdisk (oeqa.selftest.wic.Wic)
Test creation of qemux-86-directdisk image ... ok
2016-12-17 13:54:33 - test_rootfs_artifacts (oeqa.selftest.wic.Wic)
Test usage of rootfs plugin with rootfs paths ... FAIL
2016-12-17 13:55:00 - test_rootfs_indirect_recipes (oeqa.selftest.wic.Wic)
Test usage of rootfs plugin with rootfs recipes ... FAIL
2016-12-17 13:55:10 - test_sdimage_bootpart (oeqa.selftest.wic.Wic)
Test creation of sdimage-bootpart image ... ok
2016-12-17 13:55:15 - test_systemd_bootdisk (oeqa.selftest.wic.Wic)
Test creation of systemd-bootdisk image ... ok
2016-12-17 13:55:22 - test_unsupported_subcommand (oeqa.selftest.wic.Wic)
Test unsupported subcommand ... ok
2016-12-17 13:55:23 - test_wic_image_type (oeqa.selftest.wic.Wic)
Test building wic images by bitbake ... ok
2016-12-17 13:55:37 - test_wrong_compressor (oeqa.selftest.wic.Wic)
Test how wic breaks if wrong compressor is provided ... ok

======================================================================
FAIL: test_rootfs_artifacts (oeqa.selftest.wic.Wic)
Test usage of rootfs plugin with rootfs paths
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ed/git/yocto/poky/meta/lib/oeqa/utils/decorators.py", line 109, in wrapped_f
    return func(*args, **kwargs)
  File "/home/ed/git/yocto/poky/meta/lib/oeqa/selftest/wic.py", line 186, in test_rootfs_artifacts
    % bbvars).status
  File "/home/ed/git/yocto/poky/meta/lib/oeqa/utils/commands.py", line 114, in runCmd
    raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, result.output))
AssertionError: Command 'wic create directdisk-multi-rootfs -b /home/ed/git/yocto/poky/build/tmp/sysroots/qemux86-64/usr/share -k /home/ed/git/yocto/poky/build/tmp/deploy/images/qemux86-64 -n /home/ed/git/yocto/poky/build/tmp/sysroots/x86_64-linux --rootfs-dir rootfs1=/home/ed/git/yocto/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs --rootfs-dir rootfs2=/home/ed/git/yocto/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs' returned non-zero exit status 1:
Error: _exec_cmd: bitbake -e rootfs1 returned '1' instead of 0
output: Loading cache...done.
ERROR: Nothing PROVIDES 'rootfs1'
Loaded 1319 entries from dependency cache.

Summary: There was 1 ERROR message shown, returning a non-zero exit code.
Creating image(s)...

======================================================================
FAIL: test_rootfs_indirect_recipes (oeqa.selftest.wic.Wic)
Test usage of rootfs plugin with rootfs recipes
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ed/git/yocto/poky/meta/lib/oeqa/utils/decorators.py", line 109, in wrapped_f
    return func(*args, **kwargs)
  File "/home/ed/git/yocto/poky/meta/lib/oeqa/selftest/wic.py", line 170, in test_rootfs_indirect_recipes
    % wks).status)
  File "/home/ed/git/yocto/poky/meta/lib/oeqa/utils/commands.py", line 114, in runCmd
    raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, result.output))
AssertionError: Command 'wic create directdisk-multi-rootfs --image-name core-image-minimal --rootfs rootfs1=core-image-minimal --rootfs rootfs2=core-image-minimal' returned non-zero exit status 1:
Error: _exec_cmd: bitbake -e rootfs1 returned '1' instead of 0
output: Loading cache...done.
ERROR: Nothing PROVIDES 'rootfs1'
Loaded 1319 entries from dependency cache.

Summary: There was 1 ERROR message shown, returning a non-zero exit code.
Checking basic build environment...
Done.

Creating image(s)...

----------------------------------------------------------------------
Ran 28 tests in 249.657s

FAILED (failures=2)
2016-12-17 13:55:38,254 - selftest - INFO - Finished
2016-12-17 13:55:38,255 - selftest - INFO - Removing the include from local.conf
2016-12-17 13:55:38,256 - selftest - INFO - Removing the include from bblayers.conf



Wic seems to break on this line of your code: rsize_bb = get_bitbake_var('ROOTFS_SIZE', image_name)

You can reproduce it by using directdisk-multi-rootfs:

wic create directdisk-multi-rootfs --image-name core-image-minimal --rootfs rootfs1=core-image-minimal --rootfs rootfs2=core-image-minimal

$ wic create directdisk-multi-rootfs --image-name core-image-minimal --rootfs rootfs1=core-image-minimal --rootfs rootfs2=core-image-minimal
Checking basic build environment...
Done.

>>> bitbake -e core-image-minimal
Creating image(s)...

>>> bitbake -e rootfs1
Error: _exec_cmd: bitbake -e rootfs1 returned '1' instead of 0
output: Loading cache...done.
ERROR: Nothing PROVIDES 'rootfs1'
Loaded 1319 entries from dependency cache.

Summary: There was 1 ERROR message shown, returning a non-zero exit code.

My guess would be it's partly due to the fact that we aren't passing the image name to get_bitbake_var, but of course ROOTFS_SIZE is emitted into the wicenvs by the bbclass, it isn't a normal bitbake variable, so I'd expect it to fail in cases where it's run externally without wicenvs written, unless we adjust both parts of the code using ROOTFS_SIZE to use the actual bitbake vars directly and then also pass in the image name. Thoughts?

The problem here as I can see is that rootfs1/rootfs2 is used as an image name. In multi-rootfs mode it's not the case. wic is called with --rootfs rootfs1=core-image-minimal --rootfs2=core-image-minimal in this mode. This is not handled correctly in the code.

I'll take it as it blocks my bug.

Sorry about my inability to follow up on the regression, I've been fighting fires 99% of the time at work recently, not much time to dive into anything else.

No problem.

The issue should be fixed by this patch: http://lists.openembedded.org/pipermail/openembedded-core/2017-January/130950.html

The patch has been merged to master.