| Summary: | Enhancement proposal for wic functionality | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] OE-Core | Reporter: | Igor Stoppa <igor.stoppa> |
| Component: | devtools / tool chain | Assignee: | Ed Bartosh <eduard.bartosh> |
| Status: | VERIFIED FIXED | QA Contact: | Daniel Istrate <daniel.alexandrux.istrate> |
| Severity: | normal | ||
| Priority: | Medium | CC: | alexander.kanevskiy, bogdanx.a.voiculescu, eduard.bartosh, meta.mr.watcher, meta.watcher |
| Version: | unspecified | ||
| Target Milestone: | 1.9 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | Yes (doc changes required) | |
|
Description
Igor Stoppa
2015-06-08 08:34:46 UTC
(In reply to comment #0) > I need to create an image with multiple partitions, with several constraints: > > 0) the image must support UEFI mode, so I need to have an UEFI partition, > although this is not mandatory and also the MBR mode might be required > This should be supported already. See scripts/lib/image/canned-wks/mkefidisk.wks for details. > 1) at least 2 partitions are rootfs. > This should be supported already. See http://lists.openembedded.org/pipermail/openembedded-core/2014-March/091185.html for details. > 2) depending on the boot mode selected @ 0), there will be a need to install > the bootloader to the MBR This is default mode for syslinux, so should be supported already. > 3) syslinux must be supported as bootloader It is supported. syslinux is a default bootloader for wic. > 4) each of the two partitions will have its own specific recipe > I am not sure if this affects wic directly, but it should be considered when > calling it I think wic is not affected here. Bitbake will produce 2 rootfs if called with 2 recipes. > 5) similarly to point 4), this might not be a requirement for wic, but in > case multiple architectures are specified, each must produce an image I'd suggest to move this into separate bug as it's out of wic scope. > 6) I would like to provide to wic just the root directory for each partition > that needs to be populated (the boot partition and the 2 root partitions) > and let it create the partitions and image This is not supported by wic as far as I know. However, it should not be a show stopper as wic can use images produced by bitbake to populate partitions. Considering 7 it's not a big overhead. > 7) I need to be able to optionally specify that the individual partitions > should be made available also as individual files This is supported by bitbake. > 8) I would prefer to avoid giving command line parameters to wic and rely > solely on the .wks file Makes sense. So far I see only one feature to implement: - it should be possible to provide image or directory as a rootfs in wks file. Currently only image is supported. Can you share your recipes and wks files please? The best way to confirm that all above features work is to try to create an image. (In reply to comment #1) > > 3) syslinux must be supported as bootloader > It is supported. syslinux is a default bootloader for wic. I need to check if it will interfere with the syslinux.cfg file I'm providing as part of the image content. [...] > > 6) I would like to provide to wic just the root directory for each partition > > that needs to be populated (the boot partition and the 2 root partitions) > > and let it create the partitions and image > This is not supported by wic as far as I know. However, it should not be a > show stopper as wic can use images produced by bitbake to populate > partitions. Considering 7 it's not a big overhead. No, but I'd prefer if it was syslinux to deal with the creation of the partitions. From my perspective, it doesn't matter too much if I have to do it manually in my recipe or if I can let wic do it for me. But, overall, I think it's really bad that every individual in need of creating a certain filesystem has to roll own solution, rather than relying on a standard tool (wic). It also would lower the bar for those who approach Yocto for the first time, as they would have less to learn about creating own partitions and could rely on a simplified mechanism. However it's not a show stopper for me. [...] > Can you share your recipes and wks files please? The best way to confirm > that all above features work is to try to create an image. As soon as I get legal clearance, I'll share them here. (In reply to comment #2) > (In reply to comment #1) > > > > 3) syslinux must be supported as bootloader > > It is supported. syslinux is a default bootloader for wic. > > I need to check if it will interfere with the syslinux.cfg file I'm > providing as part of the image content. > One more thing to check: wic should be able to install syslinux on ext partitions. So far it was not possible due to the fact that syslinux required root privileges to do that. Now it should be fixed in syslinux, but wic/bitbake should be modified to use this functionality. > [...] > > > > 6) I would like to provide to wic just the root directory for each partition > > > that needs to be populated (the boot partition and the 2 root partitions) > > > and let it create the partitions and image > > This is not supported by wic as far as I know. However, it should not be a > > show stopper as wic can use images produced by bitbake to populate > > partitions. Considering 7 it's not a big overhead. > > No, but I'd prefer if it was syslinux to deal with the creation of the > partitions. From my perspective, it doesn't matter too much if I have to do > it manually in my recipe or if I can let wic do it for me. > > But, overall, I think it's really bad that every individual in need of > creating a certain filesystem has to roll own solution, rather than relying > on a standard tool (wic). > > It also would lower the bar for those who approach Yocto for the first time, > as they would have less to learn about creating own partitions and could > rely on a simplified mechanism. Sure, I'll implement that in wic. I mentioned this as the only feature to implement so far. I'll create a bug for it. Overall note on this bug - it's too generic and confusing to me. I don't know what exactly should I do with it. So far I guessed 2 things to do: - implement providing directory as a rootfs in wks file for every partition. - implement installation of syslinux to ext partitions I'm going to create 2 separate bugs for above issues. Please, confirm this or point out what I've missed. (In reply to comment #3) > One more thing to check: wic should be able to install syslinux on ext > partitions. So far it was not possible due to the fact that syslinux > required root privileges to do that. Now it should be fixed in syslinux, but > wic/bitbake should be modified to use this functionality. ok, atm I need EFI, though but that's a separate struggle I'm having with the syslinux recipe, which doesn't seem to support EFI > > > > 6) I would like to provide to wic just the root directory for each partition > > > > that needs to be populated (the boot partition and the 2 root partitions) > > > > and let it create the partitions and image > > > This is not supported by wic as far as I know. However, it should not be a > > > show stopper as wic can use images produced by bitbake to populate > > > partitions. Considering 7 it's not a big overhead. > > > > No, but I'd prefer if it was syslinux to deal with the creation of the > > partitions. From my perspective, it doesn't matter too much if I have to do > > it manually in my recipe or if I can let wic do it for me. > > > > But, overall, I think it's really bad that every individual in need of > > creating a certain filesystem has to roll own solution, rather than relying > > on a standard tool (wic). > > > > It also would lower the bar for those who approach Yocto for the first time, > > as they would have less to learn about creating own partitions and could > > rely on a simplified mechanism. > > Sure, I'll implement that in wic. I mentioned this as the only feature to > implement so far. I'll create a bug for it. It turned out there's a catch. I actually had a warning about it from Richard, but didn't fully understand it till few days ago. The catch is that file ownership and permissions tracked by bitbake are different from what is known to the host filesystem. I saw every sort of error from systemd when I tried to boot an image that I had created without pseudo and outside of the context where pseudo can access the database with the fake_but_intended ownership and permissions. That was happening because most of the files in the root partition were not owned by root, but by user 1003, which is my own user on the machine where I created the partition. So the only way for wic to pull this off are: a) accessing the database (but there might be several, one for each rootfs created) b) creating some sort of container that already has the right information about ownership and permissions B matches pretty well with the definition of filesystem partition in a loop file, so I ended up creating the partitions in the recipe. I still dislike it, but it's a side effect of how bitbake works and there is no simple way to change it, apart from using from the beginning a separate filesystem with the correct ownership and partitions, but this requires root privileges and goes against the philosophy of bitbake :-/ To recap, I don't think adding that feature I described is of any use, given the constraints it would have to deal with. (In reply to comment #4) > Overall note on this bug - it's too generic and confusing to me. I don't > know what exactly should I do with it. > > So far I guessed 2 things to do: > - implement providing directory as a rootfs in wks file for every partition. It won't work for the reason I explained in my reply to comment #3. Unless you have some better idea. > - implement installation of syslinux to ext partitions Atm I don't need it but it might be useful to someone else or to me later on. > I'm going to create 2 separate bugs for above issues. > > Please, confirm this or point out what I've missed. (In reply to comment #6) > (In reply to comment #4) > > Overall note on this bug - it's too generic and confusing to me. I don't > > know what exactly should I do with it. Feel free to close it. > > So far I guessed 2 things to do: > > - implement providing directory as a rootfs in wks file for every partition. > > It won't work for the reason I explained in my reply to comment #3. > Unless you have some better idea. I have a proposal: currently there is a way that I find a bit messy for creating the file systems: one has to play with IMAGE_CMD_xxx which can be either a method or a single command implemented as string. You could replace that with a single call to wic, to create the _partition_ and filesystem one wants. I would have found it much more friendly than having to read and understand various classes involved in the creation of an image. For example I had to create my own method, as I wanted to specify the size of the partition, regardless from the size of the content, which doesn't seem to be supported. So there still would be a use for wic doing the partitions in a uniform way, I think. But it could be addressed as separate issue in a new bug. So wic would be called once for every partition that is needed, plus once to put them all together. > > - implement installation of syslinux to ext partitions > > Atm I don't need it but it might be useful to someone else or to me later on. > > > I'm going to create 2 separate bugs for above issues. > > > > Please, confirm this or point out what I've missed. (In reply to comment #6) > (In reply to comment #4) > > Overall note on this bug - it's too generic and confusing to me. I don't > > know what exactly should I do with it. > > > > So far I guessed 2 things to do: > > - implement providing directory as a rootfs in wks file for every partition. > > It won't work for the reason I explained in my reply to comment #3. > Unless you have some better idea. > Did you try to use approach from the link in my first comment? It looks like they managed to create image using 2 different rootfs. Here is an example from that mail: wic create directdisk-multi-rootfs-indirect.wks \ -e core-image-minimal \ --rootfs-dir rootfs1=/srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal/1.0-r0/rootfs \ --rootfs-dir rootfs2=/srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal-dev/1.0-r0/rootfs \ --rootfs-dir rootfs3=/tmp/fakerootfs /srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal/1.0-r0/rootfs and /srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal-dev/1.0-r0/rootfs are obvioulsly two different rootfs directories. > > - implement installation of syslinux to ext partitions > > Atm I don't need it but it might be useful to someone else or to me later on. > OK. So, should I close this bug then? Nothing seems to be required from wic, right? We agreed with Igor that the following functionality should work to sutisfy his requirements: 1. bitbake produces rootfs-es for every partition: bitbake boot root rescue 2. Wic uses rootfs-es and below .wks file to convert partitions to filesystems and make partitioned image out of them: image.wks: part boot --source rootfs --rootfs-dir='boot' --ondisk sda --fstype=fat32 --align 1024 --uuid <uuid> --no-fstab --type EFI --keep part root --source rootfs --rootfs-dir='root' --ondisk sda --fstype=ext4 --align 1024 --uuid <uuid> --no-fstab part rescue --source rootfs --rootfs-dir='rescue' --ondisk sda --fstype=ext4 --align 1024 --uuid <uuid> --no-fstab wic create image.wks -e boot -e root -e rescue This requires the following fixes and new functionality for wic: - fix rootfs plugin to use correct pseudo database to keep original permissions and ownership for filesystem content - add --no-fstab parameter to wks syntax to skip updating fstab - add --uuid parameter to set partition UUID - add --type paramter to set parition type - make sure bootloader configuration is not done by wic With a bit different syntax for ks file and different command line options I managed to create image with correct ownership without changing a code. Here is how: # build both image recipes $ bitbake core-image-minimal core-image-minimal-dev # show kickstart file $ cat ../scripts/lib/image/canned-wks/directdisk-indirect-recipes.wks part first --source rootfs --rootfs-dir=rootfs1 --ondisk sda --fstype=ext3 --label platform --align 1024 part second --source rootfs --rootfs-dir=rootfs2 --ondisk sda --fstype=ext3 --label secondary --align 1024 # run wic with it $ wic create directdisk-indirect-recipes -e core-image-minimal --rootfs rootfs1=core-image-minimal --rootfs rootfs2=core-image-minimal-dev Checking basic build environment... Done. Creating image(s)... Info: The new image(s) can be found here: /var/tmp/wic/build/directdisk-indirect-recipes-201506261538-sda.direct The following build artifacts were used to create the image(s): ROOTFS_DIR["platform"]: /home/ed/git/yocto/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs ROOTFS_DIR["secondary"]: /home/ed/git/yocto/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal-dev/1.0-r0/rootfs BOOTIMG_DIR: KERNEL_DIR: /home/ed/git/yocto/poky/build/tmp/deploy/images/qemux86-64 NATIVE_SYSROOT: /home/ed/git/yocto/poky/build/tmp/sysroots/x86_64-linux The image(s) were created using OE kickstart file: /home/ed/git/yocto/poky/scripts/lib/image/canned-wks/directdisk-indirect-recipes.wks # Calculate partition offsets $ /usr/sbin/fdisk -l /var/tmp/wic/build/directdisk-indirect-recipes-201506261538-sda.direct |grep -A4 Device Device Boot Start End Sectors Size Id Type /var/tmp/wic/build/directdisk-indirect-recipes-201506261538-sda.direct1 2048 49137 47090 23M 83 Linux /var/tmp/wic/build/directdisk-indirect-recipes-201506261538-sda.direct2 49152 319353 270202 132M 83 Linux $ echo '2048*512' | bc 1048576 $ echo '49152*512' | bc 25165824 # Mount partitions and look at the ownership/permissions $ sudo mount -oloop,offset=1048576 /var/tmp/wic/build/directdisk-indirect-recipes-201506261538-sda.direct /mnt $ ls -la /mnt/ total 32 drwxr-xr-x 18 root root 1024 Jun 26 15:38 . drwxr-xr-x 23 root root 4096 Mar 9 09:12 .. drwxr-xr-x 2 root root 1024 Jun 25 13:39 bin drwxr-xr-x 2 root root 1024 Jun 25 12:49 boot drwxr-xr-x 2 root root 1024 Jun 25 12:49 dev drwxr-xr-x 17 root root 1024 Jun 26 14:07 etc drwxr-xr-x 3 root root 1024 Jun 25 13:39 home drwxr-xr-x 4 root root 1024 Jun 25 13:39 lib drwx------ 2 root root 12288 Jun 26 15:38 lost+found drwxr-xr-x 2 root root 1024 Jun 25 12:49 media drwxr-xr-x 2 root root 1024 Jun 25 12:49 mnt drwxr-xr-x 2 root root 1024 Jun 25 12:49 proc drwxr-xr-x 2 root root 1024 Jun 25 12:49 run drwxr-xr-x 2 root root 1024 Jun 25 13:39 sbin drwxr-xr-x 2 root root 1024 Jun 25 12:49 sys drwxrwxrwt 2 root root 1024 Jun 25 12:49 tmp drwxr-xr-x 9 root root 1024 Jun 25 13:13 usr drwxr-xr-x 7 root root 1024 Jun 25 13:03 var $ sudo umount /mnt $$ sudo mount -oloop,offset=25165824 /var/tmp/wic/build/directdisk-indirect-recipes-201506261538-sda.direct /mnt [ed@vmed build (work)]$ ls -la /mnt/ total 36 drwxr-xr-x 18 root root 1024 Jun 26 15:38 . drwxr-xr-x 23 root root 4096 Mar 9 09:12 .. drwxr-xr-x 2 root root 2048 Jun 23 13:41 bin drwxr-xr-x 2 root root 1024 Jun 23 11:46 boot drwxr-xr-x 2 root root 1024 Jun 23 11:46 dev drwxr-xr-x 21 root root 2048 Jun 26 14:07 etc drwxr-xr-x 3 root root 1024 Jun 23 13:41 home drwxr-xr-x 6 root root 2048 Jun 23 13:41 lib drwx------ 2 root root 12288 Jun 26 15:38 lost+found drwxr-xr-x 2 root root 1024 Jun 23 11:46 media drwxr-xr-x 2 root root 1024 Jun 23 11:46 mnt drwxr-xr-x 2 root root 1024 Jun 23 11:46 proc drwxr-xr-x 2 root root 1024 Jun 23 11:46 run drwxr-xr-x 2 root root 2048 Jun 23 13:41 sbin drwxr-xr-x 2 root root 1024 Jun 23 11:46 sys drwxrwxrwt 2 root root 1024 Jun 23 11:46 tmp drwxr-xr-x 9 root root 1024 Jun 23 11:46 usr drwxr-xr-x 7 root root 1024 Jun 23 11:46 var Igor, please confirm that above scenario works for you. I put fixes to poky-contrib:ed/wic/misc branch: https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=ed/wic/misc wic was able to produce image with 3 partitions using this .wks: part boot --source rootfs --rootfs-dir=bootfs --ondisk sda --fstype=ext4 --align 1024 --size 128 --part-type C12A7328-F81F-11D2-BA4B-00A0C93EC93B part root --source rootfs --rootfs-dir=rootfs --ondisk sda --fstype=ext4 --align 1024 --size 100 --uuid 65A205ED-63DD-4CB5-A8A6-0FFBE1D4238D part dev --source rootfs --rootfs-dir=devfs --ondisk sda --fstype=ext4 --align 1024 --size 200 --uuid 65A205ED-63DD-4CB5-A8A6-0FFBE1D4238E bootloader --ptable gpt and this command line: wic create --debug directdisk-two-rootfs -e core-image-minimal --rootfs bootfs=core-image-minimal --rootfs rootfs=core-image-minimal --rootfs devfs=core-image-minimal-dev -o /tmp/wic/ Note: I had to use --fstype=ext4 for boot partition as I don't have a recipe to produce EFI boot rootfs. Please change it to --fstype=vfat for real usage and verification. (In reply to comment #11) > Note: I had to use --fstype=ext4 for boot partition as I don't have a recipe > to produce EFI boot rootfs. I do not understand why that is an issue: shouldn't wic just take any file provided and copy into the partition? Sure, it won't boot if you do not have the correct files, but what has that to do with the --fstype ? > Please change it to --fstype=vfat for real usage and verification. AFAIK it should be FAT32, not vfat. Should I really use vfat? (In reply to comment #12) > (In reply to comment #11) > > > Note: I had to use --fstype=ext4 for boot partition as I don't have a recipe > > to produce EFI boot rootfs. > > I do not understand why that is an issue: shouldn't wic just take any file > provided and copy into the partition? > wic uses mcopy for to make an image out of directory. mcopy takes too long to convert big directories or just hangs, I don't know. > Sure, it won't boot if you do not have the correct files, but what has that > to do with the --fstype ? > > > Please change it to --fstype=vfat for real usage and verification. > > AFAIK it should be FAT32, not vfat. Should I really use vfat? for vfat fstype wic uses mkdosfs without specifying type of the file system. in this mode mkdosfs decides itself which FAT size to use. Here is an info about that from mkdsfs man: -F FAT-size Specifies the type of file allocation tables used (12, 16 or 32 bit). If nothing is specified, mkfs.fat will automatically select between 12, 16 and 32 bit, whatever fits better for the filesystem size. I'd suggest you to try if current approach works for you and create a separate bug if it doesn't. It applies to every feature you'd like to have. I'd really prefer to have one bug per real issue instead of 'generic thoughts' kind of bugs like this one. (In reply to comment #13) > wic uses mcopy for to make an image out of directory. mcopy takes too long > to convert big directories or just hangs, I don't know. But it must be made to work. Btw, it used to work for me. > I'd suggest you to try if current approach works for you and create a > separate bug if it doesn't. I'm ok with creating other bugs and make them blocking this one, but it's simpler to use it to track everything. It also contains the timing information for planning purposes, so I do not want to close it till the entire functionality is implemented. But we can have blockers, rather than everything in one bug. > It applies to every feature you'd like to have. I'd really prefer to have > one bug per real issue instead of 'generic thoughts' kind of bugs like this > one. I propose to create blockers for this one, for the reasons I explained above. (In reply to comment #14) > (In reply to comment #13) > > > wic uses mcopy for to make an image out of directory. mcopy takes too long > > to convert big directories or just hangs, I don't know. > > But it must be made to work. Btw, it used to work for me. > It must be made to work on real boot rootfs. I don't have one, but I hope wic will work with your boot directory. I see no reason to fix it for big rootfs as converting them to fat file system is not realistic use case. It's just a waste of time to fix that from my point of view. It could be even that it works, but very slow. > > I'd suggest you to try if current approach works for you and create a > > separate bug if it doesn't. > > I'm ok with creating other bugs and make them blocking this one, but it's > simpler to use it to track everything. > It also contains the timing information for planning purposes, so I do not > want to close it till the entire functionality is implemented. > > But we can have blockers, rather than everything in one bug. OK, agree. > > > It applies to every feature you'd like to have. I'd really prefer to have > > one bug per real issue instead of 'generic thoughts' kind of bugs like this > > one. > > I propose to create blockers for this one, for the reasons I explained above. Feel free to create them if you see them. (In reply to comment #15) > I see no reason to fix it for big rootfs as converting them to fat file > system is not realistic use case. It's just a waste of time to fix that from > my point of view. It could be even that it works, but very slow. I think now I see your concern, but I was proposing to just use touch for creating a couple of files, not to use a full rootfs. All you need is to create the directory and slap one or two files in it. Or you can use the patch I provided you some weeks ago, which already does create the proper EFI directory. I cannot post here the errors I'm getting, but it's failing. I will show you offline what is going wrong for me. I've looked at the info you've provided offline and it looks like you're using old version of wic. As I pointed out in my example wic should be taken from ed/wic/misc branche of poky-contrib git repo. Now all my patches have been already accepted and you can get them either from poky or from openembedded-core. this seems to be fixed long ago. feel free to open separate bugs for the specific issues if any. Verified on master: f78f90240a3041be426c6ba9085849352164ee8b The features required were already implemented in wic based on comment #10 |