Bug 1763 - [USE CASE] More universally bootable live images
Summary: [USE CASE] More universally bootable live images
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: deployment (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium enhancement
Target Milestone: 1.5
Assignee: Darren Hart
QA Contact:
URL:
Whiteboard: (Waiting on EFI/SYSLINUX/HYBRID Patches)
Depends on:
Blocks:
 
Reported: 2011-11-09 10:44 UTC by Damien Lespiau
Modified: 2013-10-18 16:11 UTC (History)
9 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
Create a valid bot sector for the .iso image (1.02 KB, patch)
2011-11-10 10:35 UTC, Damien Lespiau
no flags Details | Diff
Darren Hart's USB-ZIP image creation script (1.38 KB, application/x-sh)
2012-04-03 08:42 UTC, Paul Eggleton
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Damien Lespiau 2011-11-09 10:44:16 UTC
Right now it seems that the solution to create "live" images that can boot on netbook/laptop/desktop class hardware is to dd the generated .hddimg image. That only works for a limited number of BIOSes (for some not quite clear reason at this point).

Fedora has a tool to do that:
http://git.fedorahosted.org/git/?p=livecd;a=tree

and it makes images that work on a wide range of BIOSes and EFI bootloaders with some extra goodies (like dm-snapshot to make a read/write rootfs, the resulting image can be dd'ed on a USB key or burned to a CD-ROM, squashfs compression, ...).

FWIW, meego has a fork of this tool:
http://meego.gitorious.org/meego-developer-tools/image-creator/trees/master/mic/imgcreate

Poky could benefit from having something like this to create live images that boot everywhere.
Comment 1 Richard Purdie 2011-11-09 15:59:34 UTC
As far as I can tell the missing piece we have is http://git.fedorahosted.org/git?p=livecd;a=blob;f=tools/livecd-iso-to-disk.sh i.e. the piece for writing the generated image onto the usb stick. dd doesn't quite seem to be enough.

If we had a version of that script does that resolve 95% of the problem? (assuming we write out one of our live images and make the install option at the syslinux prompt more obvious)
Comment 2 Damien Lespiau 2011-11-10 04:12:40 UTC
I think there's something else, we've always dd'ed meego images onto USB disks and it worked everywhere.
Comment 3 Richard Purdie 2011-11-10 04:15:56 UTC
Do you have one of those images handy? Can you confirm whether they have a partition table or not? (show up as /dev/sdX or /dev/sdX1)
Comment 4 Damien Lespiau 2011-11-10 07:03:50 UTC
These .iso files are dd'ed on USB keys and seems to be generated by http://git.fedorahosted.org/git/?p=livecd;a=blob;f=imgcreate/live.py.

Note that the meego fork of this tool seems to be before the introduction of EFI bootloaders support.

$ file meego-netbook-ia32-1.2.0.20110909.0924.iso 
meego-netbook-ia32-1.2.0.20110909.0924.iso: # ISO 9660 CD-ROM filesystem data 'netbook-ia32-x86_64-201109090924' (bootable)

FWIW the linux command line is:
append initrd=initrd0.img root=CDLABEL=netbook-ia32-x86_64-201109090924 rootfstype=iso9660 ro liveimg quiet

$ sudo hexdump -C -n 512 /dev/sdb
00000000  fa 31 ed 8e d5 bc 00 7c  fb fc 66 31 db 66 31 c9  |.1.....|..f1.f1.|
00000010  21 f6 74 26 f6 04 7f 75  21 38 4c 04 74 1c 66 3d  |!.t&...u!8L.t.f=|
00000020  21 47 50 58 75 10 80 7c  04 ed 75 0a 66 8b 4c 34  |!GPXu..|..u.f.L4|
00000030  66 8b 5c 38 eb 04 66 8b  4c 08 66 53 66 51 06 57  |f.\8..f.L.fSfQ.W|
00000040  8e dd 8e c5 52 be 00 7c  bf 00 06 b9 00 01 f3 a5  |....R..|........|
00000050  ea 55 06 00 00 52 b4 41  bb aa 55 31 c9 30 f6 f9  |.U...R.A..U1.0..|
00000060  cd 13 72 16 81 fb 55 aa  75 10 83 e1 01 74 0b 66  |..r...U.u....t.f|
00000070  c7 06 fb 06 b4 42 eb 15  eb 00 5a 51 b4 08 cd 13  |.....B....ZQ....|
00000080  83 e1 3f 5b 51 0f b6 c6  40 50 f7 e1 53 52 50 bb  |..?[Q...@P..SRP.|
00000090  00 7c b9 04 00 66 a1 b0  07 e8 44 00 0f 82 80 00  |.|...f....D.....|
000000a0  66 40 80 c7 02 e2 f2 66  81 3e 40 7c fb c0 78 70  |f@.....f.>@|..xp|
000000b0  75 09 fa bc ec 7b ea 44  7c 00 00 e8 83 00 69 73  |u....{.D|.....is|
000000c0  6f 6c 69 6e 75 78 2e 62  69 6e 20 6d 69 73 73 69  |olinux.bin missi|
000000d0  6e 67 20 6f 72 20 63 6f  72 72 75 70 74 2e 0d 0a  |ng or corrupt...|
000000e0  66 60 66 31 d2 66 03 06  f8 7b 66 13 16 fc 7b 66  |f`f1.f...{f...{f|
000000f0  52 66 50 06 53 6a 01 6a  10 89 e6 66 f7 36 e8 7b  |RfP.Sj.j...f.6.{|
00000100  c0 e4 06 88 e1 88 c5 92  f6 36 ee 7b 88 c6 08 e1  |.........6.{....|
00000110  41 b8 01 02 8a 16 f2 7b  cd 13 8d 64 10 66 61 c3  |A......{...d.fa.|
00000120  e8 1e 00 4f 70 65 72 61  74 69 6e 67 20 73 79 73  |...Operating sys|
00000130  74 65 6d 20 6c 6f 61 64  20 65 72 72 6f 72 2e 0d  |tem load error..|
00000140  0a 5e ac b4 0e 8a 3e 62  04 b3 07 cd 10 3c 0a 75  |.^....>b.....<.u|
00000150  f1 cd 18 f4 eb fd 00 00  00 00 00 00 00 00 00 00  |................|
00000160  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001b0  94 00 00 00 00 00 00 00  7f 74 40 0c 00 00 80 00  |.........t@.....|
000001c0  01 00 17 3f e0 6e 00 00  00 00 00 78 1b 00 00 00  |...?.n.....x....|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200

So yes it has a partition table as well and 1 primary partition

$ ll /dev/sdb*
brw-rw---- 1 root disk 8, 16 Nov 10 14:55 /dev/sdb
brw-rw---- 1 root disk 8, 17 Nov 10 14:55 /dev/sdb1

Once mounted:

$ ll -R /mnt/
/mnt/:
total 4
dr-xr-xr-x 2 root root 2048 Sep  9 17:26 isolinux
dr-xr-xr-x 2 root root 2048 Sep  9 17:26 LiveOS

/mnt/isolinux:
total 8277
-r--r--r-- 1 root root    2048 Sep  9 17:27 boot.cat
-r--r--r-- 1 root root 2271856 Sep  9 17:26 initrd0.img
-r--r--r-- 1 root root   24576 Sep  9 17:26 isolinux.bin
-r--r--r-- 1 root root     959 Sep  9 17:26 isolinux.cfg
-r--r--r-- 1 root root  231718 Sep  9 17:26 splash.jpg
-r--r--r-- 1 root root  159392 Sep  9 17:26 vesamenu.c32
-r--r--r-- 1 root root 5783840 Sep  9 17:26 vmlinuz0

/mnt/LiveOS:
total 890964
-r--r--r-- 1 root root      8192 Sep  9 17:26 osmin.img
-r--r--r-- 1 root root 912338944 Sep  9 17:27 squashfs.img
Comment 5 Damien Lespiau 2011-11-10 10:35:17 UTC
Created attachment 279 [details]
Create a valid bot sector for the .iso image
Comment 6 Darren Hart 2011-11-15 13:31:20 UTC
I'm interested in how they made this more palatable to a wide range BIOSes. When I've run into boot failures with the current live images, switching to the USB-ZIP format as documented in README.hardware for the x86 BSPs has resolved the issue. So seeing this accomplished without USB-ZIP is nice - especially since the commands to generate that format take an extrememly long time.

As I am currently looking at integrating support for EFI, one thing I had been considering was revamping the x86 live images to include syslinux and efi boot support on the same live image. However, I'm trying to reconcile that with the idea that these image are tailored to BSPs - a "universally bootable" image seems at odds with that concept. If I'm building for the n450, I wouldn't be terribly surprised if that doesn't boot on my netbook.

That said, I think there is definitely room for improvement here - I'm just not sure "universally bootable" should be a goal here.

Thoughts?
Comment 7 Richard Purdie 2011-11-18 06:26:16 UTC
I've tested the patch and its defintely an improvement as the .iso image works on one of my devices I've previously had problems with. Thanks Damien!

There are a few other ways we should try and improve the images and we're looking at that...
Comment 8 Song Liu 2012-01-30 16:01:14 UTC
Saul, would you please take a look at this and see how much effort it needs? 

Thanks,
Song
Comment 9 Darren Hart 2012-03-14 22:46:55 UTC
This is going to take quite a bit of thought to get right. Saul and I discussed this a bit more recently. Some others have made requests of support for various different types of images. To do this right, we will need more time than we have before we need to stop introducing major functionality changes into the 1.2 release. This really needs to be a 1.3 target.
Comment 10 Paul Eggleton 2012-04-03 08:42:12 UTC
Created attachment 430 [details]
Darren Hart's USB-ZIP image creation script

I've added Darren's mkzipimage.sh script from bug 1984 which automates the process of formatting a USB disk for booting in USB-ZIP mode; if we don't remove the need for USB-ZIP usage we should add this to scripts/contrib for 1.3.
Comment 11 Ross Burton 2012-04-05 12:32:18 UTC
I thought I'd mention in case relevant people are not aware that Matthew Garrett at Red Hat has been working on making Fedora ISOs boot on Macs (thus EFI). The resulting image is a true Frankenstein's monster of partitions but it boots on pretty much everything.
Comment 12 Darren Hart 2012-05-02 16:35:42 UTC
I think this is something I'll be spending time on as part of the many overlapping boot issues I'm working with. Saul, would you like to assign to me?
Comment 13 Ross Burton 2012-05-02 16:36:56 UTC
The isohybrid patch makes images work on a lot more machines, could it be backported to denzil?
Comment 14 Darren Hart 2012-05-02 17:33:36 UTC
(In reply to comment #13)
> The isohybrid patch makes images work on a lot more machines, could it be
> backported to denzil?

Does this work in lieu of the USB ZIP format for the machine you were having trouble with? If so, I think I'd like to see isohybrid pulled into 1.2.1 as it was a very simple change.
Comment 15 Ross Burton 2012-05-02 17:37:22 UTC
Yes, the isohybrid patch makes the iso images work from a USB stick on my Toshiba pinetrail netbook.
Comment 16 Damien Lespiau 2012-05-02 17:39:30 UTC
Yes, that patch was enough to not have to resort to the USB-ZIP when I tested it originally.
Comment 17 Darren Hart 2012-05-02 17:54:07 UTC
That's great. Would either of you be able to apply the patch to denzil, test it, and send it to out for inclusion for 1.2.1. Be sure to mention the point release and CC Scott Garman (the 1.2.1 maintainer).
Comment 18 Darren Hart 2012-05-02 17:55:16 UTC
(In reply to comment #17)
> That's great. Would either of you be able to apply the patch to denzil, test
> it, and send it to out for inclusion for 1.2.1. Be sure to mention the point
> release and CC Scott Garman (the 1.2.1 maintainer).

This is the one to backport:

2b05d35 bootimg.bbclass: Tweak iso images to become hybrid images
Comment 19 Ross Burton 2012-05-18 19:35:33 UTC
Confirming that the isohybrid patch fixes .iso booting on my Acer Aspire S3.  The .hddimg hangs at the syslinux boot: prompt.
Comment 20 Darren Hart 2012-05-18 19:45:52 UTC
(In reply to comment #19)
> Confirming that the isohybrid patch fixes .iso booting on my Acer Aspire S3.
> The .hddimg hangs at the syslinux boot: prompt.

I'd wager the USB ZIP format could fix this as well. Still, one image to rule them all is very appealing. I need to investigate the fedora live images and see about how to get EFI working with the hybrid images. If that works, we could deep six hddimg completely, which would be awesome.
Comment 21 Ross Burton 2012-05-29 15:30:23 UTC
I'm using hddimg for Guacamayo by writing the hddimg to a USB stick and then booting/working from it.  Please don't get rid of it!
Comment 22 Darren Hart 2012-05-29 15:52:11 UTC
(In reply to comment #21)
> I'm using hddimg for Guacamayo by writing the hddimg to a USB stick and then
> booting/working from it.  Please don't get rid of it!

Is there a compelling reason to use hddimg over the hybrid iso image?
Comment 23 Ross Burton 2012-05-29 16:05:59 UTC
hddimg boots up with a persistant writable rootfs, AFAIK the hybrid ISO images are not persistent (I may be wrong).
Comment 24 Darren Hart 2013-07-16 17:41:01 UTC
We're looking at replacing live images, see 4103.
Comment 25 Darren Hart 2013-09-23 18:59:46 UTC
Pending on the EFI/SYSLINUX/HYBRID patches from Jason Wessel to land, that will impact if this is a 1.5 or a 1.6 target.
Comment 26 Darren Hart 2013-10-18 16:11:33 UTC
24ffda4 bootimage.bbclass, grub-efi.bbclass: Improve EFI & PCBIOS+EFI ISO support