Bug 3847 - New partitioning description and tooling
Summary: New partitioning description and tooling
Status: VERIFIED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: deployment (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: High enhancement
Target Milestone: 1.5.1
Assignee: Tom Zanussi
QA Contact: Ionut Chisanovici
URL:
Whiteboard:
Depends on:
Blocks: 1950 4075 4106 4778 4779 4780 4781 5325
  Show dependency tree
 
Reported: 2013-02-05 23:01 UTC by Darren Hart
Modified: 2015-08-07 12:58 UTC (History)
11 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Yes (doc changes required)


Attachments
boot output from wic-generated minnow boot (49.41 KB, application/octet-stream)
2013-10-23 21:33 UTC, Tom Zanussi
no flags Details
boot output from wic-generated nuc boot (31.46 KB, text/plain)
2013-10-23 21:38 UTC, Tom Zanussi
no flags Details
boot output from wic-generated fri2 boot (50.51 KB, application/octet-stream)
2013-10-25 03:58 UTC, Tom Zanussi
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Darren Hart 2013-02-05 23:01:56 UTC
We currently don't have any generic mechanism for creating properly partitioned images. This results in the custom scripts like mkefidisk and such that do this manually, outside of bitbake. The lack of this feature hurts our x86 live images as we are currently creating disk volumes, and not complete disk images. Many firmware implementations fail to boot these types of images.

There is also interest from developers in being able to describe their desired final image and have bitbake assemble this image. There are many pieces to such a mechanism, and ultimately this should be able to be run outside of bitbake, but should also be used by bitbake as the image creator.

For the purposes of this bug, we are looking to define (or adopt) a partitioning description language. It needs to be able to describe the partitions, their type, their offsets, lengths, flags, etc. Anything that a user may want to do with libparted. A python tool using the pyparted (python-parted) module should then be able to accept as input the description file and an image (or device) which it will then partition.

Ultimately the filesystems images (.ext4 for example) created by bitbake will be written to these partitions at their specified offset in the file. This avoids the need for root permissions for working with the loop device tools.
Comment 1 Darren Hart 2013-02-05 23:03:48 UTC
I would recommend something along the lines of python config rather than XML,as we don't use XML much in Yocto and the files are meant to be hand crafted.
Comment 2 venkata ramana g 2013-06-14 05:51:55 UTC
Came up with a configuration format to meet binary configuration requirement. bug 3252. Added sample configuration as an attachment. Also prototyped the parser. Can we discuss further to freeze on a common configuration format.
Comment 3 venkata ramana g 2013-07-17 06:39:47 UTC
Updated Binary Configuration format and design in wiki. As this feature need to work with in current poky and also outside.This feature can be implemented under Binary configuration feature exporting rootfs by adding required config commands.

http://wiki.yoctoproject.org/wiki/Binary_configuration_support
Comment 4 Tom Zanussi 2013-07-30 23:12:07 UTC
As an overview and starting point, let's review the way image building
in yocto currently works.  Basically we start with an image recipe
that inherits from image.bbclass, which in turn adds a rootfs task
that creates a /rootfs directory containing all the installed packages
included directly or via dependencies in the particular image recipe.

Once that's done, the image class creates a list of 'image commands'
for each of the defined IMAGE_FSTYPES (usually added by the machine
config or image recipe) - for every IMAGE_FSTYPE 'type', there's a
matching IMAGE_CMD_type, which is either one of the base types defined
in image_types.bbclass or a type added by a layer's own
image_types.bbclass 'extension' (see below).

'image commands' are essentially self-contained shell command snippets
that are invoked on the rootfs contents, one by one.  So, for
instance, if a machine defines IMAGE_FSTYPES as "ext3 cpio.gz", the
'ext3 command' defined by IMAGE_CMD_ext3 is applied e.g.:

 $ genext2fs -b $ROOTFS_SIZE ... ${IMAGE_NAME}.rootfs.ext3
 $ tune2fs -j ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ext3

producing image-xxx.ext3, and then the 'cpio' IMAGE_CMD is applied,
producing a separate image-xxx.cpio i.e. two separate IMAGE_FSTYPES
produce two images after application of the IMAGE_CMDs.  There are
also 'compression' commands that are applied to the results of the
types (COMPRESSION_TYPES/COMPRESS_CMD_), used to produce for example
the final cpio.gz or tar.bz2.

So most of the basic 'image commands' defined in image_types.bbclass
essentially just create simple filesystems corresponding to the
FSTYPES incorporating the rootfs contents.  But there are other image
commands that are higher-level and act on the output of the simpler
filesystem-creating commands to create more complex images that
incorporate those filesystems and add a bootloader, for instance.  The
'live' and 'directdisk' images discussed below are of that type, and
other BSPs and images add their own high-level FSTYPES via their
layers which make their custom FSTYPES available for building
board-specific bootable images, for instance.

User-defined types are typically added by creating a class that
inherits image_types.bbclass and adding it to IMAGE_CLASSES:

image_types_foo.bbclass:
  IMAGE_CMD_bar = "some shell commands"
  IMAGE_CMD_baz = "some more shell commands"

# include this in e.g. machine.conf
foo-default-settings.inc
  IMAGE_CLASSES += "image_types_foo"

The above commands are picked up and made available because of the
below in image.bbclass:

  inherit ${IMAGE_CLASSES}

So basically, the user can now add:

IMAGE_FSTYPES += "bar baz"

to have the 'bar' and 'baz' commands invoked to do whatever their
shell commands do e.g. create a bootable image specialized to work on
a specific machine or SOC family, etc.

In any case, there are a couple of these higher-level image-types that
are treated differently in a hard-coded way by image.bbclass.  They
are:

'vmdk'
'live'

These essentially correspond to higher-level image assembly 'commands'
that take the output of the previous IMAGE_FSTPES and assemble it into
a bootable image, which is a multi-step processs.

Specifically, for the above two types:

If IMAGE_TYPES contains 'vmdk', image.bbclass explicitly inherits
image-vmdk.bbclass.  image-vmdk does a couple things first:

  - creates a dependency on do_rootfs, ensuring e.g. the ext3 rootfs
    gets generated first
  - sets the ROOTFS variable for boot-directdisk to find
  - finally inherits boot-directdisk.

boot-directdisk is where the rubber hits the road, so to speak.  At a
high level, it:

  - generates the syslinux config file
  - creates 2 partitions:
    - fat16 for boot files
    - ext3 for the rootfs
  - copies the syslinux.cfg, ldlinux.sys, and kernel to the fat16 partition
  - dd's the rootfs to the ext3 partition

The vmdk also appends its own stuff onto IMAGE_CMD_ext3:
 - tune2fs -m 0.5 ... rootfs.ext3 // reserve space for root

Likewise, if IMAGE_TYPES contains 'live', image.bbclass explicitly
inherits image-live.bbclass.  image-live does a few things first:

  - creates two dependencies on do_rootfs:
    - one for INITRD_IMAGE, ensuring the initramfs rootfs gets
    generated first
      - creates the initramfs cpio.gz copies that to hddimg/initrd
    - one for the rootfs, ensuring it gets created before the next step
  - sets the ROOTFS variable for bootimg to find
  - finally inherits bootimg.

bootimg is where the rubber hits the road, so to speak.  At a high
level, it creates the .hddimg file:

  - generates the EFI or grub config file
  - installs bzImage into ./hddimg/vmlinuz
  - copies initramfs...cpio.gz to ./hddimg/initrd
  - copies rootfs.ext3 to ./hddimg/rootfs.img
  - [if PCBIOS] copies syslinux.cfg, menu, ldlinux.sys in to ./hddimg/
  - [if EFI] create /EFI/BOOT, copy bootxxx.efi, grub.cfg in to ./EFI/BOOT/
  - mkdosfs, creating .hddimg file basically
  - mcopy ./hddimg* into .hddimg
  - [if PCBIOS] run syslinux on the .hddimg

It also creates the .iso file, which is essentially a repeat of the
above but using:
  - mkisofs
  - isohybrid

An interesting side-note about the history of how the above two type
(live and directdisk) were used in the past - basically both types
used to be implemented in an even more hard-coded way: basically for
each image recipe e.g. core-image-minimal.bb, core-image-sato.bb, etc,
there were two more matching recipes - core-image-minimal-live.bb and
core-image-minimal-directdisk.bb, etc.

What they would do was essentially hard-code the same inherit in the
recipe that the image.bbclass does with code e.g.

in poky-image-minimal-directdisk.bb:
  inherit boot-directdisk

Note that the directdisk type was pretty much deprecated in favor of
live images i.e. until the build appliance came along and added the
hard-coded 'vmdk' type into image.bbclass, only the 'live' type was
used by anything.

It's instructive to go through a couple examples of user-defined types
actually used in the wild to get a better idea concerning what needs
to be covered by a revamping of the current system.

First, let's take a look at meta-fsl-arm.  The imx23evk.conf machine
configuration in that layer adds a couple IMAGE_FSTYPES,
'uboot.mxsboot-sdcard' and 'sdcard'.

meta-fsl-arm/imx23evk.conf
  include conf/machine/include/mxs-base.inc
  SDCARD_ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ext3"
  IMAGE_FSTYPES ?= "tar.bz2 ext3 uboot.mxsboot-sdcard sdcard"

The included mxs-base.inc file includes this:

  include conf/machine/include/fsl-default-settings.inc

which in turn adds this:

  IMAGE_CLASSES += "image_types_fsl"

According to the above definition of IMAGE_FSTYPES, the
'uboot.mxsboot-sdcard' IMAGE_CMD will be executed first, followed by
the 'sdcard' IMAGE_CMD.

The image_types_fsl.bbclass referenced above adds the definitions of
those IMAGE_CMDs:

  inherit image_types
  IMAGE_CMD_uboot.mxsboot-sdcard = "mxsboot sd ${DEPLOY_DIR_IMAGE}/u-boot-${MACHINE}.${UBOOT_SUFFIX} \
                                      ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.uboot.mxsboot-sdcard"

So first the uboot.mxsboot-sdcard command executes the 'mxsboot' tool
to create rootfs.uboot.mxsboot-sdcard.

Once that's done, it moves on to the IMAGE_CMD_sdcard command, which
again is a fairly involved shell script that does:

  - calculate alignments and sdcard size
  - dd's if=/dev/zero ... to initializes a sparse file
  - executes SDCARD_GENERATION_COMMAND
    - there are several of these, chosen by SOC override
  - using 'generate_mxs_sdcard' for this example:
    - SDCARD_ROOTFS has already been defined by the caller
      parted SDCARD msdos label
      if IMAGE_BOOTLOADER imx-bootlets
        parted primary 1024
        parted primary ROOTFS_SIZE
        dd 4 bytes
        dd ..linux.sb of=SDCARD seek=2052
      if IMAGE_BOOTLOADER uboot
        parted primary 1024 2048
        parted primary ...
        parted primary ROOTFS_SIZE
        dd ${IMAGE_NAME}.rootfs.uboot.mxsboot-sdcard
        mkfs vfat ..
        mcopy boot.img
        mcopy devicetree
      set partition type to 0x53
      dd ROOTFS

Withouth going into details, the above creates several partitions and
twiddles various machine-specific bits to get a bootable image.  Note
of course that one step is to dd the output of the previous
uboot.mxsboot-sdcard IMAGE_CMD into one of the partitions.

Similar but significantly different steps are carried out for the
other machine types served by this specialized image_types class.

Other BSPs do similar things e.g. meta-raspberrypi.  Still others do
nothing but document a manual way of creating such images, and yet
others have a specialized tool that does everything from a single
command-line invocation.

Finally, we have one other example in Yocto of image creation in
scripts/contrib/mkefidisk.sh.  This is a shell script that essentially
creates a partitioned EFI-bootable directdisk image from a live
.hddimg, which is what's produced by the meta-intel builds.  It
basically does on the host what the 'install to hd' menu item does if
you boot the live image, but adds EFI support:

	# delete partition table
	dd if=/dev/zero of=$DEVICE bs=512 count=2

	# uses MSDOS by default instead of GPT because backup table
	# needs to go on last block of device

	# create target partitions
	mkpart primary 0 BOOT_SIZE
	set 1 boot on
	mkpart primary ROOTFS_START ROOTFS_END
	mkpart primary SWAP_START 100%

	# make target filesystems
	mkfs.vfat $BOOTFS -n "efi"
	mkfs.ext3 $ROOTFS
	mkswap $SWAP

	# copy rootfs to target ROOTFS
	mount .hddimg/rootfs.img
	copy to ROOTFS

	# copy swap settings to etc/tab
	echo swap settings to ROOTFS/etc/tab

	# make efi dirs in and copy stuff to BOOTFS
	mkdir $BOOTFS/EFI/BOOT
	cp .hddimg/vmlinuz to BOOTFS
	cp .hddimg/EFI/BOOT/* to $BOOTFS/EFI/BOOT (booti*.efi and grub.cfg)
	delete 'install' entry, initrd lines, root= kernel params, and LABEL= strings from grub
	add root= with target rootfs
	add target rootfs, ro, rootwait etc to cmdline

Essentially, we should be able to do this directly without having a
'live' image in between.

Which brings us to the specific requirements of this bug/enhancement.

In fact what we currently have in Yocto is essentially a multi-level
plugin architecture.  At the highest level, we have things like
'sdcard', 'vmdk', and 'live', which are essentially plugins that act
on the products of lower-level plugins such as 'ext3', 'cpio.gz'.  At
both levels, and arbitrarily intermixed, we have the lowest-level
functions such as 'parted' commands, etc.  We also have inheritance
relationships at play here e.g. vmdk essentially using _append to add
something to the base, and we have stacking as well e.g. applying
compression on top of the output of previous steps.

It would be much nicer to be in a position to take advantage of actual
language support for such features.  The current plugin scheme is
understandable given that we're dealing with self-contained shell
scripts and mixed-languages, but if we could get rid of those and
promote everything to full-fledged Python code, we could then take
advantage of those language features and accomplish the same thing
with the backing of the language.  Additionally, given that the newer
parts of the system are mainly pure Python code and one of the
original goals of the current work was to replace the current set of
partitioning commands with pyparted, it would make sense to replace
those parts with Python equivalents as well.

Additionally, it should be obvious from the above examples that
real-world partitioning requires more than just a simple partition
language.

A simple partition language could handle extremely simple use cases
and/or the simpler parts of some of the above flows, but for any
non-trivial use case, there needs to be a higher-level ability to
orchestrate the assembly of arbitrarily complex images.  Thus the
nominal requirement to create a partitioning description needs to be
viewed in the context of the overall task of image creation.

We also have the explicit requirement that whatever we create, it has
to be able to generate images both from within the build system as
well via external scripts.

Because image creation and partition description isn't really a new
problem, I looked around for existing 'partition description
languages' that might fit the bill, but surprisingly didn't find
anything that really covered partition description in the context of
image creation.  I did however find partition description in the
context of 'target install', which I think is very close to what we
need for partitioned image creation as well, with the benefit
therefore it will also be useful for installation as well.

In any case, what we're talking about is syntax and not the
implementation behind it, so if there's a syntax that can be adapted
for our purposes, there doesn't seem to be any reason for reinventing
our own.  The specific syntax I'm specifically referring to is the
Fedora kickstart syntax for describing partitions and bootloaders:

 clearpart
 part
 bootloader

 and possibly autopart

Though the Fedora kickstart syntax also addresses other areas that
overlap with the overall deployment effort, such as install, package
selections, user setup, network setup, I'm explicitly not including or
even suggesting we use similar syntax for those - this covers only
partitioning and image creation and nothing outside that - other tools
and/or integration layers can adopt or create their own syntax.

The one exception to that is installation - it would be nice to be
able to reuse the same syntax and code for installation, and in fact,
the kickstart syntax for those kinds of extensions would be easy to
add e.g. things like the --ondisk parameter to image creation, which
we may want to adopt in the external image creation tools anyway to
allow for putting some parts of an image on separate disks, etc.

The kickstart documentation is here for reference:

http://fedoraproject.org/wiki/Anaconda/Kickstart

Again, the 3 commands we're interested in making use of are:

 clearpart
 part
 bootloader

And again, this is just meant to be a syntactical starting point.

Implementation-wise, unlike Fedoara kickstart files, the whole thing
is implemented as Python code.

The overall setup of our simple Yocto version of these (.yks?) files
is:

  def pre():
    free-form python or named 'plugin' commands

  clearpart commands
  part commands
  bootloader commands
  named 'plugin' commands

  def post():
    free-form python or named 'plugin' commands

Basically, the 'clearpart' command is obvious and uninteresting - it's
simply a command to clear a disk of partitions or clear a single
partition.  It has a few options, such as --all or --sparse (our own
addition to the standard Fedora kickstart syntax).

The 'bootloader' command is a little more interesting - it's the
command that actually installs the bootloader.  It handles the
standard Fedora options such as --location=partition or --location=mbr
and or whatever makes sense.  We could also enhance the syntax if
needed to mention specific bootloaders, etc.  The current assumption
is that the bootloader command is smart enough to figure out from the
'part' commands how to install a bootloader, not sure yet, but
whatever makes sense.

The 'part' command is the most important command - it actually creates
the partitions and default versions automatically create and install a
filesystem.  For example, this 'part' command by default creates an
ext3 filesystem and copies ROOTFS to it:

part	  /		--fstype=ext3		--size=grow

Note that ROOTFS must point to the appropriate ROOTFS before it's
invoked.  This can be done either from within the Yocto build system
as it already is in that context, or via the pre() implementation if
it's an external script.

Here's the command to create an EFI boot partition (again this is
standard Fedora syntax as well):

part	  /boot/efi	--fstype=efi		--grow	--maxsize=200

Implmentation-wise, this command invokes the encapsulated set of
commands named (for example) 'efi-boot.py' See the efidisk.yks example
below for details (efidisk.yks is essentially a replacement for
mkefidisk.sh).

The --fstype=efi parameter or a non-hyphenated second parameter to the
'part' command can be used to provide an unlimited set of
partition-specific sub-commands.  This is also consistent with Fedora
kickstart - for example Fedora kickstart adds this to deal with a
specific type of biosboot partition:

part biosboot --fstype=biosboot --size=1

In our case, we'd like to be able to use that second parameter to
allow users to add and use their own named set of partitioning
commands - users simply implement the python (or wrapped shell) code
to implement the function and add it to the system and use it from the
'part' command.  This essentially enables the same type of plug-in
architecture as already exists in Yocto, but now usable from both the
build and external contexts.

A tiny extension on that would be to allow the addition of top-level
commands as well, which would mainly be useful for re-use.

Finally, the pre() and post() .yks functions - these are essentially
the equivalents of %pre and %post in .ks files, but again implemented
as python code.  These functions can be used to do anything, but
typically are there to do pre-image creation calculations, create
and/or stage pre-image-creation artifacts, or do more complex assembly
tasks.  See the examples below for more concrete ideas.

Finally, a couple other advantages of adopting this format is that we
can probably take advantage of the already-existent parsing code for
the kickstart 'language', of course extending it and adapting it to
our needs.  We might also want to make use of some of the anaconda
python code for certain partitioning tasks, but that's neither here
nor there...

So for this particular bug, what I think makes sense for a M4
deliverable would be the following:

- define the specific 'part', 'clearpart', and 'bootloader' syntax and
  create/poach a .yks syntax parser based on that.

- create the basic 'plugin' architecture needed to encapsulate and
  implement the 'part' subcommands

- replace all of the partitioning/bootloader commands that currently
  exist in bootimg.bbclass/directdisk.bbclass/vmdk.bbclass with their
  .yks equivalents (the .yks commands will be built on top of
  pyparted, so this essentially means adding pyparted into the build
  environment, removing the current uses of parted and replacing them
  with with pyparted calls.)  This also means that the .yks commands
  need to be available as a python module that can be used both from
  within the build system and the .yks scripts.

- create a version of the mkefidisk.sh as an external .yks file, which
  does the same thing doesn't do the .hddimg dance i.e. directly
  creates a partitioned bootable image as an external script.

- create an equivalent meta-fsl-arm imx31pdk image (or any other
  interesting image) using .yks.

- note that this doesn't touch the image-types.bbclass and all the
  base IMAGE_CMDs, etc, but we somehow need to get access to those,
  and probably other things from the build system as well, in order to
  create the prerequisites for image creation from external scripts.


Examples:


====
vmdk.yks:
====

def pre():
    generate rootfs
    generate syslinux config file
    copy syslinux.cfg, ldlinux.sys, and kernel to staging

clearpart --all
part	  /boot		--fstype=fat16		--type=syslinux	--size=fit
part	  /		--fstype=ext3		--size=grow(0.5)
bootloader		--location=partition	--append="console=ttyS0,115200"

** the rootfs and syslinux.cfg files should have already been
   generated and made available before the first command.  This can be
   accomplished via a 'pre' section.

** the 'part /' command automatically encapsulates the basic default
   behavior of dd'ing the rootfs to the given partition, in addition
   to actually creating the partition (ext3 here)

** the 'part /boot' command automatically encapsulates the basic steps
   required to set up and copy the bootloader files into the /boot
   partition i.e. it encapsulates staging the syslinux.cfg,
   ldlinux.sys and kernel files

** the 'bootloader' command does the actual bootloader install

** the --size=grow(0.50) accomplishes what the vmdk recipe's tune2fs
   append accomplishes.  This could also be accomplished in a 'post'
   section.

====
efidisk.yks: (this is essentially a replacement for mkefidisk.sh)
====

def pre():
    generate rootfs
    generate syslinux config file
    copy syslinux.cfg, ldlinux.sys, and kernel to staging

clearpart --all
part	  /boot/efi	--fstype=efi		--grow	--maxsize=200
part	  /		--fstype=ext3		--size=256000
part	  swap		--size=300%
bootloader		--location=partition	--append="console=ttyS0,115200"

** The fstype=efi partition type encapsulates the logic to create an
   EFI partition (whatever the best way to do it is, TBD) e.g:

   # uses MSDOS by default instead of GPT because backup table
   # needs to go on last block of device
   mkpart primary 0 BOOT_SIZE
   set 1 boot on
   # make target filesystems
   mkfs.vfat $BOOTFS -n "efi"
   # make efi dirs in and copy stuff to BOOTFS
   mkdir $BOOTFS/EFI/BOOT
   cp .hddimg/vmlinuz to BOOTFS
   cp .hddimg/EFI/BOOT/* to $BOOTFS/EFI/BOOT (booti*.efi and grub.cfg)
   delete 'install' entry, initrd lines, root= kernel params, and LABEL= strings from grub
   add root= with target rootfs
   add target rootfs, ro, rootwait etc to cmdline

** The other partitions use the standard rootfs and swap mechanisms

** The bootloader line actually installs the bootloader

** An invisible post() section adds all fs info to /etc/fstab
   # copy swap settings to etc/tab
   echo swap settings to ROOTFS/etc/tab


====
live-pcbios.yks:
====

def pre():
    generate rootfs
    generate INITRD_IMAGE rootfs
    generate grub config file

clearpart --all
part	  live-pcbios
bootloader	--location=mbr	--append="console=ttyS0,115200"

** The special partition type live-pcbios encapsulates this:
   copy INITRD_IMAGE rootfs (cpio.gz) into ./hddimg/initrd
   install bzImage into ./hddimg/vmlinuz
   copy rootfs.ext3 to ./hddimg/rootfs.img
   copy syslinux.cfg, menu, ldlinux.sys in to ./hddimg/
   mkdosfs, creating .hddimg file basically
   mcopy ./hddimg* into .hddimg
   run syslinux on the .hddimg

** We assume that the base rootfs etc, needed for assembling the image
   is already available - in the Yocto build system, this is
   accomplished by do_rootfs dependencies and the like.  An external
   script can do this or any arbitrarily complex setup in a pre()
   function designed for that purpose.  Note that all of the pre()
   code could itself be encapsulated in a named object/'command'.


====
live-efi.yks:
====

def pre():
    generate rootfs
    generate INITRD_IMAGE rootfs
    generate EFI config file

clearpart --all
part	  live-efi

** The special partition type live-efi encapsulates this:
   copy INITRD_IMAGE rootfs (cpio.gz) into ./hddimg/initrd
   installs bzImage into ./hddimg/vmlinuz
   copies rootfs.ext3 to ./hddimg/rootfs.img
   create /EFI/BOOT, copy bootxxx.efi, grub.cfg in to ./EFI/BOOT/
   mkdosfs, creating .hddimg file basically
   mcopy ./hddimg* into .hddimg

** We assume that the base rootfs etc, needed for assembling the image
   is already available - in the Yocto build system, this is
   accomplished by do_rootfs dependencies and the like.  An external
   script can do this or any arbitrarily complex setup in a pre()
   function designed for that purpose.  Note that all of the pre()
   code could itself be encapsulated in a named object/'command'.


====
fsl-imx31pdk-uboot.yks:
====

def pre():
    generate rootfs
    generate uboot.mxsboot-sdcard:
      "mxsboot sd ${DEPLOY_DIR_IMAGE}/u-boot-${MACHINE}.${UBOOT_SUFFIX} \
       ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.uboot.mxsboot-sdcard"
    calculate and assign BOOTSPACE, ROOTFS_SIZE

clearpart --all	--sparse
part	  --fstype=none		--offset=1M	--size=1M
part	  --fstype=none		--offset=2M	--size=BOOT_SPACE
part	  --fstype=none		--type=0x53	--size=ROOTFS_SIZE
bootloader	--location=partition	--append="console=ttyS0,115200"

def post():
    dd rootfs.uboot.mxsboot-sdcard to partition 0
    copy kernel to partition 1
    set up devicetree
    dd ROOTFS to partition 2

** Note that the mxs processor family requires a special partition
   type (0x53), which is set in the third part command.

** Note that this pre() method sets some variables picked up by the
   part command.

** We assume that the base rootfs etc, needed for assembling the image
   is already available - in the Yocto build system, this is
   accomplished by do_rootfs dependencies and the like.  An external
   script can do this or any arbitrarily complex setup in a pre()
   function designed for that purpose.  Note that all of the pre()
   code could itself be encapsulated in a named object/'command'.  In
   this case, we need to generate the special uboot.mxsboot-sdcard
   first, which will be copied into one of partitions created in the
   next step.

** In this case, we have only raw partitions and rely on the code in
   post() to handle the assembly.  Note that all of the post() code
   could itself be encapsulated in a named object/'command'.
Comment 5 venkata ramana g 2013-08-02 12:03:14 UTC
From the comment above, I understand that, you are suggesting to adapt the syntax format of Fedora kickstart for export filesystem and you are also mapping the commands to corresponding python files.

We also have implemented a generic configuration model, and a method to map any configuration command to correspoding configuration function with similar configuration requirements. I did put up a link to the wiki documentation that I did write earlier in the comments above. I can also send the patches for the same. I would highly appreciate if I can get some pointers on the same from you.

I am trying to suggest a complete configuration model within which any configuration requirement,like the export filesystem requirement as suggested by you can seamlessly be plugged in. I did also suggest a config syntax (based on simple BNF). 
Suggested this Config BNF syntax to allow even a nested config commands to be possible.
Ex:
diskimage( 
        partitions (
                part( /boot, fstype=fat16, type=syslinux, size=fit)
                part( /,     fstype=ext3,  type=syslinux, size=grow(0.5))
		   )
        bootloader (location=partition, append="console=ttyS0,115200")
	)

However We can easily change our parser to follow the same confi
g syntax as you suggested (adopt from fedora), or support both.

I suggest to have a discussion and so that we do not end up in solving same problems.
Comment 6 venkata ramana g 2013-08-05 05:52:39 UTC
Sorry missed the commas in example
diskimage( 
        partitions (
                part( /boot, fstype=fat16, type=syslinux, size=fit),
                part( /,     fstype=ext3,  type=syslinux, size=grow(0.5))
		   ),
        bootloader (location=partition, append="console=ttyS0,115200")
	)
Comment 7 Tom Zanussi 2013-08-16 20:27:30 UTC
On the one hand, I'm not too particular about the syntax itself - I view it and the parser as a relatively minor and superficial aspect of the overall tool, and something that could be easily replaceable with something else.

On the other hand, though, I wonder why we need to invent a new syntax for this problem, which seems to be a very similar problem to that already solved to a large degree by kickstart and the kickstart syntax.  Also, to me, the kickstart syntax also seems more human-readable and writeable, and is somewhat familiar to people already.

Actually, my proposal of using kickstart as a starting point was only meant to cover the requirements of partition description and tooling using a human-readable/writeable format, and it happened to be the closest thing I found and that I think covers it well enough to base on.  But after after looking at the binary configuration tool writeup, I notice that kickstart also seems to cover other areas defined by the configuration tool writeup:

- package selection:
  http://fedoraproject.org/wiki/Anaconda/Kickstart#Chapter_3._Package_Selection

- network:
  http://fedoraproject.org/wiki/Anaconda/Kickstart#network

- user/passwords:
  http://fedoraproject.org/wiki/Anaconda/Kickstart#user

- services:
  http://fedoraproject.org/wiki/Anaconda/Kickstart#services

and a whole host of others...

Which is not to say that it should be used in place of your proposal - I'm just curious if it was considered and if so what the advantages of your syntax and configuration model would be.

I do appreciate that one other aspect of your model is that you'd like to have per-package configuration, and I do kind of like what you've done there, but I'm wondering if this is the right place to discuss that aspect - it seems somewhat similar to the buildroot menuconfig mechanism, which in Yocto was essentially implemented by Hob.  And in any case, I'm not really seeing what that would allow you to accomplish beyond what you could do by implementing the various configuration categories exemplified by kickstart.

Again, I'm not saying one way or another is better, but trying to ask questions to evaluate how this partitioning proposal of mine fits (or doesn't, or do we care) with something larger, and what form that something larger would best take.

I will say, however, that I don't think we need to settle on the answers before starting on the implementation of this bug - as mentioned, it should be implemented in such a way that replacing the syntax would be a minor job.
Comment 8 Tom Zanussi 2013-08-23 18:51:39 UTC
Having independently analyzed the problem and come to the conclusion that kickstart might make sense, I did a quick search of projects that used pykickstart, the python library for parsing all the various variants of the language and as a result realized that another project that used it was Meego in its 'Meego Image Creator' (MIC) tool for creating Meego images.

MIC itself was based on Redhat LiveCD etc, and has been inherited by the Tizen project, which still calls it 'mic' instead of, say, 'tic' as you might expect.

Also it seems Tizen has erased the original MIC git history and shows only 3 commits, and one of those just creates an empty repository, which doesn't exactly inspire confidence.

Still, I'll be looking at the code to see if I can learn something from it or even use portions, but in any case it's nice to see some independent validation of this direction by another product (ironically one of our own but still)...
Comment 9 Tom Zanussi 2013-09-27 02:24:55 UTC
Patchset posted to OE ml.
Comment 10 Tom Zanussi 2013-10-04 22:25:44 UTC
External 'wic' command implemented:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=75c143a7aef46ecea07cf33edd2b1a0192e10149

In 1.6, this will be integrated into the build system - a new bugzilla entry will be created to track that work.
Comment 11 Ionut Chisanovici 2013-10-14 08:22:55 UTC
Hi,

I tried to test the wic command and the resulting image which was built for NUC but I didn't succeed to boot the image on NUC.

Below the steps that I've done:

On Fedora 19:

1. clean the environment
2. Edit and add BBLAYERS in bblayers.conf for the NUC BSP
BBLAYERS ?= " \
  /home/ionut/work/poky/meta \
  /home/ionut/work/poky/meta-yocto \
  /home/ionut/work/poky/meta-yocto-bsp \
  /home/ionut/work/poky/meta-intel \
  /home/ionut/work/poky/meta-intel/meta-nuc \
  "
3. Edit and add in local.conf: 
MACHINE ?= "nuc"

4. bitbake core-image-minimal
5. wic create mkefidisk -e core-image-minimal

The output:

[ionut@ionut-fedora build]$ wic create mkefidisk -e core-image-minimal
Checking basic build environment...
Done.

Creating image(s)...

Info: The new image(s) can be found here:
  /var/tmp/wic/build/mkefidisk-201310141002-sda.direct

The following build artifacts were used to create the image(s):
  ROOTFS_DIR:      /home/ionut/work/poky/build/tmp/work/nuc-poky-linux/core-image-minimal/1.0-r0/rootfs
  BOOTIMG_DIR:     /home/ionut/work/poky/build/tmp/work/nuc-poky-linux/core-image-minimal/1.0-r0/core-image-minimal-1.0/hddimg
  KERNEL_DIR:      /home/ionut/work/poky/build/tmp/sysroots/nuc/usr/src/kernel
  NATIVE_SYSROOT:  /home/ionut/work/poky/build/tmp/sysroots/x86_64-linux


The image(s) were created using OE kickstart file:
  /home/ionut/work/poky/scripts/lib/image/canned-wks/mkefidisk.wks

6. sudo dd if=/var/tmp/wic/build/mkefidisk-201310141002-sda.direct of=/dev/sdh (my usb device)
7. sync

I wasn't able to boot the image from NUC.

On Ubunt 13.10:

1. clean the environment
2. Edit and add BBLAYERS in bblayers.conf for the NUC BSP
BBLAYERS ?= " \
  /home/ionut/work/poky/meta \
  /home/ionut/work/poky/meta-yocto \
  /home/ionut/work/poky/meta-yocto-bsp \
  /home/ionut/work/poky/meta-intel \
  /home/ionut/work/poky/meta-intel/meta-nuc \
  "
3. Edit and add in local.conf: 
MACHINE ?= "nuc"

4. bitbake core-image-minimal
5. wic create mkefidisk -e core-image-minimal

The output:

 ionut@ionut-ubuntu:~/work/poky/build$ wic create directdisk -e core-image-minimal
Traceback (most recent call last):
File "/home/ionut/work/poky/scripts/wic", line 44, in <module>
from image.engine import *
File "/home/ionut/work/poky/scripts/lib/image/engine.py", line 40, in <module>
from mic import msger, creator
File "/home/ionut/work/poky/scripts/lib/mic/creator.py", line 21, in <module>
from mic import msger, rt_util
File "/home/ionut/work/poky/scripts/lib/mic/rt_util.py", line 26, in <module>
from mic import bootstrap, msger
File "/home/ionut/work/poky/scripts/lib/mic/bootstrap.py", line 24, in <module>
import rpm
ImportError: No module named rpm

6. ionut@ionut-ubuntu: sudo apt-get install python-rpm

7. ionut@ionut-ubuntu:~/work/poky/build$ wic create mkefidisk -e core-image-minimal
Checking basic build environment...
Done.

Creating image(s)...

Traceback (most recent call last):
  File "/home/ionut/work/poky/scripts/wic", line 179, in <module>
    ret = main()
  File "/home/ionut/work/poky/scripts/wic", line 174, in main
    invoke_subcommand(args, parser, wic_help_usage, subcommands)
  File "/home/ionut/work/poky/scripts/lib/image/help.py", line 73, in invoke_subcommand
    subcommands.get(args[0], subcommand_error)[0](args[1:], usage)
  File "/home/ionut/work/poky/scripts/wic", line 95, in wic_create_subcommand
    find_artifacts(options.image_name)
  File "/home/ionut/work/poky/scripts/lib/image/engine.py", line 102, in find_artifacts
    return (rootfs_dir, kernel_dir, hdddir, staging_data_dir, native_sysroot)
UnboundLocalError: local variable 'hdddir' referenced before assignment
Comment 12 Tom Zanussi 2013-10-14 13:43:33 UTC
It looks like it's not finding the boot artifacts for EFI which is what the mkefidisk script is looking for.   The nuc BSP apparently doesn't generate EFI artifacts, so that's why it's not finding them - wic can only use the artifacts its pointed to - the -e option tries to make easy for the user, but if the image its pointed to isn't appropriate for the script, it can't do much about that.

So you can either manually point wic at appropriate EFI artifacts, or try the directdisk script, which I've successfully generated and booted an image for sugarbay, which should be close to what you're trying with the nuc.
Comment 13 Tom Zanussi 2013-10-14 18:49:26 UTC
Closing this again since there the problem appears to be user error in one case and in the other, I'm able to create and successfully boot a directdisk image on the sugarbay, which is the same use case as the directdisk nuc test.

So the basic functionality of this feature is there - if there's a problem with nuc or a specific use case, please open a new bug for those - we don't want to be reopening a high bug for things like this.
Comment 14 David Nyström 2013-10-15 14:36:31 UTC
(In reply to comment #8)
> Having independently analyzed the problem and come to the conclusion that
> kickstart might make sense, I did a quick search of projects that used
> pykickstart, the python library for parsing all the various variants of the
> language and as a result realized that another project that used it was
> Meego in its 'Meego Image Creator' (MIC) tool for creating Meego images.
> 
> MIC itself was based on Redhat LiveCD etc, and has been inherited by the
> Tizen project, which still calls it 'mic' instead of, say, 'tic' as you
> might expect.
> 
> Also it seems Tizen has erased the original MIC git history and shows only 3
> commits, and one of those just creates an empty repository, which doesn't
> exactly inspire confidence.
> 

mic development repo at https://github.com/01org/mic seems pretty active ?
Comment 15 Tom Zanussi 2013-10-15 14:46:47 UTC
(In reply to comment #14)
> (In reply to comment #8)
> > Having independently analyzed the problem and come to the conclusion that
> > kickstart might make sense, I did a quick search of projects that used
> > pykickstart, the python library for parsing all the various variants of the
> > language and as a result realized that another project that used it was
> > Meego in its 'Meego Image Creator' (MIC) tool for creating Meego images.
> > 
> > MIC itself was based on Redhat LiveCD etc, and has been inherited by the
> > Tizen project, which still calls it 'mic' instead of, say, 'tic' as you
> > might expect.
> > 
> > Also it seems Tizen has erased the original MIC git history and shows only 3
> > commits, and one of those just creates an empty repository, which doesn't
> > exactly inspire confidence.
> > 
> 
> mic development repo at https://github.com/01org/mic seems pretty active ?

Yes, I realized when I wrote that that I was looking at the wrong branch.
Comment 16 David Nyström 2013-10-15 15:28:37 UTC
(In reply to comment #15)
> (In reply to comment #14)
> > (In reply to comment #8)
> > 
> > mic development repo at https://github.com/01org/mic seems pretty active ?
> 
> Yes, I realized when I wrote that that I was looking at the wrong branch.

Does this still mean that we intent to fork bits of the MIC for OE image creation ?
Perhaps we should consider removing the IMAGE bb recipes allround, add needed features to the MIC, and let OE images be defined via the kickstart syntax ?
Comment 17 Darren Hart 2013-10-15 15:39:58 UTC
We should of course reuse anything we can. Tom, in looking at the "right" branch, do you feel there is opportunity to collaborate with the Tizen folks on mic?

David, as to removing the image classes, I think it would be more likely that the image classes get rewritten to use mic, if anything. People still expect to be able to assemble image via bitbake. However, that isn't to say you couldn't use mic outside of the bitbake environment to assemble images from the deployed kernel, bootloader, and rootfs.

If these needs much more discussion, we should take it to the mailing list and open new bugs once a conclusion has been reached.
Comment 18 Tom Zanussi 2013-10-15 16:48:55 UTC
(In reply to comment #16)
> (In reply to comment #15)
> > (In reply to comment #14)
> > > (In reply to comment #8)
> > > 
> > > mic development repo at https://github.com/01org/mic seems pretty active ?
> > 
> > Yes, I realized when I wrote that that I was looking at the wrong branch.
> 
> Does this still mean that we intent to fork bits of the MIC for OE image
> creation ?
> Perhaps we should consider removing the IMAGE bb recipes allround, add
> needed features to the MIC, and let OE images be defined via the kickstart
> syntax ?

Obviously I'd rather not long-term fork, but on the other hand it depends on how much of mic is actually useful to us in OE.  At the moment, it's a means to an end i.e. reusing established syntax and parsing framework for two commands, partition and bootloader.  I originally intended on just hacking those pieces out and leaving the rest behind, but decided to keep everything intact in case we might want a deeper integration that makes use of all the other things in mic.  As it stands, the parts I added aren't really making much use of anything beyond the parsing framework.

If it makes sense for us to reuse other parts, for example the package selection and configuration of assembled images, then yes, I'd say we should work upstream and collaborate with mic.

Regardless, whatever we do, I think it does make sense to make the current OE image creation be driven by the kickstart syntax, considering that it offers a superset of functionality and is customizable.  How exactly the customizabe image aspect is exposed to users from within the build framework is an open question at this point however.
Comment 19 Tom Zanussi 2013-10-15 16:56:39 UTC
(In reply to comment #17)
> We should of course reuse anything we can. Tom, in looking at the "right"
> branch, do you feel there is opportunity to collaborate with the Tizen folks
> on mic?
> 

As I said in the previous post, it depends on how much of mic we want to use - if it's more than just the current partition and bootloader commands, then probably, otherwise we probably don't care about upstream since at this point we're just reusing the parser and command framework, with mostly our own content focused on OE artifacts.

> David, as to removing the image classes, I think it would be more likely
> that the image classes get rewritten to use mic, if anything. People still
> expect to be able to assemble image via bitbake. However, that isn't to say
> you couldn't use mic outside of the bitbake environment to assemble images
> from the deployed kernel, bootloader, and rootfs.
> 

We can keep the current image classes and reimplement them on top of the kickstart files, and provide a means of customization by virtue of the fact that the kickstart files are available, or we could create something completely different in OE, or whatever we want to do with wic functionality.

Regardless of what we do in OE, we need to keep the external form - it was/is an explicit requirement.

> If these needs much more discussion, we should take it to the mailing list
> and open new bugs once a conclusion has been reached.
Comment 20 Ionut Chisanovici 2013-10-16 09:59:24 UTC
Based on what I have tested and what I have discussed together with Tom, Darren and Nitin (Darren and Nitin they tested this as well), the following conclusion was taken: the wic image can successfully build an image (directdisk or mkefisidk) but I wasn't able to boot on any of the following BSP's: NUC, FRI2 and Sugarbay using the core-image-minimal directdisk image.
As stated above, Darren and Nitin tested this as well, and they also couldn't booted their BSP's. They faced the same issues as I did: "The generated image is booting to grub prompt on minnow. But after choosing the grub option "boot", the screen goes blank, and nothing happens. I see the vmlinuz"

Steps that were taken in order to create the image for the NUC and Sugarbay using wic command are:

1. clean the env

2. source the env

3. Edit ${BUILD-DIR}/conf/bblayers.conf and add the meta-intel and meta-nuc or meta-sugarbay:
BBLAYERS ?= " \
  /home/ionut/work/poky/meta \
  /home/ionut/work/poky/meta-yocto \
  /home/ionut/work/poky/meta-yocto-bsp \
  /home/ionut/work/poky/meta-intel \
  /home/ionut/work/poky/meta-intel/meta-nuc \
"
or 
BBLAYERS ?= " \
  /home/ionut/work/poky/meta \
  /home/ionut/work/poky/meta-yocto \
  /home/ionut/work/poky/meta-yocto-bsp \
  /home/ionut/work/poky/meta-intel \
  /home/ionut/work/poky/meta-intel/meta-sugarbay \
"

4. Edit ${BUILD-DIR}/conf/local.conf and set the target machine to nuc or sugrabay:

MACHINE ?= "nuc"
or
MACHINE ?= "sugarbay"

5. bitbake core-image-minimal

6a. wic create directdisk -e core-image-minimal

6b. wic create /home/ionut/work/poky/scripts/lib/image/canned-wks/directdisk.wks -o /var/tmp/wic --rootfs-dir /home/ionut/work/poky/build/tmp/work/nuc-poky-linux/core-image-minimal/1.0-r0/rootfs --bootimg-dir /home/ionut/work/poky/build/tmp/sysroots/nuc/usr/share --kernel-dir /home/ionut/work/poky/build/tmp/sysroots/nuc/usr/src/kernel --native-sysroot /home/ionut/work/poky/build/tmp/sysroots/x86_64-linux

Both commands were used to generate the image, same results when tryied to boot the BSP's.

When using the wic command on Ubuntu the python-rpm and python-urlgrabber are required in order for the command to work. This should also be documented somehow.

7. sudo dd if=/var/tmp/wic/build/directdisk-201310151208-sda.direct of=/dev/sde

8. For NUC I even remove the microHDD so I can have only the usb media connected and the FS from it.

8. Booting the NUC or Sugarbay using usb media containing the images from step 6a or 6b, the devices are starting to boot but the screen goes black and nothing happens.

Conclusion: The wic image is successfully creating the images but the images are not usable for the moment, the BSP's cannot boot from them. In my opinion the feature is not resolved and should be reopened.
Comment 21 Alexandru Georgescu 2013-10-16 10:57:48 UTC
Based on IonutC's comments the images built with wic don't boot, so the feature needs to stay reopened.
Comment 22 Tom Zanussi 2013-10-16 13:16:28 UTC
Patches were submitted yesterday and have been pulled in.  Please test with poky/master.
Comment 23 Tom Zanussi 2013-10-16 20:30:59 UTC
I've submitted a new patch for the populate-ext bug.  Applied on top of current master (68a41d2afd3392c59f78398913aa3855225cc52d) I was able to build, boot, and log in as root, directdisk images for crownbay and nuc.  I don't have a minnow or anything that we have an efi BSP for, so wasn't able to test the mkefidisk version, though I did generate an image and the ownerships looked fine.

Incidentally I was able to boot nuc even without Darren's lba fix.
Comment 24 Tom Zanussi 2013-10-16 20:32:47 UTC
(In reply to comment #23)
> I've submitted a new patch for the populate-ext bug.  Applied on top of
> current master (68a41d2afd3392c59f78398913aa3855225cc52d) I was able to
> build, boot, and log in as root, directdisk images for crownbay and nuc.  I
> don't have a minnow or anything that we have an efi BSP for, so wasn't able
> to test the mkefidisk version, though I did generate an image and the
> ownerships looked fine.
> 
> Incidentally I was able to boot nuc even without Darren's lba fix.

And I should mention for completeness that crownbay also has never had problems with or without the lba fix.
Comment 25 Darren Hart 2013-10-16 22:48:06 UTC
(In reply to comment #24)
> (In reply to comment #23)
> > I've submitted a new patch for the populate-ext bug.  Applied on top of
> > current master (68a41d2afd3392c59f78398913aa3855225cc52d) I was able to
> > build, boot, and log in as root, directdisk images for crownbay and nuc.  I
> > don't have a minnow or anything that we have an efi BSP for, so wasn't able
> > to test the mkefidisk version, though I did generate an image and the
> > ownerships looked fine.
> > 
> > Incidentally I was able to boot nuc even without Darren's lba fix.
> 
> And I should mention for completeness that crownbay also has never had
> problems with or without the lba fix.

Minnow also boots successfully with this fix and login works.
Comment 26 Ionut Chisanovici 2013-10-17 13:42:23 UTC
Still not working.
Today I tested only with my nuc BSP and using the latest commit 529bf977e956175bd8405ebffc88194192e44740.

Things that I've done:
Under my user ionut
1. clean env:

ionut@ionut-ubuntu:~/work$ rm -rf poky
ionut@ionut-ubuntu:~/work$ git clone git://git.yoctoproject.org/poky
Cloning into 'poky'...
remote: Counting objects: 206022, done.
remote: Compressing objects: 100% (52376/52376), done.
remote: Total 206022 (delta 149189), reused 205695 (delta 148862)
Receiving objects: 100% (206022/206022), 96.44 MiB | 246 KiB/s, done.
Resolving deltas: 100% (149189/149189), done.
ionut@ionut-ubuntu:~/work$ cd poky/
ionut@ionut-ubuntu:~/work/poky$ git clone git://git.yoctoproject.org/meta-intel
Cloning into 'meta-intel'...
remote: Counting objects: 7587, done.
remote: Compressing objects: 100% (2616/2616), done.
remote: Total 7587 (delta 4154), reused 7439 (delta 4006)
Receiving objects: 100% (7587/7587), 2.34 MiB | 515 KiB/s, done.
Resolving deltas: 100% (4154/4154), done.
ionut@ionut-ubuntu:~/work/poky$ git checkout master
Already on 'master'
ionut@ionut-ubuntu:~/work/poky$ git reset --hard master
HEAD is now at 529bf97 update-rcd.bbclass: fix host/target test
ionut@ionut-ubuntu:~/work/poky$ git pull
Already up-to-date.
ionut@ionut-ubuntu:~/work/poky$ cd meta-intel/
ionut@ionut-ubuntu:~/work/poky/meta-intel$ git checkout master
Already on 'master'
ionut@ionut-ubuntu:~/work/poky/meta-intel$ git reset --hard master
HEAD is now at 27d2663 emgd-driver-bin*: Modify LICENSE to reflect true license
ionut@ionut-ubuntu:~/work/poky/meta-intel$ git pull
Already up-to-date.
2. source to "bnuc" build dir;
3. rm -rf /var/tmp/wic/build/* (clean the location of the resulting image)
4. edit and update ${BUILD-DIR}/conf/local.conf and add MACHINE ? = "nuc"
5. edit and update ${BUILD-DIR}/conf/bblayers.conf and add
BBLAYERS ?= " \
  /home/ionut/work/poky/meta \
  /home/ionut/work/poky/meta-yocto \
  /home/ionut/work/poky/meta-yocto-bsp \
  /home/ionut/work/poky/meta-intel \
  /home/ionut/work/poky/meta-intel/meta-nuc \
  "
6. bitbake core-image-minimal

Build Configuration:
BB_VERSION        = "1.21.0"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "Ubuntu-13.04"
TARGET_SYS        = "x86_64-poky-linux"
MACHINE           = "nuc"
DISTRO            = "poky"
DISTRO_VERSION    = "1.5+snapshot-20131017"
TUNE_FEATURES     = "m64"
TARGET_FPU        = ""
meta              
meta-yocto        
meta-yocto-bsp    = "master:529bf977e956175bd8405ebffc88194192e44740"
meta-intel        
meta-nuc          = "master:27d2663112bbc6e2cf075bdbbd6e68e1a15476d6"

7. wic create /home/ionut/work/poky/scripts/lib/image/canned-wks/directdisk.wks -o /var/tmp/wic --rootfs-dir /home/ionut/work/poky/bnuc/tmp/work/nuc-poky-linux/core-image-minimal/1.0-r0/rootfs --bootimg-dir /home/ionut/work/poky/bnuc/tmp/sysroots/nuc/usr/share --kernel-dir /home/ionut/work/poky/bnuc/tmp/sysroots/nuc/usr/src/kernel --native-sysroot /home/ionut/work/poky/bnuc/tmp/sysroots/x86_64-linux
8. sudo dd if=/var/tmp/wic/build/directdisk-201310171351-sda.direct of=/dev/sdh

Things that I've discovered:

1. My nuc already had an core-image-sato installed on it, on its micro hdd so /dev/sda2 was already configured with the rootfs for the core-image-sato image.
2. Cating the /boot/syslinux.cfg file from the usb media where the wic image was dd, I saw that the root partition should also be on /dev/sda2
ionut@ionut-ubuntu:~/work/poky$ cat /media/ionut/boot/syslinux.cfg
PROMPT 0
TIMEOUT 0

ALLOWOPTIONS 1
SERIAL 0 115200

DEFAULT boot
LABEL boot
KERNEL /vmlinuz
APPEND label=boot root=/dev/sda2 rootwait rootfstype=ext3 video=vesafb vga=0x318 console=tty0
3. When booting the nuc my usb media partitions (wic image partitions) are mounted under /dev/sdb1 (the rootfs) and /dev/sdb2 (boot) so the rootfs is loaded directly from the already installed core-image-sato from nuc disk.
4. Physically removed the micro hdd disk from nuc and booting only with usb media containing the wic image. Now the boot process is successfully performed, the rootfs (is mounted) is the one from the usb media (/dev/sda2) but it doesn't allowed me to login due to the following error: "login: can't create groups: Operation not permitted"
5. Doing ll (ls -l) on the rootfs on the usb media (/dev/sda2), I saw that the ownership of the folders is not correct, the folders are created under my user ionut and not root:
ionut@ionut-ubuntu:~/work/poky$ ls -la /media/ionut/27e41918-c17a-4318-a919-3f85e7bf1575/
total 31
drwxr-xr-x  17 root  root   1024 oct 17  2013 .
drwxr-x---+  4 root  root   4096 oct 17 14:26 ..
drwxr-xr-x   2 ionut ionut  1024 oct 17 14:21 bin
drwxr-xr-x   2 ionut ionut  1024 oct 17 14:21 boot
drwxr-xr-x   2 ionut ionut  1024 oct 17 14:21 dev
drwxr-xr-x  17 ionut ionut  1024 oct 17  2013 etc
drwxr-sr-x   3 ionut ionut  1024 oct 17 14:21 home
-rw-r--r--   1 ionut ionut     0 oct 17 14:21 init
drwxr-xr-x   4 ionut ionut  1024 oct 17 14:21 lib
drwx------   2 root  root  12288 oct 17 14:21 lost+found
drwxr-xr-x   2 ionut ionut  1024 oct 17 14:21 media
drwxr-xr-x   2 ionut ionut  1024 oct 17 14:21 mnt
drwxr-xr-x   2 ionut ionut  1024 oct 17 14:21 proc
drwxr-xr-x   2 ionut ionut  1024 oct 17 14:21 run
drwxr-xr-x   2 ionut ionut  1024 oct 17 14:21 sbin
drwxr-xr-x   2 ionut ionut  1024 oct 17 14:21 sys
lrwxrwxrwx   1 root  root      8 oct 17  2013 tmp -> /var/tmp
drwxr-xr-x   8 ionut ionut  1024 oct 17 14:21 usr
drwxr-xr-x   7 ionut ionut  1024 oct 17 14:21 var

I hope this will help.

Ionut
Comment 27 Tom Zanussi 2013-10-17 13:53:45 UTC
The following patch, posted yesterday, but not yet in master, will fix the 'login: can't create groups: Operation not permitted' problem:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=tzanussi/wic-fixes-1&id=fd09f894b40dadb8bc8b267a26dbf52e22a53d66

Also, what have you done differently this time that allows you to now boot the nuc - yesterday, you only had a black screen?
Comment 28 Ionut Chisanovici 2013-10-17 14:10:44 UTC
For the black screens yesterday, the image was created with "wic create directdisk -e core-image-minimal" and then dd to usb media.
Today I see that using the above command the nuc is also booting from it.
Comment 29 Tom Zanussi 2013-10-17 15:18:30 UTC
For your problem with booting from sda instead of sdb, which what you want to do, you can modify the .wks file to have it match your hardware.  The easiest way is to just make a copy of the canned directdisk script and change a couple lines:

$ cd scripts/lib/image/canned-wks
$ cp cp directdisk.wks directdisksdb.wks

edit direcdisksdb.wks and change '--ondisk sda' to '--ondisk sdb' in the 'part' lines i.e. to:

part /boot --source bootimg --ondisk sdb --fstype=msdos --label boot --active --align 1024
part / --source rootfs --ondisk sdb --fstype=ext3 --label platform --align 1024

Now used directdisksdb to create the image instead of directdisk

e.g.

$ wic create directdisksdb -e core-image-minimal

Of course, make sure you have the correct machine selected in local.conf, etc, etc

That should give you a boot image that will boot off of sdb.
Comment 30 Ionut Chisanovici 2013-10-18 11:39:11 UTC
Ok, applying the changes you made on http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=tzanussi/wic-fixes-1&id=fd09f894b40dadb8bc8b267a26dbf52e22a53d66 and changing the sda to sdb in directdisk wks file, I have successfully booted the wic image on nuc and I was able to login using root, the wic image is fully functional now.

I will test the efi image on my FRI2 BSP and if that will be ok I will set this feature on verified.

Later on today my update about the FRI2 and wic efi image.
Comment 31 Ionut Chisanovici 2013-10-18 13:57:20 UTC
Tested with FRI2 and the wic efi images but it's not working, it's not booting.

Here I have 2 cases for the FRI2 boot menu:
1. Booting directly from EFI USB media containing the wic image which sent me directly in a GRUB command line and nothing else, no boot:
grub>
2. Booting directly from USB media (not with the EFI option) which sent me in a black screen, no boot.

Also, I saw that when I used "wic create mkefidisksdb -e core-image-minimal", the /EFI/BOOT/ folder from the wic image doesn't contain any .efi file. Is this normal ?
Comment 32 Tom Zanussi 2013-10-18 14:14:52 UTC
(In reply to comment #31)
> Tested with FRI2 and the wic efi images but it's not working, it's not
> booting.
> 
> Here I have 2 cases for the FRI2 boot menu:
> 1. Booting directly from EFI USB media containing the wic image which sent
> me directly in a GRUB command line and nothing else, no boot:
> grub>
> 2. Booting directly from USB media (not with the EFI option) which sent me
> in a black screen, no boot.
> 
> Also, I saw that when I used "wic create mkefidisksdb -e
> core-image-minimal", the /EFI/BOOT/ folder from the wic image doesn't
> contain any .efi file. Is this normal ?

No, it's not normal, can you list what's in /var/tmp/wic/build/hdd/boot/EFI/BOOT
 and what's in BOOTIMG_DIR/EFI/BOOT.  Below is what I have:
 
[trz@empanada build]$ wic create mkefidisksdb -e core-image-minimal
Checking basic build environment...
Done.

Creating image(s)...

Info: The new image(s) can be found here:
  /var/tmp/wic/build/mkefidisksdb-201310180907-sdb.direct

The following build artifacts were used to create the image(s):
  ROOTFS_DIR:      /home/trz/yocto/yocto-image/build/tmp/work/fri2-poky-linux/core-image-minimal/1.0-r0/rootfs
  BOOTIMG_DIR:     /home/trz/yocto/yocto-image/build/tmp/work/fri2-poky-linux/core-image-minimal/1.0-r0/core-image-minimal-1.0/hddimg
  KERNEL_DIR:      /home/trz/yocto/yocto-image/build/tmp/sysroots/fri2/usr/src/kernel
  NATIVE_SYSROOT:  /home/trz/yocto/yocto-image/build/tmp/sysroots/x86_64-linux


The image(s) were created using OE kickstart file:
  /home/trz/yocto/yocto-image/scripts/lib/image/canned-wks/mkefidisksdb.wks



$ ls -al /var/tmp/wic/build/hdd/boot/EFI/BOOT
total 412
drwxr-xr-x 2 trz trz   4096 Oct 18 09:07 .
drwxr-xr-x 3 trz trz   4096 Oct 18 09:07 ..
-rw-r--r-- 1 trz trz 409600 Oct 18 09:07 bootia32.efi
-rw-rw-r-- 1 trz trz    246 Oct 18 09:07 grub.cfg

$ ls -al /home/trz/yocto/yocto-image/build/tmp/work/fri2-poky-linux/core-image-minimal/1.0-r0/core-image-minimal-1.0/hddimg/EFI/BOOT
total 412
drwxr-xr-x 2 trz trz   4096 Oct 17 16:57 .
drwxr-xr-x 3 trz trz   4096 Oct 17 16:57 ..
-rw-r--r-- 1 trz trz 409600 Oct 17 16:57 bootia32.efi
-rw-r--r-- 1 trz trz    485 Oct 17 16:57 grub.cfg
Comment 33 Ionut Chisanovici 2013-10-18 14:19:32 UTC
Same here:

ionut@ionut-ubuntu:~/work/poky/bfri$ ls -la /var/tmp/wic/build/hdd/boot/EFI/BOOT/
total 412
drwxr-xr-x 2 ionut ionut   4096 oct 18 16:30 .
drwxr-xr-x 3 ionut ionut   4096 oct 18 16:30 ..
-rw-r--r-- 1 ionut ionut 409088 oct 18 16:30 bootia32.efi
-rw-rw-r-- 1 ionut ionut    246 oct 18 16:30 grub.cfg
ionut@ionut-ubuntu:~/work/poky/bfri$ ls -la tmp/work/fri2-poky-linux/core-image-minimal
core-image-minimal/           core-image-minimal-initramfs/ 
ionut@ionut-ubuntu:~/work/poky/bfri$ ls -la tmp/work/fri2-poky-linux/core-image-minimal/1.0-r0/core-image-minimal-1.0/hddimg/EFI/BOOT/
bootia32.efi  grub.cfg      
ionut@ionut-ubuntu:~/work/poky/bfri$ ls -la tmp/work/fri2-poky-linux/core-image-minimal/1.0-r0/core-image-minimal-1.0/hddimg/EFI/BOOT/
total 412
drwxr-xr-x 2 ionut ionut   4096 oct 18 15:58 .
drwxr-xr-x 3 ionut ionut   4096 oct 18 15:58 ..
-rw-r--r-- 1 ionut ionut 409088 oct 18 15:58 bootia32.efi
-rw-r--r-- 1 ionut ionut    485 oct 18 15:58 grub.cfg
Comment 34 Tom Zanussi 2013-10-18 14:33:11 UTC
(In reply to comment #32)
> (In reply to comment #31)
> > Tested with FRI2 and the wic efi images but it's not working, it's not
> > booting.
> > 
> > Here I have 2 cases for the FRI2 boot menu:
> > 1. Booting directly from EFI USB media containing the wic image which sent
> > me directly in a GRUB command line and nothing else, no boot:
> > grub>
> > 2. Booting directly from USB media (not with the EFI option) which sent me
> > in a black screen, no boot.
> > 
> > Also, I saw that when I used "wic create mkefidisksdb -e
> > core-image-minimal", the /EFI/BOOT/ folder from the wic image doesn't
> > contain any .efi file. Is this normal ?
> 
> No, it's not normal, can you list what's in
> /var/tmp/wic/build/hdd/boot/EFI/BOOT
>  and what's in BOOTIMG_DIR/EFI/BOOT.  Below is what I have:
>  
> [trz@empanada build]$ wic create mkefidisksdb -e core-image-minimal
> Checking basic build environment...
> Done.
> 
> Creating image(s)...
> 
> Info: The new image(s) can be found here:
>   /var/tmp/wic/build/mkefidisksdb-201310180907-sdb.direct
> 
> The following build artifacts were used to create the image(s):
>   ROOTFS_DIR:     
> /home/trz/yocto/yocto-image/build/tmp/work/fri2-poky-linux/core-image-
> minimal/1.0-r0/rootfs
>   BOOTIMG_DIR:    
> /home/trz/yocto/yocto-image/build/tmp/work/fri2-poky-linux/core-image-
> minimal/1.0-r0/core-image-minimal-1.0/hddimg
>   KERNEL_DIR:     
> /home/trz/yocto/yocto-image/build/tmp/sysroots/fri2/usr/src/kernel
>   NATIVE_SYSROOT: 
> /home/trz/yocto/yocto-image/build/tmp/sysroots/x86_64-linux
> 
> 
> The image(s) were created using OE kickstart file:
>   /home/trz/yocto/yocto-image/scripts/lib/image/canned-wks/mkefidisksdb.wks
> 
> 
> 
> $ ls -al /var/tmp/wic/build/hdd/boot/EFI/BOOT
> total 412
> drwxr-xr-x 2 trz trz   4096 Oct 18 09:07 .
> drwxr-xr-x 3 trz trz   4096 Oct 18 09:07 ..
> -rw-r--r-- 1 trz trz 409600 Oct 18 09:07 bootia32.efi
> -rw-rw-r-- 1 trz trz    246 Oct 18 09:07 grub.cfg
> 
> $ ls -al
> /home/trz/yocto/yocto-image/build/tmp/work/fri2-poky-linux/core-image-
> minimal/1.0-r0/core-image-minimal-1.0/hddimg/EFI/BOOT
> total 412
> drwxr-xr-x 2 trz trz   4096 Oct 17 16:57 .
> drwxr-xr-x 3 trz trz   4096 Oct 17 16:57 ..
> -rw-r--r-- 1 trz trz 409600 Oct 17 16:57 bootia32.efi
> -rw-r--r-- 1 trz trz    485 Oct 17 16:57 grub.cfg

I also see it in the image itself:

ls -al /run/media/trz/efi/EFI/BOOT/
total 406
drwx------ 2 trz trz   2048 Oct 18 09:07 .
drwx------ 3 trz trz   2048 Oct 18 09:07 ..
-rw-r--r-- 1 trz trz 409600 Oct 18 09:07 bootia32.efi
-rw-r--r-- 1 trz trz    246 Oct 18 09:07 grub.cfg

So I don't know why you're not seeing it in your image.
Comment 35 Ionut Chisanovici 2013-10-18 14:48:18 UTC
You are right, the .efi image is there for me also. I have recreated the image and dd to usb media.
I think, there was an dd error or something else ....
Anyway I cannot boot from it, the same issue with the efi usb boot it sends me in the grub cli and no boot and using normal usb boot it sends me in a black screen.
Comment 36 Tom Zanussi 2013-10-18 14:59:34 UTC
OK, I believe Darren and Nitin have been successful booting a wic-generated image using mkefidisk.  I believe it was minnow, but I also remember something in a thread about fri2.  Perhaps they could chime in about that.

As a baseline, have you successfully booted the same build with the output of scripts/contrib/mkefidisk.sh script?
Comment 37 Ionut Chisanovici 2013-10-21 08:09:09 UTC
> As a baseline, have you successfully booted the same build with the output
> of scripts/contrib/mkefidisk.sh script?

I'm unable to use the script, maybe I'm not doing things right:

ionut@ionut-ubuntu:~/work/poky/scripts/contrib$ ./mkefidisk.sh /dev/sdh /home/ionut/work/poky/bfri/tmp/deploy/images/fri2/core-image-minimal-fri2-20131018114232.hddimg /dev/mmcblk10
ERROR: Device /dev/sdh does not exist or is not writable
Usage: mkefidisk.sh DEVICE HDDIMG TARGET_DEVICE
       DEVICE: The device to write the image to, e.g. /dev/sdh
       HDDIMG: The hddimg file to generate the efi disk from
       TARGET_DEVICE: The device the target will boot from, e.g.  /dev/mmcblk0

ionut@ionut-ubuntu:~/work/poky/scripts/contrib$ cat /proc/mounts | grep med
/dev/sdh1 /media/ionut/usb vfat rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,showexec,utf8,flush,errors=remount-ro 0 0
Comment 38 Ionut Chisanovici 2013-10-21 10:06:41 UTC
A sudo in front of the command did the trick :D.
Anyway the same issue ... the grub cli and no boot.
Maybe it is truly a fri2 problem and maybe somebody else should test the EFI image.
Comment 39 Tom Zanussi 2013-10-23 21:33:38 UTC
Created attachment 1574 [details]
boot output from wic-generated minnow boot

I just got a new minnow board and was able to generate and boot an image out of the box using 'wic create mkefisk -e core-image-minimal'.  Details below.

$ wic create mkefidisk -e core-image-minimal
Checking basic build environment...
Done.

Creating image(s)...

Info: The new image(s) can be found here:
  /var/tmp/wic/build/mkefidisk-201310230946-sda.direct

The following build artifacts were used to create the image(s):
  ROOTFS_DIR:      /home/trz/yocto/yocto-image/build/tmp/work/minnow-poky-linux\
/core-image-minimal/1.0-r0/rootfs
  BOOTIMG_DIR:     /home/trz/yocto/yocto-image/build/tmp/work/minnow-poky-linux\
/core-image-minimal/1.0-r0/core-image-minimal-1.0/hddimg
  KERNEL_DIR:      /home/trz/yocto/yocto-image/build/tmp/sysroots/minnow/usr/sr\
c/kernel
  NATIVE_SYSROOT:  /home/trz/yocto/yocto-image/build/tmp/sysroots/x86_64-linux


The image(s) were created using OE kickstart file:
  /home/trz/yocto/yocto-image/scripts/lib/image/canned-wks/mkefidisk.wks

$ sudo dd if=/var/tmp/wic/build/mkefidisk-201310230946-sda.direct of=/dev/sdb
[sudo] password for trz:
182274+0 records in
182274+0 records out
93324288 bytes (93 MB) copied, 14.4777 s, 6.4 MB/s
[trz@empanada ~]$ sudo eject /dev/sdb
Comment 40 Tom Zanussi 2013-10-23 21:38:42 UTC
Created attachment 1575 [details]
boot output from wic-generated nuc boot

I was able to generate and boot an image out of the box using 'wic create directdisksdb -e core-image-minimal'.  Details below.

directdisksdb is simply a copy of directdisk.wks with --ondisk sda changed to --ondisk sdb in the two part lines:

$ cp /home/trz/yocto/yocto-image/scripts/lib/image/canned-wks/directdisk.wks /home/trz/yocto/yocto-image/scripts/lib/image/canned-wks/directdisksdb.wks

Part lines changed to:

  part /boot --source bootimg --ondisk sdb --fstype=msdos --label boot --active\
 --align 1024
  part / --source rootfs --ondisk sdb --fstype=ext3 --label platform --align 10\
24


$ wic create directdisksdb -e core-image-minimalChecking basic build environmen\
t...
Done.

Creating image(s)...

Info: The new image(s) can be found here:
  /var/tmp/wic/build/directdisksdb-201310231131-sdb.direct

The following build artifacts were used to create the image(s):
  ROOTFS_DIR:      /home/trz/yocto/yocto-image/build/tmp/work/nuc-poky-linux/co\
re-image-minimal/1.0-r0/rootfs
  BOOTIMG_DIR:     /home/trz/yocto/yocto-image/build/tmp/sysroots/nuc/usr/share
  KERNEL_DIR:      /home/trz/yocto/yocto-image/build/tmp/sysroots/nuc/usr/src/k\
ernel
  NATIVE_SYSROOT:  /home/trz/yocto/yocto-image/build/tmp/sysroots/x86_64-linux


The image(s) were created using OE kickstart file:
  /home/trz/yocto/yocto-image/scripts/lib/image/canned-wks/directdisksdb.wks
Comment 41 Tom Zanussi 2013-10-23 21:45:08 UTC
(In reply to comment #38)
> A sudo in front of the command did the trick :D.
> Anyway the same issue ... the grub cli and no boot.
> Maybe it is truly a fri2 problem and maybe somebody else should test the EFI
> image.

You should be able to build and boot a fri2 image simply using 'wic create mkefidisk -e core-image-minimal' with the fri2 machine selected in local.conf and dd'ing to USB stick as in the other examples.

You seem to have had problems with bad writes to the USB stick causing problems. You can try:

dd if=/dev/zero of=/dev/sdf bs=1M count=512

Also, remember to eject /dev/xxx before removing.

Again, if you can't even get the mkefidisk.sh script to generate a bootable image, you're not likely to be successful using a wic-genarated image either.
You should just be able to use /dev/sda as the target if you're booting of a USB disk.
Comment 42 Ionut Chisanovici 2013-10-24 06:36:27 UTC
(In reply to comment #41)
> (In reply to comment #38)
> > A sudo in front of the command did the trick :D.
> > Anyway the same issue ... the grub cli and no boot.
> > Maybe it is truly a fri2 problem and maybe somebody else should test the EFI
> > image.
> 
> You should be able to build and boot a fri2 image simply using 'wic create
> mkefidisk -e core-image-minimal' with the fri2 machine selected in
> local.conf and dd'ing to USB stick as in the other examples.
> 
> You seem to have had problems with bad writes to the USB stick causing
> problems. You can try:
> 
> dd if=/dev/zero of=/dev/sdf bs=1M count=512
> 
> Also, remember to eject /dev/xxx before removing.
> 
> Again, if you can't even get the mkefidisk.sh script to generate a bootable
> image, you're not likely to be successful using a wic-genarated image either.
> You should just be able to use /dev/sda as the target if you're booting of a
> USB disk.

Hi Tom,

I don't think that this is a dd problem but a FRI2 bsp problem with the one I have.
I will test today on a new board that we received the other days from Nitin, I think, which has EFI boot capabilities.
Later the outcome.

Ionut C
Comment 43 Ionut Chisanovici 2013-10-24 09:06:59 UTC
Tried to create the mkefidisk image for chiefriver bsp with wic but now I got this message:

ionut@ionut-ubuntu:~/work/poky/bchiefriver$ wic create mkefidisk -e core-image-minimal
Checking basic build environment...
Done.

Creating image(s)...

External command 'getenforce' not found, exiting.
  (Please install 'getenforce' on your host system)

Is that normal ? Why should I have getenforce installed on ubuntu ?
Comment 44 Darren Hart 2013-10-24 11:20:03 UTC
Trying to help out while sitting in on ELC-E presentations... please forgive me if I've missed some context.

Note that the FRI2 has 2 firmware options: Kontron's and the Intel-provided fastboot.

I have never seen the Kontron BIOS successfully boot an EFI image.

from the text above, it appears that Ionut is using the Kontron firmware.
Comment 45 Ionut Chisanovici 2013-10-24 11:23:16 UTC
(In reply to comment #44)
> Trying to help out while sitting in on ELC-E presentations... please forgive
> me if I've missed some context.
> 
> Note that the FRI2 has 2 firmware options: Kontron's and the Intel-provided
> fastboot.
> 
> I have never seen the Kontron BIOS successfully boot an EFI image.
> 
> from the text above, it appears that Ionut is using the Kontron firmware.

Yes, I do, that is the bios installed on my fri2 and I agree with you that my problem with fri bsp is related to bios and not to wic image.
Anyway, that's the reason for which I'm trying now to create and EFI image for chiefriver bsp.
Comment 46 Tom Zanussi 2013-10-24 12:50:06 UTC
(In reply to comment #43)
> Tried to create the mkefidisk image for chiefriver bsp with wic but now I
> got this message:
> 
> ionut@ionut-ubuntu:~/work/poky/bchiefriver$ wic create mkefidisk -e
> core-image-minimal
> Checking basic build environment...
> Done.
> 
> Creating image(s)...
> 
> External command 'getenforce' not found, exiting.
>   (Please install 'getenforce' on your host system)
> 
> Is that normal ? Why should I have getenforce installed on ubuntu ?

It sohuldn't be necessary - here's a patch I created for it yesterday after testing on yet another distro:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=tzanussi/wic-fixes-4&id=45cc6b9ffb8b141807584d2c49bcedaa599f62c5
Comment 47 Tom Zanussi 2013-10-25 03:58:40 UTC
Created attachment 1580 [details]
boot output from wic-generated fri2 boot

And here's a successful fri2 wic image creation and boot.

$ wic create mkefidisk -e core-image-minimal
Checking basic build environment...
Done.

Creating image(s)...

Info: The new image(s) can be found here:
  /var/tmp/wic/build/mkefidisk-201310242232-sda.direct

The following build artifacts were used to create the image(s):
  ROOTFS_DIR:      /home/trz/yocto/yocto-image/build/tmp/work/fri2-poky-linux/core-image-minimal/1.0-r0/rootfs
  BOOTIMG_DIR:     /home/trz/yocto/yocto-image/build/tmp/work/fri2-poky-linux/core-image-minimal/1.0-r0/core-image-minimal-1.0/hddimg
  KERNEL_DIR:      /home/trz/yocto/yocto-image/build/tmp/sysroots/fri2/usr/src/kernel
  NATIVE_SYSROOT:  /home/trz/yocto/yocto-image/build/tmp/sysroots/x86_64-linux


The image(s) were created using OE kickstart file:
  /home/trz/yocto/yocto-image/scripts/lib/image/canned-wks/mkefidisk.wks


The only thing I can say as to why it doesn't work for you is you have bad media (I had to dd /dev/zero as mentioned before), and it takes awhile to boot and show up on the serial console.
Comment 48 Tom Zanussi 2013-10-25 04:04:51 UTC
(In reply to comment #44)
> Trying to help out while sitting in on ELC-E presentations... please forgive
> me if I've missed some context.
> 
> Note that the FRI2 has 2 firmware options: Kontron's and the Intel-provided
> fastboot.
> 
> I have never seen the Kontron BIOS successfully boot an EFI image.
> 
> from the text above, it appears that Ionut is using the Kontron firmware.

I successfully created a wic mkefidisk image and booted it on fri2, see attached boot output.
Comment 49 Tom Zanussi 2013-10-25 04:07:48 UTC
This bug has reached the point of ridiculousness.  I've now successfully booted everything I've tried using mkefidisk and directdisk on fri2, minnow, crownbay, nuc, and sugarbay, while nobody has booted anything except for nuc directdisk.  My office must have some kind of magical powers.
Comment 50 Ionut Chisanovici 2013-10-25 06:43:51 UTC
(In reply to comment #49)
> This bug has reached the point of ridiculousness.  I've now successfully
> booted everything I've tried using mkefidisk and directdisk on fri2, minnow,
> crownbay, nuc, and sugarbay, while nobody has booted anything except for nuc
> directdisk.  My office must have some kind of magical powers.

:) funny :)

Anyway, I'm unable to build a wic efi image for chiefriver due to bitbake which is not creating the EFI/BOOT folder in tmp/work/chiefriver-poky-linux/core-image-minimal/1.0-r0/core-image-minimal-1.0/hddimg/ as I think it should be.

Tried this two times, even with the poky folder deleted and then recloned.

[ionut@ionut-fedora bchiefriver]$ wic create mkefidisksdb -e core-image-minimal
Checking basic build environment...
Done.

Creating image(s)...

Info: The new image(s) can be found here:
  /var/tmp/wic/build/mkefidisksdb-201310250941-sdb.direct

The following build artifacts were used to create the image(s):
  ROOTFS_DIR:      /home/ionut/work/poky/bchiefriver/tmp/work/chiefriver-poky-linux/core-image-minimal/1.0-r0/rootfs
  BOOTIMG_DIR:     /home/ionut/work/poky/bchiefriver/tmp/work/chiefriver-poky-linux/core-image-minimal/1.0-r0/core-image-minimal-1.0/hddimg
  KERNEL_DIR:      /home/ionut/work/poky/bchiefriver/tmp/sysroots/chiefriver/usr/src/kernel
  NATIVE_SYSROOT:  /home/ionut/work/poky/bchiefriver/tmp/sysroots/x86_64-linux


The image(s) were created using OE kickstart file:
  /home/ionut/work/poky/scripts/lib/image/canned-wks/mkefidisksdb.wks
[ionut@ionut-fedora bchiefriver]$ ls -la tmp/work/chiefriver-poky-linux/core-image-minimal/1.0-r0/core-image-minimal-1.0/hddimg
total 21036
drwxr-xr-x 2 ionut ionut    4096 oct 24 17:13 .
drwxrwxr-x 4 ionut ionut    4096 oct 24 17:13 ..
-rw-r--r-- 1 ionut ionut 7128334 oct 24 17:13 initrd
-r--r--r-- 1 ionut ionut   58843 oct 24 17:13 ldlinux.sys
-rw-r--r-- 1 ionut ionut 9835520 oct 24 17:13 rootfs.img
-rw-r--r-- 1 ionut ionut     258 oct 24 17:13 syslinux.cfg
-rw-r--r-- 1 ionut ionut 6093520 oct 24 17:13 vmlinuz

When dd'ing the image there is no .efi file on the usb media in /EFI/BOOT/
Comment 51 Tom Zanussi 2013-10-25 18:20:27 UTC
(In reply to comment #50)
> (In reply to comment #49)
> > This bug has reached the point of ridiculousness.  I've now successfully
> > booted everything I've tried using mkefidisk and directdisk on fri2, minnow,
> > crownbay, nuc, and sugarbay, while nobody has booted anything except for nuc
> > directdisk.  My office must have some kind of magical powers.
> 
> :) funny :)
> 
> Anyway, I'm unable to build a wic efi image for chiefriver due to bitbake
> which is not creating the EFI/BOOT folder in
> tmp/work/chiefriver-poky-linux/core-image-minimal/1.0-r0/core-image-minimal-
> 1.0/hddimg/ as I think it should be.
> 
> Tried this two times, even with the poky folder deleted and then recloned.
> 
> [ionut@ionut-fedora bchiefriver]$ wic create mkefidisksdb -e
> core-image-minimal
> Checking basic build environment...
> Done.
> 
> Creating image(s)...
> 
> Info: The new image(s) can be found here:
>   /var/tmp/wic/build/mkefidisksdb-201310250941-sdb.direct
> 
> The following build artifacts were used to create the image(s):
>   ROOTFS_DIR:     
> /home/ionut/work/poky/bchiefriver/tmp/work/chiefriver-poky-linux/core-image-
> minimal/1.0-r0/rootfs
>   BOOTIMG_DIR:    
> /home/ionut/work/poky/bchiefriver/tmp/work/chiefriver-poky-linux/core-image-
> minimal/1.0-r0/core-image-minimal-1.0/hddimg
>   KERNEL_DIR:     
> /home/ionut/work/poky/bchiefriver/tmp/sysroots/chiefriver/usr/src/kernel
>   NATIVE_SYSROOT: 
> /home/ionut/work/poky/bchiefriver/tmp/sysroots/x86_64-linux
> 
> 
> The image(s) were created using OE kickstart file:
>   /home/ionut/work/poky/scripts/lib/image/canned-wks/mkefidisksdb.wks
> [ionut@ionut-fedora bchiefriver]$ ls -la
> tmp/work/chiefriver-poky-linux/core-image-minimal/1.0-r0/core-image-minimal-
> 1.0/hddimg
> total 21036
> drwxr-xr-x 2 ionut ionut    4096 oct 24 17:13 .
> drwxrwxr-x 4 ionut ionut    4096 oct 24 17:13 ..
> -rw-r--r-- 1 ionut ionut 7128334 oct 24 17:13 initrd
> -r--r--r-- 1 ionut ionut   58843 oct 24 17:13 ldlinux.sys
> -rw-r--r-- 1 ionut ionut 9835520 oct 24 17:13 rootfs.img
> -rw-r--r-- 1 ionut ionut     258 oct 24 17:13 syslinux.cfg
> -rw-r--r-- 1 ionut ionut 6093520 oct 24 17:13 vmlinuz
> 
> When dd'ing the image there is no .efi file on the usb media in /EFI/BOOT/

It seems we're back to the same problem that caused this bug to be reopened - you're creating an mkefidisk image using a BSP that doesn't have efi support built.

Like the nuc, the chiefriver BSP doesn't add efi support, so you can't expect to create and boot an efi image for it, whether it's generated by wic or not.

You might try adding efi support, but then again there might be a reason it doesn't already have efi support.
Comment 52 Darren Hart 2013-11-14 16:07:50 UTC
For Alex G and QA, this feature was included in the 1.5 release and Tom and Ionut have gone around a few times about a particular use case, but as an enhancement, this bug needs to be verified and closed. Can you please do that?

Any issues discovered using wic need to be opened as new Bugs.
Comment 53 Ionut Chisanovici 2013-11-15 08:08:47 UTC
(In reply to comment #52)
> For Alex G and QA, this feature was included in the 1.5 release and Tom and
> Ionut have gone around a few times about a particular use case, but as an
> enhancement, this bug needs to be verified and closed. Can you please do
> that?
> 
> Any issues discovered using wic need to be opened as new Bugs.

Hi,

You are right, this will be closed as the core of the wic command is performing as it should.
Comment 54 Ionut Chisanovici 2013-11-15 08:09:27 UTC
Verified.