Bug 10835

I actually have a WIP to create a bootpart replacement for bootimg, but still need to work out the details of not using the image-live infrastructure.

Please, review: http://lists.openembedded.org/pipermail/openembedded-core/2017-January/131174.html

Sent v2 as v1 conflicts with master due to rss merge.
http://lists.openembedded.org/pipermail/openembedded-core/2017-January/131813.html

sent v3: http://lists.openembedded.org/pipermail/openembedded-core/2017-January/131937.html

removed patches that add tasks
fixed test cases,
rebased on top of patches that make use of artifacts from DEPLOY_DIR_IMAGE

(In reply to comment #4)
> sent v3:
> http://lists.openembedded.org/pipermail/openembedded-core/2017-January/
> 131937.html
> 
> removed patches that add tasks
> fixed test cases,
> rebased on top of patches that make use of artifacts from DEPLOY_DIR_IMAGE

v3 feedback:

- with this approach, a custom source plugin would be needed if user wants to add custom artifacts (meta-intel has one case where an "rmc" database needs to be installed in ESP). 
- use "inherit "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), 'image-wic', '', d)}"" or via IMAGE_CLASSES?
- you'd need to remove 'live' from IMAGE_FSTYPES to see this is working
- what needs to change to avoid building all three bootloaders in wic-tools.bb? 

Overall, I believe the best approach with the artifacts would be that they are prepared outside wic and the output would be the final /boot dir somewhere. It'd also make the source plugins simpler and easier to maintain.

note about build artifacts in general: some update mechanisms, like the one with meta-swupd depend on build artifacts to be present in do_rootfs (and /boot).

> - with this approach, a custom source plugin would be needed if user wants to add custom artifacts (meta-intel has one case where an "rmc" database needs to be installed in ESP). 

How is this different from previous approach?

> - use "inherit "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), 'image-wic', '', d)}"" or via IMAGE_CLASSES?

Can you elaborate a bit more? I'm not sure I understand this.

> - you'd need to remove 'live' from IMAGE_FSTYPES to see this is working

Why?

> - what needs to change to avoid building all three bootloaders in wic-tools.bb? 

So far I don't have a good idea how to solve it. It was a quick urgent change to support recipe-specific sysroots. Now we need to think how to do it better.

One idea could be to expect that all wic dependencies are present in image sysroot. This would work, but can be hard to maintain as if .wks file starts using new plugin user will have to modify list of dependencies in image recipe.

> Overall, I believe the best approach with the artifacts would be that they are prepared outside wic and the output would be the final /boot dir somewhere. It'd also make the source plugins simpler and easier to maintain.

I 100% agree and support this point of view. However, considering that in some cases wic needs to add root device to the kernel command line it's not that simple. Another thing is a legacy - there are a lot of boilerplate code in wic codebase that writes bootloader configuration files. That code need to be removed carefully, without breaking existing functionality.

BTW, I'm going to work on refactoring wic codebase (bug 10619). Feel free to join me there.

(In reply to comment #6)
> > - with this approach, a custom source plugin would be needed if user wants to add custom artifacts (meta-intel has one case where an "rmc" database needs to be installed in ESP). 
> 
> How is this different from previous approach?

scratch my comment above, I was wrong about that.

> 
> > - use "inherit "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), 'image-wic', '', d)}"" or via IMAGE_CLASSES?
> 
> Can you elaborate a bit more? I'm not sure I understand this.

there was an unconditional 'inherit image-wic'. This comment was about making it conditional to check whether wic is used.

> 
> > - you'd need to remove 'live' from IMAGE_FSTYPES to see this is working
> 
> Why?

The goal is to get rid of do_bootimg task as a dependency too. It's pointless to build, e.g., rootfs.img if a raw image is created by wic.

> 
> > - what needs to change to avoid building all three bootloaders in wic-tools.bb? 
> 
> So far I don't have a good idea how to solve it. It was a quick urgent
> change to support recipe-specific sysroots. Now we need to think how to do
> it better.

OK

> One idea could be to expect that all wic dependencies are present in image
> sysroot. This would work, but can be hard to maintain as if .wks file starts
> using new plugin user will have to modify list of dependencies in image
> recipe.

To me this sounds obvious. It'd be the image recipe maintainers to decide what wic needs to build the image.

> 
> > Overall, I believe the best approach with the artifacts would be that they are prepared outside wic and the output would be the final /boot dir somewhere. It'd also make the source plugins simpler and easier to maintain.
> 
> I 100% agree and support this point of view. However, considering that in
> some cases wic needs to add root device to the kernel command line it's not

Could it modify the existing bootloader configs?


> that simple. Another thing is a legacy - there are a lot of boilerplate code
> in wic codebase that writes bootloader configuration files. That code need
> to be removed carefully, without breaking existing functionality.
> 
> BTW, I'm going to work on refactoring wic codebase (bug 10619). Feel free to
> join me there.

I'll have a look.

> > > - use "inherit "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), 'image-wic', '', d)}"" or via IMAGE_CLASSES?
> > 
> > Can you elaborate a bit more? I'm not sure I understand this.
> 
> there was an unconditional 'inherit image-wic'. This comment was about
> making it conditional to check whether wic is used.
>

I tried this and then decided not to go this way as it can cause not easily understandable errors when user would want to build wic image without adding wic to IMAGE_TYPES, e.g. bitbake sic-image-minimal would produce something like that.
 
> > > - you'd need to remove 'live' from IMAGE_FSTYPES to see this is working
> > 
> > Why?
> 
> The goal is to get rid of do_bootimg task as a dependency too. It's
> pointless to build, e.g., rootfs.img if a raw image is created by wic.
> 

The goal stated in this bug is to get rid of hddimg usage in wic code. Please create new bug if we need to get rid of live as well.

> > One idea could be to expect that all wic dependencies are present in image
> > sysroot. This would work, but can be hard to maintain as if .wks file starts
> > using new plugin user will have to modify list of dependencies in image
> > recipe.
> 
> To me this sounds obvious. It'd be the image recipe maintainers to decide
> what wic needs to build the image.
> 

Unfortunately this is true for the usage scenario when wic uses -e option, i.e. when it uses artefacts provided by the image build. I'd love to make it the only mode available, but it's not the case. Some users use wic manually and point to kernel, bootloaders and rootfs without using image recipe. This case would fail if wic doesn't have a recipe that contains native tools, which is quite tricky to achieve without wic-tools or similar recipe.  

> > I 100% agree and support this point of view. However, considering that in
> > some cases wic needs to add root device to the kernel command line it's not
> 
> Could it modify the existing bootloader configs?
>
Sure it can, but those configs can be quite different, especially if wic is pointed out to some custom bootloader build.

(In reply to comment #8)


> > > > - you'd need to remove 'live' from IMAGE_FSTYPES to see this is working
> > > 
> > > Why?
> > 
> > The goal is to get rid of do_bootimg task as a dependency too. It's
> > pointless to build, e.g., rootfs.img if a raw image is created by wic.
> > 
> 
> The goal stated in this bug is to get rid of hddimg usage in wic code.
> Please create new bug if we need to get rid of live as well.


Most of my feedback has the assumption that do_bootimg is dropped as well. I'll leave it to Saul to clarify what

"we would need to remove the dependency on hddimg which currently creates the artifacts needed to construct the FAT boot partition."

and 

"to create a bootpart replacement for bootimg"

mean.

The patchset has been merged to master.

(In reply to comment #10)
> The patchset has been merged to master.

It's still open whether they fix the bug.

not for me :)
otherwise I'd not set its status to RESOLVED/FIXED

I still don't know what else needs to be done here. Can you elaborate? And please provide practical evidence (breakages, tracebacks, error messages) that bug is not fixed.