Bug 11567 - Trying to create an new layer with the yocto-layer script fails.
Summary: Trying to create an new layer with the yocto-layer script fails.
Status: RESOLVED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: 2.4
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 2.6 M1
Assignee: Stephano Cetola
QA Contact:
URL:
Whiteboard: May 15 2018: RESOLVED
Depends on:
Blocks:
 
Reported: 2017-05-23 18:42 UTC by Phlip Balister
Modified: 2018-05-15 10:36 UTC (History)
10 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Done (doc changes complete)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Phlip Balister 2017-05-23 18:42:14 UTC
So I start adding a new layer to my custom distribution using this section of the manual:

5.1.8. Creating a General Layer Using the yocto-layer Script

but I can't find the yocto-layer script. I did find it in the meta-poky repo.

We should have a layer creation script that works for all distributions and is not poky specific.
Comment 1 Leonardo Sandoval Gonzalez 2017-05-23 20:09:24 UTC
(In reply to comment #0)
> So I start adding a new layer to my custom distribution using this section
> of the manual:
> 
> 5.1.8. Creating a General Layer Using the yocto-layer Script
> 
> but I can't find the yocto-layer script. I did find it in the meta-poky repo.
> 

it is under the script folder, so belongs to oe-core:
~/poky> git ls | grep yocto-layer
100755 blob d58facaa4186425ee105e578216b9b86ce5a24da    scripts/yocto-layer

> We should have a layer creation script that works for all distributions and
> is not poky specific.

the script is not tied to any poky version, as far as I know
Comment 2 Paul Eggleton 2017-05-24 04:01:51 UTC
I'm afraid yocto-layer *is* from poky, it is not in OE-Core. The scripts directory in the poky repo does have some poky-only components, though I think they are limited to yocto-layer and yocto-bsp.

I do however agree it would be good to move this functionality into OE-Core, and it shouldn't be too much of a problem given that there's actually not much to the template (unlike yocto-bsp from which yocto-layer was originally derived). Having said that rather than a standalone script we might choose to make it part of bitbake-layers, with the template and perhaps even the plugin itself in OE-Core.
Comment 3 Leonardo Sandoval Gonzalez 2017-05-24 13:59:40 UTC
(In reply to comment #2)
> I'm afraid yocto-layer *is* from poky, it is not in OE-Core. The scripts
> directory in the poky repo does have some poky-only components, though I
> think they are limited to yocto-layer and yocto-bsp.

you are right, this is the list of files/folder that belong to poky [1]

poky_scripts = ['scripts/yocto-bsp', 'scripts/yocto-kernel', 'scripts/yocto-layer', 'scripts/lib/bsp']

> 
> I do however agree it would be good to move this functionality into OE-Core,
> and it shouldn't be too much of a problem given that there's actually not
> much to the template (unlike yocto-bsp from which yocto-layer was originally
> derived). Having said that rather than a standalone script we might choose
> to make it part of bitbake-layers, with the template and perhaps even the
> plugin itself in OE-Core.

[1] http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe/tree/tests/test_mbox_mailinglist.py#n38
Comment 4 Leonardo Sandoval Gonzalez 2017-06-15 21:01:10 UTC
(In reply to comment #2)
> I'm afraid yocto-layer *is* from poky, it is not in OE-Core. The scripts
> directory in the poky repo does have some poky-only components, though I
> think they are limited to yocto-layer and yocto-bsp.
> 
> I do however agree it would be good to move this functionality into OE-Core,

base on the following text, I believe you mean bitbake instead of oe-core.

> and it shouldn't be too much of a problem given that there's actually not
> much to the template (unlike yocto-bsp from which yocto-layer was originally
> derived). Having said that rather than a standalone script we might choose
> to make it part of bitbake-layers, with the template and perhaps even the
> plugin itself in OE-Core.

so you mean we should migrate the poky/scripts/lib/bsp relevant templates into bitbake? if this is the case, also some parts of scripts/lib/bsp/engine.py must be migrated. IMO, using the poky engine is too much for this task so my approach would to have all meta layer creation logic inside the bitbbake/lib/bblayers/action.py.

Let me know what you think.
Comment 5 Leonardo Sandoval Gonzalez 2017-06-16 22:59:10 UTC
First approach, still not send to the ML

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=lsandov1/bitbake-layers-create-layer
Comment 6 Leonardo Sandoval Gonzalez 2017-06-19 17:37:06 UTC
patch sent to the bb ML:

http://lists.openembedded.org/pipermail/bitbake-devel/2017-June/008748.html
Comment 7 brian avery 2017-07-14 00:40:24 UTC
any status or feedback on this?
Once it is accepted, are we planning to remove yocto-layers and update the docs?  If so, could we make a bug to that effect and make it dependent on this one?

I was about to enter in a yocto-layers bug and it may now be irrelevant :)!
Comment 8 Leonardo Sandoval Gonzalez 2017-07-19 15:25:48 UTC
(In reply to comment #7)
> any status or feedback on this?
> Once it is accepted, are we planning to remove yocto-layers and update the
> docs?  If so, could we make a bug to that effect and make it dependent on
> this one?
> 

All good questions that I do not have a definite answer. In my opinion, both code/docs can coexist. what I did is focused on bitbake and the latter is also used outside our oe-core world (perhaps oe-core is the project that uses it more). The only item missing is the documentation on bitbake and for this I have update this entry. 

> I was about to enter in a yocto-layers bug and it may now be irrelevant :)!
Comment 9 Leonardo Sandoval Gonzalez 2017-08-16 19:39:37 UTC
This feature has been merged into master [1] but it certainly needs to be documented somewhere, thus passing this to Scott for further directions.

Scott, let me know if you need help documenting this feature. I can draft some text then you clean it.

[1] 884f74250d: Leonardo Sandoval: Wed Aug 9 09:24:16 2017 +0100: action: new bitbake-layer plugin to create a simple layer
Comment 10 Scott Rifenbark 2017-08-16 19:57:54 UTC
Leonardo, 

Tell me the impact to the user.  Is the script named the same?  Does it just live somewhere else now?  If you can describe the feature from the point of view of the user that would be using it then I can work something up. 

Thanks, 
Scott
Comment 11 Leonardo Sandoval Gonzalez 2017-08-21 19:23:10 UTC
(In reply to comment #10)
> Leonardo, 
> 
> Tell me the impact to the user.  Is the script named the same?

yes. The script name is the same (bitbake-layers). I did not implement the script, just a plugin used by the script as I describe below.

>  Does it just
> live somewhere else now? 

The script lives in the bitbake repository but what I have proposed lives into openeembedded-core, the latter being a plugin used by the former to create a simple layer.

> If you can describe the feature from the point of
> view of the user that would be using it then I can work something up. 
>

This is what the user will see when asking for help to the script

------
~/poky-testing/poky-yocto-layer/build> bitbake-layers --help
NOTE: Starting bitbake server...
usage: bitbake-layers [-d] [-q] [-F] [--color COLOR] [-h] <subcommand> ...

BitBake layers utility

optional arguments:
  -d, --debug           Enable debug output
  -q, --quiet           Print only errors
  -F, --force           Force add without recipe parse verification
  --color COLOR         Colorize output (where COLOR is auto, always, never)
  -h, --help            show this help message and exit

subcommands:
  <subcommand>
    add-layer           Add a layer to bblayers.conf.
    remove-layer        Remove a layer from bblayers.conf.
    flatten             flatten layer configuration into a separate output
                        directory.
    layerindex-fetch    Fetches a layer from a layer index along with its
                        dependent layers, and adds them to conf/bblayers.conf.
    layerindex-show-depends
                        Find layer dependencies from layer index.
    show-layers         show current configured layers.
    show-overlayed      list overlayed recipes (where the same recipe exists
                        in another layer)
    show-recipes        list available recipes, showing the layer they are
                        provided by
    show-appends        list bbappend files and recipe files they apply to
    show-cross-depends  Show dependencies between recipes that cross layer
                        boundaries.
    create-layer        Create a basic layer

Use bitbake-layers <subcommand> --help to get help on a specific command
------

What's new is the 'create-layer' plugin. Looking for help into the latter

------
~/poky-testing/poky-yocto-layer/build> bitbake-layers create-layer --help
NOTE: Starting bitbake server...
usage: bitbake-layers create-layer [-h] [--priority PRIORITY]
                                   [--example-recipe-name EXAMPLERECIPE]
                                   layerdir

Create a basic layer

positional arguments:
  layerdir              Layer directory to create

optional arguments:
  -h, --help            show this help message and exit
  --priority PRIORITY, -p PRIORITY
                        Layer directory to create
  --example-recipe-name EXAMPLERECIPE, -e EXAMPLERECIPE
                        Filename of the example recipe
------


In resume, this plugin creates a simple layer, with a way to configure the layer's priority and the example recipe name.

 
> Thanks, 
> Scott

Scott, let me know if this is enough info, otherwise I can elaborate something better.
Comment 12 Scott Rifenbark 2017-08-21 21:31:55 UTC
Thanks... I can work with this for now.  

Scott
Comment 13 Scott Rifenbark 2017-11-08 22:28:21 UTC
Hi, 

See http://www.yoctoproject.org/docs/2.5/dev-manual/dev-manual.html#managing-layers and http://www.yoctoproject.org/docs/2.5/dev-manual/dev-manual.html#creating-a-general-layer-using-the-bitbake-layers-script for changes to the dev-manual regarding the bitbake-layers script and culling out of the yocto-layers script. 

Let me know of any changes.

Thanks,
Scott
Comment 14 Kristi 2018-01-25 01:06:45 UTC
Taking ownership of this bug. This one has been in REVIEW status for over two months.  I am leaving it open, but could someone to look over the material? 

Thanks! 
Kristi
Comment 15 Scott Rifenbark 2018-03-22 09:32:52 UTC
Hi, 

Part of this bug is complete.  I have purged the following tools from the YP manual set:

 * yocto-bsp
 * yocto-kernel
 * yocto-layer

Following sections have had some rewrites:

 * https://www.yoctoproject.org/docs/2.5/dev-manual/dev-manual.html#managing-layers
 * https://www.yoctoproject.org/docs/2.5/dev-manual/dev-manual.html#creating-a-general-layer-using-the-bitbake-layers-script - Here, the topic of layer creation has been isolated.  It was previously combined with the 'bitbake-layers add-layer' command, which was not really the best.  That topic is not in a new section noted in the next bullet.
 * https://www.yoctoproject.org/docs/2.5/dev-manual/dev-manual.html#adding-a-layer-using-the-bitbake-layers-script

Re-assigning to me. 

Note: Two of the three sections were completed for this bug and it was in review for over two months with no interest from anyone to verify.  I am going to leave this now in review for a reasonable time and if no activity results I will mark it RESOLVED.

Thanks,
Scott
Comment 16 Scott Rifenbark 2018-03-27 15:15:18 UTC
Hi,

I completed a detailed section on creating a BSP layer in the BSP manual.  I used the BeagleBone hardware as an example to examine.  Please look this over and comment as needed.

https://www.yoctoproject.org/docs/2.5/bsp-guide/bsp-guide.html#creating-a-new-bsp-layer-using-the-bitbake-layers-script

Thanks,
scott
Comment 17 Stephano Cetola 2018-05-15 09:58:59 UTC
This text looks good. No other changes needed:

https://www.yoctoproject.org/docs/2.5/bsp-guide/bsp-guide.html#creating-a-new-bsp-layer-using-the-bitbake-layers-script
Comment 18 Scott Rifenbark 2018-05-15 10:36:08 UTC
Thanks Stephano, 

I set the doc flag to "done".

Scott