Plan is using MetaCPAN API [0] to avoid any need to touch "fetcher" or actually run CPAN locally.

[0] https://github.com/metacpan/metacpan-api/blob/master/docs/API-docs.md

This is a good task for incoming Yocto folks. It is also a great intro to recipe-tool / devtool. I'm taking this with the hopes of delegating it to someone in the future.

Once the perldeps enhancement is in, this will be easier to do (simpler than the metacpan API plan I had previously started on).

Turns out the implementation in Comment 1 is actually quite doable.

Need to use either package data or layer index (via rest API) to query what recipes are available and then create a "missing" list to iterate recipetool on.

Would also benefit from implementation of ptest logic (look at what is in /t folder and add to FILES_${PN}-ptest and intelligently add ptest requirements vs. what is provided in e.g. /t/lib).

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=timo/recipetool-perl-v3

Rebased on top of latest tip of poky.

Still on the back burner simmering.

Still valid.

Hi Tim, 
I would like to play with this, but I am not sure how to use it? 
I tried something like: devtool add https://metacpan.org/pod/Aard but the handler is never called, I get an error before that: 

Traceback (most recent call last):                                                                                 
  File "/home/jstephan/upstream/yocto/poky/scripts/recipetool", line 111, in <module>                              
    ret = main()                                                                                                   
          ^^^^^^                                                                                                   
  File "/home/jstephan/upstream/yocto/poky/scripts/recipetool", line 100, in main                                  
    ret = args.func(args)                                                                                          
          ^^^^^^^^^^^^^^^                                                                                          
  File "/home/jstephan/upstream/yocto/poky/scripts/lib/recipetool/create.py", line 509, in create_recipe
    check_single_file(dirlist[0], fetchuri)
  File "/home/jstephan/upstream/yocto/poky/scripts/lib/recipetool/create.py", line 915, in check_single_file
    with open(fn, 'r', errors='surrogateescape') as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'Aard'

It needs you to use the "Download" link (usually a tarball) e.g. https://cpan.metacpan.org/authors/id/M/MG/MGV/Aard-0.001.tar.gz

Like most other recipetool plugins, it needs to be pointing to the source.

However, I have not touched this code in quite some time and not sure what state it is in at the moment. The last time I worked on it I was blocked by the "nested" build/runtime/test dependencies in the ElasticSearch results. This made the _build_dependency_query(), _runtime_dependency_query() and _test_dependency_query() functions not useful.

It was not possible to get the "inner" values as far as I could see. If you have some ElasticSearch expertise, perhaps you can help.

Rebased on top of latest "master".

https://git.yoctoproject.org/poky-contrib/log/?h=timo/recipetool-perl-v3

(In reply to Tim Orling from comment #9)
> It needs you to use the "Download" link (usually a tarball) e.g.
> https://cpan.metacpan.org/authors/id/M/MG/MGV/Aard-0.001.tar.gz
> 
> Like most other recipetool plugins, it needs to be pointing to the source.
> 
> However, I have not touched this code in quite some time and not sure what
> state it is in at the moment. The last time I worked on it I was blocked by
> the "nested" build/runtime/test dependencies in the ElasticSearch results.
> This made the _build_dependency_query(), _runtime_dependency_query() and
> _test_dependency_query() functions not useful.
> 
> It was not possible to get the "inner" values as far as I could see. If you
> have some ElasticSearch expertise, perhaps you can help.

Oh right! thank you! 
Unfortunately I do not have ElasticSearch expertise at all :/

Hi Tim, 
I would like to help on this to get it merged  :)

I recently sent a patch adding an optional callback "process_url" that can be implemented by plugin to handle urls. See https://lists.openembedded.org/g/openembedded-core/message/190840. I use it to handle pypi url for python packages (see https://lists.openembedded.org/g/openembedded-core/message/190841). I think it would be nice to implement it here so we can use "devtool add https://metacpan.org/pod/Aard". This is not merged yet. 

I can add this if you want. 

Also, you mentioned issues with elastic search, can you explain me how I can reproduce them and see if I can help to fix it? 


Cheers
Julien

The issue is that we would like to have build/runtime/test dependencies be determined from the metacpan.org metadata. Because of how nested Elastic Search references work, we cannot query the dependencies (they are inside a nested element that is sort of like a foreign key).

I was also looking at just querying the META.json or META.yml files but those are inconsistent per CPAN package/tarball.

I might need to talk to the upstream metacpan.org devs to see if there is a non-obvious way to do the query.

Still valid.