Bug 1656

Summary: Recipe creation/import script
Product: [Build System, Metadata & Runtime] Meta-yocto Reporter: Mark Hatle <mark.hatle>
Component: meta-yoctoAssignee: Kai Kang <kai.kang>
Status: VERIFIED FIXED QA Contact:
Severity: enhancement    
Priority: Low CC: hongxu.jia, jessica.zhang, josh, poky.bs.watcher, poky.watcher, sgw, song.liu, wd, yi.zhao, zhenfeng.zhao
Version: unspecified   
Target Milestone: 1.4 M1   
Hardware: x86   
OS: Multiple   
Whiteboard: (P3) (Test Case Ready)(Patch Merged)
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: ---

Description Mark Hatle 2011-10-05 12:02:58 UTC
A script or similar system that would allow you to give is an upstream URL, tarball, patch set, package (SRPM, or debian style) and generate a recipe based on those instructions.

The goal is to facilitate a starting point for someone who is not yet familar with writing recipes by hand.  Add options to be able to specify autoconf or not.. (perhaps even automatically detected based on the contents of the upstream sources or build instructions embedded in a package.)
Comment 1 Joshua Lock - Disabled 2011-10-18 10:30:32 UTC
I started to look at the feasibility of such a project and found
a colleague had already done some similar work for MeeGo - Autospectacle:

https://meego.gitorious.org/meego-developer-tools/autospectacle

Autospectacle uses various clever heuristics such as:
* detecting build dependencies by scanning configure files for
pkg-config and qmake for well-known patterns (i.e. AC_CHECK_LIB)
* determining license by keeping an internal map of license file
sha256sums to the equivalent license file.
* Checking spec files, debian control files, doap files, pkg-config
files and finally freshmeat for the package description.
* etc, etc.

In my mind there are two, possibly complementary, approaches to this task:
1) We can get users a long way by providing a script which takes a URI
as argument and begins to fill in a copy of the skeleton recipe with:
   - SRC_URI
   - SRC_URI checksums
   - description (using similar heuristics to autospectacle)
2) We can easily expand on 1) to perform more of the functions of
autospectacle but my opinion is that once we start to:
   a) generate a significant set of heuristics
   b) expand downloaded files to poke around inside and,
   c) interact with BitBake
we need to do so within a more structured program than a script. There
are two approaches for this in my mind:
   i) Develop a general program infrastructure for automating packaging
of software, where heuristics and other pieces (templates) are data
files arranged by target system. A user can run the program with a
target package and system and the program will generate as much of a
packaging file as possible. Clearly we'll only deliver heuristics and
data for Yocto but we'll steward a project that is capable of much more.
   ii) integrate into the build system to use the build system
infrastructure (functionality, work directories, etc) to boot-strap a
packaging effort.
Comment 2 Kai Kang 2011-12-31 00:09:43 UTC
change Target Milestone to m3
Comment 3 Kai Kang 2012-01-08 19:33:58 UTC
change Target Milestone back to M2
Comment 4 Kai Kang 2012-01-30 19:19:06 UTC
(In reply to comment #0)
> A script or similar system that would allow you to give is an upstream URL,
> tarball, patch set, package (SRPM, or debian style) and generate a recipe based
                                        ~~~~~~
> on those instructions.
> 
For debian style package, the control file extracted from .deb package doesn't contain the URI of the source repo or tar ball. So how to get it? If can't it, I wonder it is not worth to support debain style package
Comment 5 Kai Kang 2012-05-08 03:38:26 UTC
Patches are merged
Comment 6 Hongxu Jia 2012-10-26 06:03:09 UTC
Here is my test result:

1. The script "create-recipe" only support "tarball", not support "rpm"
2. Run script with tarball, there are some unexpected log messages, just like:

build$ create-recipe https://fedorahosted.org/releases/n/e/newt/newt-0.52.14.tar.gz
Downloading package:
https://fedorahosted.org/releases/n/e/newt/newt-0.52.14.tar.gz
Unpacking to : /tmp/oV5xfuTpAa
before ...Parsing content ....
tar (child): /home/jiahongxu/yocto/poky/build/newt/*.tar.bz2:cannot open: that file or direcory.
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
Can NOT get license from package itself.
Please update the license and license file manually.
Create bb file: /home/jiahongxu/yocto/poky/build/newt/newt_0.52.bb
Comment 7 Kai Kang 2012-10-26 06:07:56 UTC
(In reply to comment #6)
> Here is my test result:
> 
> 1. The script "create-recipe" only support "tarball", not support "rpm"
> 2. Run script with tarball, there are some unexpected log messages, just
> like:
> 
> build$ create-recipe
> https://fedorahosted.org/releases/n/e/newt/newt-0.52.14.tar.gz
> Downloading package:
> https://fedorahosted.org/releases/n/e/newt/newt-0.52.14.tar.gz
> Unpacking to : /tmp/oV5xfuTpAa
> before ...Parsing content ....
> tar (child): /home/jiahongxu/yocto/poky/build/newt/*.tar.bz2:cannot open:
> that file or direcory.
> tar (child): Error is not recoverable: exiting now
> tar: Child returned status 2
> tar: Error is not recoverable: exiting now
> Can NOT get license from package itself.
> Please update the license and license file manually.
> Create bb file: /home/jiahongxu/yocto/poky/build/newt/newt_0.52.bb

I try to extract tar.gz and tar.bz2 and then get the source code. The error that file doesn't exist should go to /dev/null.
I will check what cause that.
Comment 8 Kai Kang 2012-11-15 08:20:20 UTC
Yocto Commit e48fe4ed7aca0f16becd17ec6cc1978a4031c111 update the script to remove the warning information.
Comment 9 Yi Zhao 2012-11-30 08:11:38 UTC
Verified with 1.4_M1. 
gitinfo: 247b6a3754c2c50318e2a73d79981ba0e9cb560b

For a .gz tarball:
##############
$ ./scripts/create-recipe https://fedorahosted.org/releases/n/e/newt/newt-0.52.14.tar.gz
Downloading package: https://fedorahosted.org/releases/n/e/newt/newt-0.52.14.tar.gz
Unpacking to       : /tmp/3gXGbuDr2L
Parsing content    ....
Create bb file: /home/zhaoyi/work/poky/poky.git/newt/newt_0.52.14.bb
#############

For a .bz2 tarball:
#############
$ ./scripts/create-recipe https://fedorahosted.org/releases/s/n/snake/snake-0.11.1.tar.bz2
Downloading package: https://fedorahosted.org/releases/s/n/snake/snake-0.11.1.tar.bz2
Unpacking to       : /tmp/8DAwvqvNwc
Parsing content    ....
Can NOT get license from package source files.
Please update the LICENSE and LIC_FILES_CHKSUM manually.
Create bb file: /home/zhaoyi/work/poky/poky.git/snake/snake_0.11.1.bb
#########