Bug 5035

Summary:	Make the official releases work with prefetched or created tarballs
Product:	[Build System, Metadata & Runtime] OE-Core	Reporter:	Laszlo Papp <lpapp>
Component:	core	Assignee:	Scott Rifenbark <srifenbark>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	Low	CC:	bluelightning, erik.boto, jefro, meta.mr.watcher, meta.watcher, richard.purdie, ross.burton, sgw, srifenbark
Version:	unspecified
Target Milestone:	1.5
Hardware:	x86
OS:	Multiple
Whiteboard:	27 September 2013: RESOLVED
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	Done (doc changes complete)

Description Laszlo Papp 2013-08-21 10:30:35 UTC

As of now, it is not possible to use only tarballs to check into a git repository. That is because packages like "gtk-doc-stub" depends on the presence of a bare clone repository inside the build/downloads/etc folder.

That will cause issues for several reasons:

* refs is an empty folder in that particular case which people cannot check into the repository

* SRCVER will not be able to be found, so a fetch job will be fired up again regardless it is already downloaded

* besides the fact that it hits the original purpose of speeding up the build, it also goes in our case to an error because of the git repository being flaky.

* this is also a regression for people coming from, for instance denzil.

Yes, there is an option for setting up local mirrors, but it is somewhat an effort and investment to make work. A slightly better migration and advancement path would be to establish an option for tarballs to just work. That way, people could check the tarballs into the build/downloads folders of the poky container repository with their layers, et al.

Hereby, I am proposing to have this feature available in place. Fixing gtk-doc-stub by removing it or so will not make the problem go away because it is a general git fetcher issue.

Comment 1 Richard Purdie 2013-08-21 15:20:59 UTC

I have several comments:

a) Putting DL_DIR under version control such as under git is not supported approach

b) The real issue here is that git does not preserve empty directories which is a git issue, not something we can control

c) Its not a regression from denzil since this was never supported and it likely only worked there by change since it was a first build, or it might have been that tarballs were generated in that release by default.

d) We already support generating tarballs of the git repositories using the BB_GENERATE_MIRROR_TARBALLS option. If you set this, tarballs of the git repositories will be placed in DL_DIR. This is not the default for performance reasons, not everybody needs this.

In your particular case I'd therefore suggest you stop checking in the /git/ and /git2/ directories of DL_DIR and turn on BB_GENERATE_MIRROR_TARBALLS instead. I'm not guaranteeing that will work since its still not a supported scenario however it should work better.

Comment 2 Laszlo Papp 2013-08-21 15:27:15 UTC

You seem to be totally wrong ...

a) it *is* supported, and the solution was actually mentioned on the mailing list how.

b) why would git add feature to address bad Yocto shortcomings? You would need to prove a _valid_ use case for tracking empty directories with git. No one has ever been to do so; so far, that is.

c) you are telling me it is not a regression when my team is blocked now due to this, and it has been working with denzil?

d) as I already suggested, and others wrote btw what you did here: this bug should be turned into a documentation issue.

So, yes, this is a bug, and I am frustrated with closing it without contacting the reporter to have further comment. It is rude and odd.

Adding Jeffrey to handle this.

Comment 3 Laszlo Papp 2013-08-21 15:28:23 UTC

Jeff, here is the relevant thread where it has been discussed: http://lists.openembedded.org/pipermail/openembedded-core/2013-August/082816.html

Comment 4 Jeff Osier-Mixon 2013-08-21 22:35:01 UTC

Thanks for bringing this issue to our attention.

The project does not support putting DL_DIR under git. It never has done so, despite the fact that it worked in a previous release, and very likely won't in the future due to the previously mentioned issues about preserving empty directories. This is one of the many thousands of decisions made by the chief architect and the development team.

For that reason, this does not qualify as a regression, as it was never a supported or documented feature. The tools may have done things in denzil that they don't do now, and they may do things now that they didn't do then, but we have to limit the definition of "regression" to items that we actually document and support.

I'm reopening this as a documentation bug so that Scott can add a note in the appropriate place like this:

"The Yocto Project does not support placing DL_DIR under git, as git does not preserve empty directories and the use of DL_DIR requires them. However, a different approach is to use the BB_GENERATE_MIRROR_TARBALLS option. If you set this, tarballs of the git repositories are placed in DL_DIR. This is not the default for performance reasons."

When this is in the docs, I recommend that this bug be closed.

Comment 5 Laszlo Papp 2013-08-22 04:18:01 UTC

Sounds good, thanks.

Comment 6 Scott Rifenbark 2013-09-18 20:37:30 UTC

Hi, 

I have made several changes to the docs to try to satisfy this issue.  I am changing the status to IN PROGRESS REVIEW.  Here are the links for review:

In the Quick Start - http://www.yoctoproject.org/docs/1.5/yocto-project-qs/yocto-project-qs.html#configuring-the-local.conf-file

In the YP Reference Manual - http://www.yoctoproject.org/docs/1.5/ref-manual/ref-manual.html#var-BB_GENERATE_MIRROR_TARBALLS

In the YP Reference Manual - http://www.yoctoproject.org/docs/1.5/ref-manual/ref-manual.html#var-DL_DIR

In the YP Reference Manual - http://www.yoctoproject.org/docs/1.5/ref-manual/ref-manual.html#sources-dev-environment

In the YP Reference Manual - http://www.yoctoproject.org/docs/1.5/ref-manual/ref-manual.html#scms

Thanks, 
Scott

Comment 7 Scott Rifenbark 2013-09-27 21:20:58 UTC

I am changing the status to RESOLVED.

Scott

Comment 8 Laszlo Papp 2013-09-27 23:13:55 UTC

Hi Scott,

thank you for your work. Sorry but I have not found the time to review this. It looks ok to me.

One minor issue I found on the third link:

"You can set this directory by defining the DL_DIR variable in the /conf/local.conf file. This directory is self-maintaining and you should not have to touch it. By default, the directory is downloads in the Build Directory.

     #DL_DIR ?= "${TOPDIR}/downloads""

TOPDIR is not the Build Directory as the line says above.