Currently when poky master trying to fetch the WR kernels(if necessary) during the building process, it will fetch from the autobuilder instead of the WR git repository. This results WR kernel poky master compiled is not update to date. See the attachment for the email discussion
Created attachment 33 [details] email discussion
Created attachment 34 [details] email discussion - 2
I have changed the owner to Josh since Josh/Richard knows more about this kernel part. Basically the problem is that git tar ball for wr kernel is: "git_git.pokylinux.org.linux-2.6-windriver.git.tar.gz" There's no any tag info in the filename due to "fullclone" flag. I know that RP/Josh pushed support for multiple SRCREV. However that hasn't happened to downloaded tar ball yet. That means we may use a very old kernel since no way to tell whether newer is available from above name. For example, on my build system my wr kernel is still very old (Jul.10). We need solve this problem with high priority to ensure latest kernel with enough test.
The recent changes to the git fetcher should update the repository before any commands are run. So regardless of the age of the tarball, once inflated the repository should be up to date. I have yet to verify that myself but that's how the code reads to me...
(In reply to comment #4) > The recent changes to the git fetcher should update the repository before any > commands are run. So regardless of the age of the tarball, once inflated the > repository should be up to date. > I have yet to verify that myself but that's how the code reads to me... However I double confirmed this point from kernel WORKDIR. 'Git log' doesn't give anything newer than Jul.10 for both kernel branch (arm_versatile_926ejs-standard) and (wrs_meta)... Anything more I may provide to help you diagnostic here?
Thanks Kevin, I did the same here and came to the conclusion you did. I have a workaround but it's not a good fix so am debugging the issue and will update the bug as I have more information.
I have pushed several changes to the fetcher code in bitbake which ensure that the require ref exists in the git clone. bitbake/fetch: ensure the mirrored repository is updated as required bitbake/git: define a forcefetch method bitbake/fetch: Respect forcefetch even when pulling fron a mirror I thought this would be enough for the wr-kernel-tools to update it's branches appropriately but that appears not to be the case. I'm reluctant to add too much special casing to the bitbake fetcher though.
I'm trying it now... btw, for your change I have some questions: + def forcefetch(self, url, ud, d): + if not self._contains_ref(ud.tag, d) or 'fullclone' in ud.parm: + return True + return False Above code means that forcefetch is always set for 'fullclone', which means we can't use local copy for kernel anymore? If yes, this will add additional ~0.5hr to download, every time when we want to build a fresh kernel? Second question is: + # Need to re-test forcefetch() which will return true if our copy is too old + if m.forcefetch(u, ud, d) or not localpath: # Next try fetching from the original uri, u Again, since forcefetch is always true for 'fullclone', that means even when autobuilder mirror provides us tar ball which consumes quite some time to download, finally we still need to clone from scratch? Also how about adding multiple revisions check here? If the local repo has contain all desired revisions, there's no need to special case 'fullclone'. We can just decide forcefetch based on revisions. But I'm not sure whether it's easy to do...
Have to say, it takes too long time to clone a kernel repo. Now 280M downloaded in one hour... autobuilder tar ball is also considered old, and thus a fresh clone is started. Atcually I'm not quite sure how it tells that autobuiler mirror is old since the 'fresh clone' decision is made immediately in do_fetch task! I don't think that autobuilder mirror is even downloaded. I'm thinking whether we can even reuse an old tar ball, by doing incremental update atop it to accelerate the whole process. Take kernel for example, is it possible to extract the old tar ball, "git fetch" from remote repo to catch up, and then repackage updated repo into a new tar ball. This way a always forceupdate for fullclone repo is then tolerable
Well, I've got some results here. First, the changes that Josh pushed does work for me now. New kernel source is fresh cloned. I checked the branch log which shows latest commit from yesterday. So thanks Josh for fixing it! However based on below comment from Josh, I'm not sure whether other issues exist: "" I'm reluctant to add too much special casing to the bitbake fetcher though "" Second, I'm wrong in previous comment and local bits do be reused. Though it's not directly reuse on tar ball, the local git repo 'clonedir' is reused. This way only the tar ball is regenerated every time rebuilding kernel which only add small time for fullclone type Last, I still don't understand why autobuilder mirror looks be skipped: NOTE: fetch http://pokylinux.org/sources/git_git.pokylinux.org.linux-2.6-windriver.git.tar.gz NOTE: fetch http://autobuilder.pokylinux.org/sources/git_git.pokylinux.org.linux-2.6-windriver.git.tar.gz NOTE: Creating tarball of git repository NOTE: fetch http://pokylinux.org/sources/ first 2 fetches quickly end, but I can access autobuilder link from browser.
(In reply to comment #10) > However based on below comment from Josh, I'm not sure whether other issues > exist: > "" > I'm reluctant to add too much special casing to the bitbake fetcher though > "" sorry, I actually referred to this one: "" I thought this would be enough for the wr-kernel-tools to update it's branches appropriately but that appears not to be the case. ""
(In reply to comment #10) > Well, I've got some results here. > > First, the changes that Josh pushed does work for me now. New kernel source is > fresh cloned. I checked the branch log which shows latest commit from > yesterday. So thanks Josh for fixing it! Brilliant! When I was testing I would cd to the checkout directory and the active/current HEAD didn't include the required ref - but it does exist in the git db. This is why I am uncertain as to whether more work is needed. > Second, I'm wrong in previous comment and local bits do be reused. Though it's > not directly reuse on tar ball, the local git repo 'clonedir' is reused. This > way only the tar ball is regenerated every time rebuilding kernel which only > add small time for fullclone type Great, I was a little worried when first reading your replies. The first clone is definitely a big sink time-wise, users really shouldn't see this and we should really just be fetching a mirror tarball and updating the tree contained therein. I'm assuming this didn't happen for you? We doubtless need more work here... > > Last, I still don't understand why autobuilder mirror looks be skipped: > > NOTE: fetch > http://pokylinux.org/sources/git_git.pokylinux.org.linux-2.6-windriver.git.tar.gz > NOTE: fetch > http://autobuilder.pokylinux.org/sources/git_git.pokylinux.org.linux-2.6-windriver.git.tar.gz > NOTE: Creating tarball of git repository > NOTE: fetch http://pokylinux.org/sources/ > > first 2 fetches quickly end, but I can access autobuilder link from browser. I don't suppose it's something as simple as needing proxy set up for wget? If not I will have to investigate further
(In reply to comment #12) > (In reply to comment #10) > > Second, I'm wrong in previous comment and local bits do be reused. Though it's > > not directly reuse on tar ball, the local git repo 'clonedir' is reused. This > > way only the tar ball is regenerated every time rebuilding kernel which only > > add small time for fullclone type > Great, I was a little worried when first reading your replies. The first clone > is definitely a big sink time-wise, users really shouldn't see this and we > should really just be fetching a mirror tarball and updating the tree contained > therein. > I'm assuming this didn't happen for you? We doubtless need more work here... yes, it's not the case in my side > > > > Last, I still don't understand why autobuilder mirror looks be skipped: > > > > NOTE: fetch > > http://pokylinux.org/sources/git_git.pokylinux.org.linux-2.6-windriver.git.tar.gz > > NOTE: fetch > > http://autobuilder.pokylinux.org/sources/git_git.pokylinux.org.linux-2.6-windriver.git.tar.gz > > NOTE: Creating tarball of git repository > > NOTE: fetch http://pokylinux.org/sources/ > > > > first 2 fetches quickly end, but I can access autobuilder link from browser. > I don't suppose it's something as simple as needing proxy set up for wget? > If not I will have to investigate further I have confirmed my ~/.wgetrc file, which still has a valid proxy setting which I have used for a long time. It'd be good if you can have a look in my side, upon which I may know whether it's my own env problem. :-)
(In reply to comment #9) > Have to say, it takes too long time to clone a kernel repo. Now 280M downloaded > in one hour... autobuilder tar ball is also considered old, and thus a fresh > clone is started. Atcually I'm not quite sure how it tells that autobuiler > mirror is old since the 'fresh clone' decision is made immediately in do_fetch > task! I don't think that autobuilder mirror is even downloaded. > > I'm thinking whether we can even reuse an old tar ball, by doing incremental > update atop it to accelerate the whole process. Take kernel for example, is it > possible to extract the old tar ball, "git fetch" from remote repo to catch up, > and then repackage updated repo into a new tar ball. This way a always > forceupdate for fullclone repo is then tolerable For this problem in particular, I've been ignoring it while we get things such that we get the correct / up to date tree first. But it is something that needs to be solved for a more rapid development cycle. For a more delivery based cycle, the mirrors can play the speedup role (as Josh mentioned further down the bug). What we've done for the Wind River kernel is done in stages: - we have a core kernel.org repo available at a known location, and use it as a reference for the clone. This is a massive time saving, and would be nice if it could be an optional argument to the SRC_UR. Josh: is something like this feasible ? - That stage above is only used if we do need to clone. The update code first tries a fetch on all the remote branches to their local counterpart. There is code to deal with deleted upstream branches, and other corner cases - If the standard incremental fetch fails, we do an optional force fetch on the remote branches (a flag must indicate that this acceptable). Both of the last 2 methods almost always mean we don't need to re-clone to get updates. Josh/Richard: can something like that be done on the tree in the sources directory ? I'm available to help implement/test/debug this if required. Or perhaps this is all in place, and I haven't seen it yet, since I'm just looking over the changes now.
The mirrors code with a file:// url specified as mirror of the git repo should allow a speeded up checkout with the existing code.
We seem to have a remaining issue whereby if the ref doesn't exist locally we pull the mirror tarball rather than updating the local copy. I haven't had much time to dive into this today unfortunately but I feel that the core of the bug is fixed, at least from the bitbake side. I need to fix this as a matter of priority tomorrow...
As the bug filed here is fixed I've re-prioritised the final piece as a fetcher enhancement for M4 filed as bug #268 - Fetcher speed enhancements when a vcs update required (i.e. git with fullclone): http://bugzilla.pokylinux.org/show_bug.cgi?id=268 Marking this bug as fixed.
Verified as now WR kernel is got from git tree, not from the old autobuilder site.