Bug 8101 - Cloning via http prevents Toaster from building
Summary: Cloning via http prevents Toaster from building
Status: VERIFIED FIXED
Alias: None
Product: Toaster
Classification: Build System, Metadata & Runtime
Component: toaster (show other bugs)
Version: 1.8
Hardware: x86 Multiple
: Low normal
Target Milestone: 2.0
Assignee: Ed Bartosh
QA Contact: Alexandru Roman
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2015-08-05 10:06 UTC by Belen Barros Pena
Modified: 2015-11-02 14:13 UTC (History)
8 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Belen Barros Pena 2015-08-05 10:06:52 UTC
If I clone the poky repo using http:// instead of git://, the resulting Toaster instance will not build. Builds fail before layer cloning starts with this error:

File "/home/yocto/toaster-master/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py", line 59, in schedule
    bec.triggerBuild(br.brbitbake_set.all(), br.brlayer_set.all(), br.brvariable_set.all(), br.brtarget_set.all())
  File "/home/yocto/toaster-master/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py", line 319, in triggerBuild
    self.setLayers(bitbake, layers)
  File "/home/yocto/toaster-master/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py", line 255, in setLayers
    raise BuildSetupException("Existing git repository at %s, but with different remotes ('%s', expected '%s'). Toaster will not continue out of fear of damaging something." % (localdirname, ", ".join(localremotes.split("\n")), giturl))
BuildSetupException: Existing git repository at /home/yocto/_poky_master.toaster_cloned, but with different remotes ('origin    git://git.yoctoproject.org/poky.git (fetch), origin    git://git.yoctoproject.org/poky.git (push), ', expected 'http://git.yoctoproject.org/git/poky'). Toaster will not continue out of fear of damaging something.

This is Michael's explanation of the problem: "It's because you have an existing repo cloned by toaster using a different remote (git url) and it can't clone it because it will overwrite the existing cloned repo"
Comment 1 brian avery 2015-08-26 21:09:30 UTC
I did:
git clone http://git.yoctoproject.org/git/poky
mkdir toaster_run
cd toaster_run/
TOASTER_MANAGED=1 TOASTER_DEVEL=1  ../poky/bitbake/bin/toaster webport=8005
-- I built bash off of master branch and it succeeded --
Comment 2 Belen Barros Pena 2015-08-27 10:05:11 UTC
It turns out the builds only fail when Toaster finds an exiting poky repository that matches the one it is trying to create. 

This is what I did:

git clone http://git.yoctoproject.org/git/poky --branch master --single-branch 8101

cd 8101

bitbake/bin/toaster

used defaults for artifacts and checkout directory, and specified /home/8101/build as the build directory

created a project selecting the master release

tried to build bash

the build failed with this message:

Existing git repository at /home/yocto/_poky_master.toaster_cloned, but with different remotes ('origin	git://git.yoctoproject.org/poky (fetch), origin	git://git.yoctoproject.org/poky (push), ', expected 'http://git.yoctoproject.org/git/poky'). Toaster will not continue out of fear of damaging something.

Then I stopped Toaster, went to my layers checkout directory and deleted the directory Toaster complains about

/home/yocto/_poky_master.toaster_cloned

Restarted Toaster, and fired a new build for bash: the build finished without problems. So the issue is not the cloning with http:// per se, but Toaster bailing out when it finds an existing poky repository in the layer checkout directory that has a different git URI to the one Toaster is supposed to clone from. 

I would have assumed that if this happens Toaster would simply make a new clone, but I really have no idea of what I am talking about, so this might cause new problems.
Comment 3 Stephen K Jolley 2015-09-14 16:30:11 UTC
Needinfo answered.  Please ask another question if you still need infomation.
Comment 4 brian avery 2015-09-15 19:30:37 UTC
changed to low priority as this is a bit of a corner case.
Comment 5 Belen Barros Pena 2015-09-22 14:54:53 UTC
(In reply to comment #4)
> changed to low priority as this is a bit of a corner case.

I am not so sure this is a corner case: I think the moment you start using imported layers, you are bound to hit this.
Comment 6 Ed Bartosh 2015-09-22 15:20:33 UTC
Brian, I'll take this if you don't mind.
I'm working on bug 7505. This bug is mentioned in bug 7505 as an 'Example 3'.
Comment 7 brian avery 2015-09-22 15:46:58 UTC
fine by me
Comment 8 Ed Bartosh 2015-09-23 13:20:10 UTC
I decided to make remote name out of git url and allow one clone to have different remotes. So, for example for url https://github.com/agherzan/meta-raspberrypi.git remote name will be https___github.com_agherzan_meta-raspberrypi.git and for url git://github.com/agherzan/meta-raspberrypi.git remote will be different: git___github.com_agherzan_meta-raspberrypi.git

The advantage of this approach is that multiple remotes will share git objects and fetching will be faster than if they're in different clones.

It would be even better to remove branch name from the directory name. This would allow to have one clone directory for one layer.
Comment 9 Ed Bartosh 2015-09-23 14:11:31 UTC
Nope, this approach will not work as multiple layers can't be accessed by bitbake without resetting git tree. It's better to have clones in separate directories as Michael suggested.
Comment 10 Ed Bartosh 2015-09-24 07:55:30 UTC
Submitted to upstream: http://lists.openembedded.org/pipermail/bitbake-devel/2015-September/006423.html
Comment 11 Ed Bartosh 2015-09-30 14:54:57 UTC
Merged into bitbake master: http://cgit.openembedded.org/bitbake/commit/?id=a9baf856a1ef7d84a47fa6615353f5b3fdb8a73b
Comment 12 Alexandru Roman 2015-11-02 14:12:14 UTC
Verified on master: 370a2916c4559c17952392b872322e8e490b124c