Bug 3306

Summary: SRC_URI's for git repos should use 'protocol=http' whenever possible
Product: [Build System, Metadata & Runtime] Other YP Layers Reporter: Evadeflow <evadeflow>
Component: layersAssignee: Richard Purdie <richard.purdie>
Status: RESOLVED DUPLICATE QA Contact:
Severity: enhancement    
Priority: Low CC: mhalstead, poky.bs.watcher, poky.watcher, richard.purdie
Version: unspecified   
Target Milestone: Future   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Yes (doc changes required)
Attachments:
Description Flags
Patch to change Yocto Project git SRC_URIs to use http
none
Updated patch to change Yocto Project SRC_URIs to use http none

Description Evadeflow 2012-10-18 14:20:33 UTC 
Now that Michael has made all the yocto git repos available via http, this should be the preferred protocol. The reasons are:

1. Doing so will not adversely affect performance (see http://git-scm.com/2010/03/04/smart-http.html), and...

2. It will enable more of Yocto's value-add tools to work correctly for a greater number of people.

As an example of something that doesn't currently work for everyone, I'm appending below the output of running 'yocto-bsp' from behind my company's firewall. Changing the SRC_URI to use 'protocol=git' should enable this to work. (It doesn't, actually, due to https://bugzilla.yoctoproject.org/show_bug.cgi?id=3305, but Darren says he has or will have a fix for this soon...)


> % yocto-bsp create tegra arm
> Would you like to use the default (3.4) kernel? (y/n) [default: y] y
> Do you need a new machine branch for this BSP (the alternative is to re-use an existing branch)? [y/n] [default: y] 
> Getting branches from remote repo git://git.yoctoproject.org/linux-yocto-3.4.git...
> Traceback (most recent call last):
>   File "/home/evadeflow/projects/poky/scripts/yocto-bsp", line 137, in <module>
>     ret = main()
>   File "/home/evadeflow/projects/poky/scripts/yocto-bsp", line 132, in main
>     invoke_subcommand(args, parser, yocto_bsp_help_usage, subcommands)
>   File "/home/evadeflow/projects/poky/scripts/lib/bsp/help.py", line 73, in invoke_subcommand
>     subcommands.get(args[0], subcommand_error)[0](args[1:], usage)
>   File "/home/evadeflow/projects/poky/scripts/yocto-bsp", line 76, in yocto_bsp_create_subcommand
>     yocto_bsp_create(machine, karch, scripts_path, bsp_output_dir, options.codedump, options.properties_file)
>   File "/home/evadeflow/projects/poky/scripts/lib/bsp/engine.py", line 1260, in yocto_bsp_create
>     run_program_lines(program_lines, codedump)
> SyntaxError: function specified for 'gen' property returned nothing : input type:"choicelist" name:"new_kbranch" gen:"bsp.kernel.all_branches" branches_base:"standard" prio:"20" msg:"Please choose a machine branch to base this BSP on:" default:"standard/base"
Comment 1 Richard Purdie 2012-10-18 14:45:55 UTC 
Not every git repository we fetch from is accessed from git.yoctoproject.org. Changing all git:// urls to use protocol=git is therefore clearly a very bad idea since some upstreams may not support http.

I'm therefore closing this as invalid since we cannot do what is being requested.
Comment 2 Evadeflow 2012-10-18 15:32:34 UTC 
(In reply to comment #1)
> Not every git repository we fetch from is accessed from
> git.yoctoproject.org. Changing all git:// urls to use protocol=git is
> therefore clearly a very bad idea since some upstreams may not support http.
> 
> I'm therefore closing this as invalid since we cannot do what is being
> requested.

Sorry if the description for this issue was misleading. I considered
writing "all Yocto Project git repos", or "all HTTP-accessible repos",
but I thought this would be understood implicitly. Of course we can't
specify 'protocol=http' for repos that are not available via
git-over-http. But there are a large number of repos that are, and
changing the protocol will allow Yocto and its associated tools to work
correctly for a larger number of people.

In other words, I'm saying this: whenever there is a choice between
cloning a git repository via git protocol or http protocol, http should
be preferred, because doing otherwise is an unnecessary impediment to
Yocto's adoption in corporate environments. Now that the Yocto Project
repos are available via http, is there any upside to leaving things in a
state where yocto-bsp does not work for people behind a firewall?
Comment 3 Evadeflow 2012-10-18 16:06:50 UTC 
Created attachment 893 [details]
Patch to change Yocto Project git SRC_URIs to use http

I was suggesting something like this, for Yocto 1.4...
Comment 4 Michael Halstead 2012-10-18 20:09:04 UTC 
The patch as it stands breaks all of the HOMEPAGE URLs it changes.
Comment 5 Evadeflow 2012-10-19 13:26:29 UTC 
(In reply to comment #4)
> The patch as it stands breaks all of the HOMEPAGE URLs it changes.

D'oh! Nice catch. I will tweak my perl one-liner and re-post...
Comment 6 Evadeflow 2012-10-19 15:01:18 UTC 
Created attachment 894 [details]
Updated patch to change Yocto Project SRC_URIs to use http

This was surprisingly tedious. I finally punted and opted for a little brute force. Here are the commands I used to generate this patch from the current master:

> grep -rlI "SRC_URI *= *\"git://git.yoctoproject.org/" * | \
>   xargs perl -spi -e "s#(SRC_URI *= *\")(git://git.yoctoproject.org/)#\1\2git/#"
> grep -rlI "SRC_URI *= *\"git://git.yoctoproject.org/" * | \
>   xargs perl -spi -e "s#(SRC_URI *= *\")(git://git.yoctoproject.org/.*)protocol=git#\1\2protocol=http#"

I think it's correct now, but please review carefully...
Comment 7 Richard Purdie 2019-03-07 15:40:28 UTC 
We have discussions in three related bugs here. Basically rather than update the metadata to fit some particiular firewall config or set of circumstances, I'd prefer to do this through our mirrors syntax. There may be some tweaking needed to make the mirrors syntax handle this but that should be possible.

Closing some bugs so we link to one bug.

*** This bug has been marked as a duplicate of bug 9738 ***