Bug 4755

Summary: robots.txt mostly excludes google from indexing the website
Product: [Infrastructure] Website Reporter: Holger Freyther <holger>
Component: web-structureAssignee: Melissa Anderson <melissa>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Undecided CC: bluelightning, jefro, mhalstead
Version: unspecified   
Target Milestone: ---   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: ---

Description Holger Freyther 2013-06-20 18:13:29 UTC 
Searching for Yocto documentation turns out to be difficult. A search for "Poky environment-setup" points to the ancient o-hand documentation but not to the Yocto website. Searching "Yocto ADT Manual" gives a google page that says that the robots.txt prevents google from indexing the page.

Looking at the robots.txt I see a "Disallow: /" which might explain it.
Comment 1 Paul Eggleton 2013-06-25 15:22:26 UTC 
I think Michael has fixed this already after I discussed it with him directly -  Michael, can you confirm?
Comment 2 Michael Halstead 2013-06-25 15:41:43 UTC 
https://www.yoctoproject.org/robots.txt was was copied over from the development server during the last security update set on June 5th. Fixed at 2013-06-20 16:00.