Bug 10755

Summary:	AUTOREV: morty definition results in a RecursionError when setting PV=SRCPV as documented
Product:	[Documentation] Reference	Reporter:	Joe Konno <joe.konno>
Component:	handbook	Assignee:	brian avery <brian.avery>
Status:	RESOLVED WORKSFORME	QA Contact:
Severity:	normal
Priority:	Medium	CC:	bluelightning, poky.doc.watcher, poky.watcher, srifenbark
Version:	2.2
Target Milestone:	2.3 M3
Hardware:	x86
OS:	Multiple
Whiteboard:	06 Jan 2017: NEEDINFO
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Joe Konno 2016-12-01 19:00:54 UTC

See: http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-AUTOREV

The morty text for AUTOREV says the following:

"""
When SRCREV is set to the value of this variable, it specifies to use the latest source revision in the repository. Here is an example:

     SRCREV = "${AUTOREV}"
                    
If you use the previous statement to retrieve the latest version of software, you need to be sure PV contains ${SRCPV}. For example, suppose you have a kernel recipe that inherits the kernel class and you use the previous statement. In this example, ${SRCPV} does not automatically get into PV. Consequently, you need to change PV in your recipe so that it does contain ${SRCPV}.
"""

For this example, I put together a powertop_%.bbappend recipe, which directs bitbake to always fetch the latest code from a git repo's master branch. In the bbappend, I have set SRCREV="${AUTOREV}" and PV="master+git${SRCPV}" as documented. When I do so, I get the following (I substituted "$HOME" to make it easier to read through paths):

"""
ERROR: ExpansionError during parsing $HOME/poky/meta/recipes-kernel/powertop/powertop_2.8.bb                   | ETA:  --:--:--
Traceback (most recent call last):
  File "$HOME/poky/bitbake/lib/bb/data_smart.py", line 386, in DataSmart.expandWithRefs(s='master+git${SRCPV}', varname='PV'):
                 try:
    >                s = __expand_var_regexp__.sub(varparse.var_sub, s)
                     try:
  File "$HOME/poky/bitbake/lib/bb/data_smart.py", line 111, in VariableParse.var_sub(match=<_sre.SRE_Match object; span=(10, 18), match='${SRCPV}'>):
                 else:
    >                var = self.d.getVarFlag(key, "_content", True)
                 self.references.add(key)
  File "$HOME/poky/bitbake/lib/bb/data_smart.py", line 737, in DataSmart.getVarFlag(var='SRCPV', flag='_content', expand=True, noweakdefault=False, parsing=False):
                     cachename = var + "[" + flag + "]"
    >            value = self.expand(value, cachename)

  File "$HOME/poky/bitbake/lib/bb/data_smart.py", line 410, in DataSmart.expand(s='${@bb.fetch2.get_srcrev(d)}', varname='SRCPV'):
         def expand(self, s, varname = None):
    >        return self.expandWithRefs(s, varname).value

  File "$HOME/poky/bitbake/lib/bb/data_smart.py", line 400, in DataSmart.expandWithRefs(s='${@bb.fetch2.get_srcrev(d)}', varname='SRCPV'):
                 except Exception as exc:
    >                raise ExpansionError(varname, s, exc) from exc

bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception RecursionError: maximum recursion depth exceeded while calling a Python object
"""

If I instead set PV="master+git${SRCREV}", which resolves to "master+gitAUTOINC" (not so lhelpful), it works fine. If I manually set SRCREV to the hash of master, it also works fine.

Are there additional conditions or caveats that need to be documented in order to use AUTOREV in my packaging?

Comment 1 Joe Konno 2016-12-02 00:55:04 UTC

I managed to fix myself by installing python3-git on my host. This is only documented as a need for "oe-selftest", though.

Assuming I'm correct, could "python3-git" be listed as an essential package for host systems? It appears AUTOREV will almost work without it, so it isn't obvious what's needed to get full AUTOREV feature support.

Comment 2 Scott Rifenbark 2016-12-02 19:16:13 UTC

Hi Joe, 

So in the YP Quick Start I have that python3 in the essential packages for Fedora.  See http://www.yoctoproject.org/docs/2.3/yocto-project-qs/yocto-project-qs.html#packages.  So I am wondering (as you are), if we need it for all the distributions.... 

Copying Paul Eggleton for input. 

Thanks, 
Scott

Comment 3 Paul Eggleton 2016-12-04 19:04:06 UTC

python3-git isn't being used by the fetcher code (I just double-checked), so it seems unlikely that installing that was what fixed this, at least directly. Were other packages installed when you installed that package?

Comment 4 Joe Konno 2016-12-05 16:04:51 UTC

@Paul - Probably, though I cannot recall. I recall having to install python3 in order for morty's bitbake to work at all, and then surfing the documentation for necessary packages for runqemu/testimage functionality. I looked at the "essentials" for Ubuntu/Debian here (my host is an Ubuntu 16.04-based system) and those packages needed for runqemu/testimage:

http://www.yoctoproject.org/docs/2.2/mega-manual/mega-manual.html#required-packages-for-the-host-development-system

http://www.yoctoproject.org/docs/2.2/mega-manual/mega-manual.html#enabling-tests

If any of those packages listed are necessary for AUTOREV to function properly, it would be nice to have those called out as an "essential" package.

Comment 5 Paul Eggleton 2016-12-05 19:28:05 UTC

I'm fairly sure that all that is needed to be able to use AUTOREV is a functioning git (command line) installation. I can't as yet explain what might have happened in your situation though, but I don't have much information to go on.

Are you able to find out which packages got installed via /var/log/dnf.log perhaps?

Comment 6 Stephen K Jolley 2017-01-03 19:35:54 UTC

It appears the information requested has been supplied.

Comment 7 Scott Rifenbark 2017-01-06 15:10:56 UTC

Stephen (and all), 

I still don't have information to document this.  Reading through the comments does not tell me if I need to make a wholesale addition of python3 package to the essentials of all the supported hosts to clear up the issue.  It looks as if we are not sure exactly what was needed to get AUTOREV feature to work.

Once I have clear direction on the any package installation changes for all the hosts, I can move forward with this documentation change.

thanks,
Scott

Comment 8 Scott Rifenbark 2017-02-02 18:39:21 UTC

Right now, 3 of the 4 distros show "python3" as an essential package.  The only one that does not is CentOS.

Scott

Comment 9 Paul Eggleton 2017-02-07 03:37:13 UTC

As of morty you need python3 to use the build system at all, so that won't be a factor here.

Joe, we need your help to isolate the package you installed or other action that resolved the issue - at the moment we're stuck. On Ubuntu I think you should be able to do something like this to see all of the packages that have been installed recently:

zgrep " install " /var/log/dpkg.log*

(zgrep since the older rotated out logs will be compressed).

Comment 10 Joe Konno 2017-02-07 15:53:58 UTC

I'll need to take the time to verify by building up a bare-bones Ubunut install-- I don't know when I'll have that kind of time-- but I did say this back in December:

"""
I managed to fix myself by installing python3-git on my host. This is only documented as a need for "oe-selftest", though.
"""

The implication is that python3-git was not installed when I filed the bug, but things started working again when it was.

It was a high-activity time on that host for myself and others, so it's not obvious which package(s) fixed me. Thus, I (at an unknown future date) or someone else will need to go through the motions of building up a bare-bones Ubuntu installation to see which "essential" packages are needed.

Comment 11 Scott Rifenbark 2017-02-07 22:38:09 UTC

Right now it appears that there is no documentation component to this bug.  If you believe that, could you set the doc flag to "No".  

Thanks, 
Scott

Comment 12 Stephen K Jolley 2017-02-16 16:10:21 UTC

Brian, can you test this and see if documentation changes are needed.

Comment 13 brian avery 2017-02-16 23:05:37 UTC

No documentation changes needed.
Could not reproduce this so am marking it as resolved works for me.

Test to verify:
  On a clean ubuntu-16.04 docker container which has only the pkgs listed in the http://www.yoctoproject.org/docs/latest/yocto-project-qs/yocto-project-qs.html#packages guide installed I based the test on Morty (tag yocto-2.2)

I changed the libunwind recipe to match the autorev/srcpv issue:
 SRCREV = "${AUTOREV}"
 PV = "1.1+git${SRCPV}"

There was no problem fetching this and then the package was commensurate with the head of the libunwind git repo.
It looked like:
libunwind-1.1+git0+f3f456c0d1-r0.i586.rpm
instead of 
libunwind-1.1+git0+bc8698fd7e-r0.i586.rpm

Comment 14 Scott Rifenbark 2017-02-20 16:12:04 UTC

Hi, 

Since Brian has marked this as RESOLVED, I am setting the doc flag to "No" as it does not affect docs any longer. 

Thanks, 
Scott