Bug 5738

Summary: Default 'noop' signature handler isn't compatible with latest codebase
Product: [Build System, Metadata & Runtime] BitBake Reporter: Martin Jansa <Martin.Jansa>
Component: bitbakeAssignee: Richard Purdie <richard.purdie>
Status: RESOLVED FIXED QA Contact:
Severity: minor    
Priority: Low CC: poky.bs.watcher, poky.watcher, rudolf.streif, sgw
Version: 1.7   
Target Milestone: Future   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Martin Jansa 2014-01-17 13:08:54 UTC
When working on https://github.com/openwebos/meta-webos/commit/9ac3a7c803e7793b3274e4998f167b6278db8042 I've noticed that this area is a bit error prone and the reporting isn't very helpful in bitbake 1.18.

With bitbake 1.21.1 (and IIRC 1.20 as well) it's better and missing OE_IMPORTS addition or typo in BB_SIGNATURE_HANDLER results in good error message:

OE qemux86-64@ ~/build/oe-core $ bitbake -S dbus
ERROR: Invalid signature generator 'OEBasicHashFoo', using default 'noop'
Available generators: basichash, noop, OEBasic, basic, OEBasicHash
...
but then it throws exception from noop hander, because dump_sigs is implemented only in SignatureGeneratorBasic not in SignatureGenerator ('noop').

NOTE: Reparsing files to collect dependency data
ERROR: An uncaught exception occured in runqueue, please see the failure below:
ERROR: Running idle function
Traceback (most recent call last):
  File "/OE/build/oe-core/bitbake/lib/bb/server/process.py", line 133, in ProcessServer.idle_commands(delay=0.1, fds=[<read-only Connection, handle 3>, <read-write Connection, handle 6>]):
                 try:
    >                retval = function(self, data, False)
                     if retval is False:
  File "/OE/build/oe-core/bitbake/lib/bb/cooker.py", line 1177, in buildTargetsIdle(server=<ProcessServer(ProcessServer-2, started)>, rq=<bb.runqueue.RunQueue instance at 0x83bff38>, abort=False):
                 try:
    >                retval = rq.execute_runqueue()
                 except runqueue.TaskFailure as exc:
  File "/OE/build/oe-core/bitbake/lib/bb/runqueue.py", line 1082, in RunQueue.execute_runqueue():
             try:
    >            return self._execute_runqueue()
             except bb.runqueue.TaskFailure:
  File "/OE/build/oe-core/bitbake/lib/bb/runqueue.py", line 1033, in RunQueue._execute_runqueue():
                     invalidtasks = self.print_diffscenetasks()
    >                self.dump_signatures()
                     self.write_diffscenetasks(invalidtasks)
  File "/OE/build/oe-core/bitbake/lib/bb/runqueue.py", line 1114, in RunQueue.dump_signatures():

    >        bb.parse.siggen.dump_sigs(self.rqdata.dataCache)

AttributeError: 'SignatureGenerator' object has no attribute 'dump_sigs'
Comment 1 Martin Jansa 2014-01-17 13:12:50 UTC
Adding empty dump_sigs helps, but it will still show error a bit later (with new -S functionality in master)

NOTE: Reparsing files to collect dependency data
ERROR: Can't find a task we're supposed to have written out? (hash: 0)?
Comment 2 Martin Jansa 2014-01-17 13:13:59 UTC
http://lists.openembedded.org/pipermail/bitbake-devel/2014-January/004336.html is the first part with empty dump_sigs()
Comment 3 Rudolf J Streif 2014-01-20 20:53:37 UTC
I noticed that too when trying my BitBake HelloWorld against the latest from master. The reason is that bitbake/lib/bb/runqueue.py when setting

workerdata = {
...
            "hashes" : bb.parse.siggen.taskhash,
            "hash_deps" : bb.parse.siggen.runtaskdeps,
            "sigchecksums" : bb.parse.siggen.file_checksum_values,
...
}

I do not think it makes sense at all to run BitBake without a signature handler. The simplest approach to fixing this would be setting the default signature handler to 'basic' instead of 'noop'.



Signed-off-by: Rudolf J Streif <rudolf.streif@linux.com>
---
 lib/bb/siggen.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index 9db29a2..159ab72 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -17,13 +17,13 @@ def init(d):
     siggens = [obj for obj in globals().itervalues()
                       if type(obj) is type and issubclass(obj, SignatureGenerator)]

-    desired = d.getVar("BB_SIGNATURE_HANDLER", True) or "noop"
+    desired = d.getVar("BB_SIGNATURE_HANDLER", True) or "basic"
     for sg in siggens:
         if desired == sg.name:
             return sg(d)
             break
     else:
-        logger.error("Invalid signature generator '%s', using default 'noop'\n"
+        logger.error("Invalid signature generator '%s', using default 'basic'\n"
                      "Available generators: %s", desired,
                      ', '.join(obj.name for obj in siggens))
         return SignatureGenerator(d)
Comment 4 Martin Jansa 2014-01-20 22:08:53 UTC
RP already merged my patch partially fixing this, but I'm fine with changing the default to "basic" too (then we should probably remove "noop" completely to reduce confusion).

I wasn't trying to use noop intentionally, I was only bitten by it by those accidents with my own handler.
Comment 5 Rudolf J Streif 2014-01-20 22:59:57 UTC
Alternatively the patch below fixes the SignatureGenerator to not break anything. That would leave the SignatureGenerator base class intact to derive others from. Removing 'noop' may break alternative implementations of signature generators that are derived from it, if there are any.

If there are any they then apparently define the three attributes taskhash, runtaskdeps and file_checksum_values and also override get_taskhash.


Signed-off-by: Rudolf J Streif <rudolf.streif@linux.com>
---
 lib/bb/siggen.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index 9db29a2..a7f1396 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -34,7 +34,10 @@ class SignatureGenerator(object):
     name = "noop"

     def __init__(self, data):
-        return
+        self.taskhash = {}
+        self.runtaskdeps = {}
+        self.file_checksum_values = {}
+

     def finalise(self, fn, d, varient):
         return
@@ -42,7 +45,7 @@ class SignatureGenerator(object):
     def get_taskhash(self, fn, task, deps, dataCache):
         return "0"

-    def set_taskdata(self, hashes, deps):
+    def set_taskdata(self, hashes, deps, checksums):
         return

     def stampfile(self, stampbase, file_name, taskname, extrainfo):