Bug 14753

Summary: bitbake cannot connect to xmlrpc socket (BBSERVER=0.0.0.0:port)
Product: [Build System, Metadata & Runtime] BitBake Reporter: Tim Orling <tim.orling>
Component: bitbakeAssignee: Richard Purdie <richard.purdie>
Status: RESOLVED FIXED QA Contact:
Severity: major    
Priority: High CC: david.reyna, poky.bs.watcher, poky.watcher, randy.macleod
Version: 4.1   
Target Milestone: 4.0 M4   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know

Description Tim Orling 2022-03-07 00:29:55 UTC
Toaster bitbake/lib/toaster/bldcontrol/localhostbecontroller.py#n448
starts an XMLRPC server instance of bitbake with (the equivalent of):

BITBAKE_UI="knotty" bitbake --server-only -B 0.0.0.0:0

(This is successful and python code confirms that the XMLRPC server is functional)

After this, Toaster bitbake/lib/toaster/bldcontrol/localhostbecontroller.py#n501 tries to run a build connecting to this server with (the equivalent of):

BITBAKE_UI="toasterui" BBSERVER="0.0.0.0:<port>" bitbake quilt-native --token ""

(It does not matter if the ui is "knotty" instead).

Since somewhere in 3.4, this fails with
WARNING: Could not connect to server at 0.0.0.0:<port> (<Fault 1: "<class 'AttributeError'>:module 'bb.server' has no attribute 'xmlrpcclient'">)

Prior to 3.4, it fails with:
xmlrpc.client.Fault: <Fault 1: "<class 'xml.parsers.expat.ExpatError'>:not well-formed (invalid token): line 181, column 17"> which is probably unrelated (more likely a python strings/bytes problem).
Comment 1 Tim Orling 2022-03-07 00:31:54 UTC
A test script I attempted to use with git bisect:
--------
#!/bin/bash

pokydir=<path/to>/poky
bitbake=${pokydir}/bitbake/bin/bitbake
builddir=<path/to/builds>/build-bitbake-xmlrpc

pushd <path/to/builds>
source ${pokydir}/oe-init-build-env ${builddir}

if BITBAKE_UI="knotty" ${bitbake} --server-only -B 0.0.0.0:0
then
   port=$(cat ${builddir}/bitbake.lock | awk '{print $2}')
   echo $port
   BITBAKE_UI="toasterui" BBSERVER="${port}" ${bitbake} quilt-native --token="" 2>&1 >> ${builddir}/toaster-ui.log
   status=$?
fi   

if [ status != 0 ]
then
    ps -ef | grep bitbake | awk '{print $2}' | xargs kill -9
    rm -rf ${builddir}/bitbake.*
    rm -rf ${builddir}/hashserve.sock
fi
exit $status
Comment 2 Tim Orling 2022-03-07 01:02:10 UTC
First reported in:
https://github.com/crops/toaster-container/issues/32
Comment 3 Tim Orling 2022-03-07 17:43:22 UTC
Patch from RP fixes the bb.server.xmlrpcclient AttributeError:


diff --git a/bitbake/lib/bb/server/xmlrpcserver.py b/bitbake/lib/bb/server/xmlrpcserver.py
index 2fa71be667..01f55538ae 100644
--- a/bitbake/lib/bb/server/xmlrpcserver.py
+++ b/bitbake/lib/bb/server/xmlrpcserver.py
@@ -11,6 +11,7 @@ import hashlib
 import time
 import inspect
 from xmlrpc.server import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler
+import bb.server.xmlrpcclient
 
 import bb


Which brings master to the same state as before 3.4:

Traceback (most recent call last):
  File "<path/to>/poky/bitbake/bin/bitbake", line 37, in <module>
    sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv),
  File "<path/to>/poky/bitbake/lib/bb/main.py", line 378, in bitbake_main
    return ui_module.main(server_connection.connection, server_connection.events,
  File "<path/to>/poky/bitbake/lib/bb/ui/toasterui.py", line 135, in main
    params.updateToServer(server, os.environ.copy())
  File "<path/to>/poky/bitbake/lib/bb/cookerdata.py", line 73, in updateToServer
    ret, error = server.runCommand(["updateConfig", options, environment, sys.argv])
  File "/usr/lib/python3.9/xmlrpc/client.py", line 1116, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python3.9/xmlrpc/client.py", line 1456, in __request
    response = self.__transport.request(
  File "/usr/lib/python3.9/xmlrpc/client.py", line 1160, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib/python3.9/xmlrpc/client.py", line 1176, in single_request
    return self.parse_response(resp)
  File "/usr/lib/python3.9/xmlrpc/client.py", line 1348, in parse_response
    return u.close()
  File "/usr/lib/python3.9/xmlrpc/client.py", line 662, in close
    raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault 1: "<class 'xml.parsers.expat.ExpatError'>:not well-formed (invalid token): line 181, column 17">
Comment 4 Tim Orling 2022-03-07 17:57:09 UTC
With knotty ui:

Traceback (most recent call last):
  File "<path/to>/poky/bitbake/bin/bitbake", line 37, in <module>
    sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv),
  File "<path/to>/poky/bitbake/lib/bb/main.py", line 378, in bitbake_main
    return ui_module.main(server_connection.connection, server_connection.events,
  File "<path/to>/poky/bitbake/lib/bb/ui/knotty.py", line 403, in main
    params.updateToServer(server, os.environ.copy())
  File "<path/to>/poky/bitbake/lib/bb/cookerdata.py", line 73, in updateToServer
    ret, error = server.runCommand(["updateConfig", options, environment, sys.argv])
  File "/usr/lib/python3.9/xmlrpc/client.py", line 1116, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python3.9/xmlrpc/client.py", line 1456, in __request
    response = self.__transport.request(
  File "/usr/lib/python3.9/xmlrpc/client.py", line 1160, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib/python3.9/xmlrpc/client.py", line 1176, in single_request
    return self.parse_response(resp)
  File "/usr/lib/python3.9/xmlrpc/client.py", line 1348, in parse_response
    return u.close()
  File "/usr/lib/python3.9/xmlrpc/client.py", line 662, in close
    raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault 1: "<class 'xml.parsers.expat.ExpatError'>:not well-formed (invalid token): line 181, column 17">
Comment 5 Tim Orling 2022-03-08 18:34:46 UTC
The code path that is currently failing is:
lib/bb/cookerdata.py.updateToServer sends an "updateConfig" command to the bitbake XMLRCP server [1]

This command in turn calls "updateConfigOpts" [2][3]

The suspected problem is that characters not permitted in XML 1.0 [4] need to be escaped in one of the payloads (options, environement, command line). Of these, environment is a likely culprit. An update to the expat library in Python itself is probably the root cause (all versions of Python 3.6+ have been getting expat security upgrades in dot releases). These are just hunches and not yet proven.


[1] https://git.openembedded.org/bitbake/tree/lib/bb/cookerdata.py#n73
[2] https://git.openembedded.org/bitbake/tree/lib/bb/command.py#n286
[3] https://git.openembedded.org/bitbake/tree/lib/bb/cooker.py#n448
[4] https://en.wikipedia.org/wiki/Valid_characters_in_XML#XML_1.0
Comment 6 Tim Orling 2022-03-08 18:46:28 UTC
The diff below is one of the places I was investigating the payloads.
The first commented out line is a test that the XML RPC server itself is functional.
The second commented out line spits out the payload variables and halts execution by raising an exception.

The next steps mentally would be to see what expat does when converting those payloads to XML (this is a streaming on-the-fly operation as I understand it).
And then... assuming this highlights an invalid character... figure out how to escape said character and sanitize the payload before streaming it through expat to XML.

diff --git a/bitbake/lib/bb/cookerdata.py b/bitbake/lib/bb/cookerdata.py
index fe5696c7048..969f3e14399 100644
--- a/bitbake/lib/bb/cookerdata.py
+++ b/bitbake/lib/bb/cookerdata.py
@@ -70,6 +70,8 @@ class ConfigParameters(object):
         options['default_loglevel'] = bb.msg.loggerDefaultLogLevel
         options['debug_domains'] = bb.msg.loggerDefaultDomains

+        #ret, error = server.runCommand(["getVariable", "BB_DEFAULT_TASK"])
+        #raise Exception("updateConfig, options{%s], environment[%s], sys.argv[%s]" % (options, environment, sys.argv))
         ret, error = server.runCommand(["updateConfig", options, environment, sys.argv])
         if error:
             raise Exception("Unable to update the server configuration with local parameters: %s" % error)
Comment 7 Tim Orling 2022-03-10 20:27:27 UTC
New commit from RP fixes Toaster:
https://patchwork.yoctoproject.org/project/bitbake/patch/20220310135344.612089-1-richard.purdie@linuxfoundation.org/

For unknown _reasons_ this is still failing on my local machine (Debian 11 and Python 3.9.2).

However, I can confirm that a web UI build of quilt-native succeeds using master-next and the following toaster-container distro combinations:

ubuntu-18.04
ubutnu-20.04
debian-10
debian-11

That is significant evidence that this issue has been fixed.
Comment 8 Tim Orling 2022-03-10 22:18:59 UTC
After rebooting the build machine and using a clean environment with only master-next, this is confirmed fixed.