Toaster bitbake/lib/toaster/bldcontrol/localhostbecontroller.py#n448 starts an XMLRPC server instance of bitbake with (the equivalent of): BITBAKE_UI="knotty" bitbake --server-only -B 0.0.0.0:0 (This is successful and python code confirms that the XMLRPC server is functional) After this, Toaster bitbake/lib/toaster/bldcontrol/localhostbecontroller.py#n501 tries to run a build connecting to this server with (the equivalent of): BITBAKE_UI="toasterui" BBSERVER="0.0.0.0:<port>" bitbake quilt-native --token "" (It does not matter if the ui is "knotty" instead). Since somewhere in 3.4, this fails with WARNING: Could not connect to server at 0.0.0.0:<port> (<Fault 1: "<class 'AttributeError'>:module 'bb.server' has no attribute 'xmlrpcclient'">) Prior to 3.4, it fails with: xmlrpc.client.Fault: <Fault 1: "<class 'xml.parsers.expat.ExpatError'>:not well-formed (invalid token): line 181, column 17"> which is probably unrelated (more likely a python strings/bytes problem).
A test script I attempted to use with git bisect: -------- #!/bin/bash pokydir=<path/to>/poky bitbake=${pokydir}/bitbake/bin/bitbake builddir=<path/to/builds>/build-bitbake-xmlrpc pushd <path/to/builds> source ${pokydir}/oe-init-build-env ${builddir} if BITBAKE_UI="knotty" ${bitbake} --server-only -B 0.0.0.0:0 then port=$(cat ${builddir}/bitbake.lock | awk '{print $2}') echo $port BITBAKE_UI="toasterui" BBSERVER="${port}" ${bitbake} quilt-native --token="" 2>&1 >> ${builddir}/toaster-ui.log status=$? fi if [ status != 0 ] then ps -ef | grep bitbake | awk '{print $2}' | xargs kill -9 rm -rf ${builddir}/bitbake.* rm -rf ${builddir}/hashserve.sock fi exit $status
First reported in: https://github.com/crops/toaster-container/issues/32
Patch from RP fixes the bb.server.xmlrpcclient AttributeError: diff --git a/bitbake/lib/bb/server/xmlrpcserver.py b/bitbake/lib/bb/server/xmlrpcserver.py index 2fa71be667..01f55538ae 100644 --- a/bitbake/lib/bb/server/xmlrpcserver.py +++ b/bitbake/lib/bb/server/xmlrpcserver.py @@ -11,6 +11,7 @@ import hashlib import time import inspect from xmlrpc.server import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler +import bb.server.xmlrpcclient import bb Which brings master to the same state as before 3.4: Traceback (most recent call last): File "<path/to>/poky/bitbake/bin/bitbake", line 37, in <module> sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv), File "<path/to>/poky/bitbake/lib/bb/main.py", line 378, in bitbake_main return ui_module.main(server_connection.connection, server_connection.events, File "<path/to>/poky/bitbake/lib/bb/ui/toasterui.py", line 135, in main params.updateToServer(server, os.environ.copy()) File "<path/to>/poky/bitbake/lib/bb/cookerdata.py", line 73, in updateToServer ret, error = server.runCommand(["updateConfig", options, environment, sys.argv]) File "/usr/lib/python3.9/xmlrpc/client.py", line 1116, in __call__ return self.__send(self.__name, args) File "/usr/lib/python3.9/xmlrpc/client.py", line 1456, in __request response = self.__transport.request( File "/usr/lib/python3.9/xmlrpc/client.py", line 1160, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib/python3.9/xmlrpc/client.py", line 1176, in single_request return self.parse_response(resp) File "/usr/lib/python3.9/xmlrpc/client.py", line 1348, in parse_response return u.close() File "/usr/lib/python3.9/xmlrpc/client.py", line 662, in close raise Fault(**self._stack[0]) xmlrpc.client.Fault: <Fault 1: "<class 'xml.parsers.expat.ExpatError'>:not well-formed (invalid token): line 181, column 17">
With knotty ui: Traceback (most recent call last): File "<path/to>/poky/bitbake/bin/bitbake", line 37, in <module> sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv), File "<path/to>/poky/bitbake/lib/bb/main.py", line 378, in bitbake_main return ui_module.main(server_connection.connection, server_connection.events, File "<path/to>/poky/bitbake/lib/bb/ui/knotty.py", line 403, in main params.updateToServer(server, os.environ.copy()) File "<path/to>/poky/bitbake/lib/bb/cookerdata.py", line 73, in updateToServer ret, error = server.runCommand(["updateConfig", options, environment, sys.argv]) File "/usr/lib/python3.9/xmlrpc/client.py", line 1116, in __call__ return self.__send(self.__name, args) File "/usr/lib/python3.9/xmlrpc/client.py", line 1456, in __request response = self.__transport.request( File "/usr/lib/python3.9/xmlrpc/client.py", line 1160, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib/python3.9/xmlrpc/client.py", line 1176, in single_request return self.parse_response(resp) File "/usr/lib/python3.9/xmlrpc/client.py", line 1348, in parse_response return u.close() File "/usr/lib/python3.9/xmlrpc/client.py", line 662, in close raise Fault(**self._stack[0]) xmlrpc.client.Fault: <Fault 1: "<class 'xml.parsers.expat.ExpatError'>:not well-formed (invalid token): line 181, column 17">
The code path that is currently failing is: lib/bb/cookerdata.py.updateToServer sends an "updateConfig" command to the bitbake XMLRCP server [1] This command in turn calls "updateConfigOpts" [2][3] The suspected problem is that characters not permitted in XML 1.0 [4] need to be escaped in one of the payloads (options, environement, command line). Of these, environment is a likely culprit. An update to the expat library in Python itself is probably the root cause (all versions of Python 3.6+ have been getting expat security upgrades in dot releases). These are just hunches and not yet proven. [1] https://git.openembedded.org/bitbake/tree/lib/bb/cookerdata.py#n73 [2] https://git.openembedded.org/bitbake/tree/lib/bb/command.py#n286 [3] https://git.openembedded.org/bitbake/tree/lib/bb/cooker.py#n448 [4] https://en.wikipedia.org/wiki/Valid_characters_in_XML#XML_1.0
The diff below is one of the places I was investigating the payloads. The first commented out line is a test that the XML RPC server itself is functional. The second commented out line spits out the payload variables and halts execution by raising an exception. The next steps mentally would be to see what expat does when converting those payloads to XML (this is a streaming on-the-fly operation as I understand it). And then... assuming this highlights an invalid character... figure out how to escape said character and sanitize the payload before streaming it through expat to XML. diff --git a/bitbake/lib/bb/cookerdata.py b/bitbake/lib/bb/cookerdata.py index fe5696c7048..969f3e14399 100644 --- a/bitbake/lib/bb/cookerdata.py +++ b/bitbake/lib/bb/cookerdata.py @@ -70,6 +70,8 @@ class ConfigParameters(object): options['default_loglevel'] = bb.msg.loggerDefaultLogLevel options['debug_domains'] = bb.msg.loggerDefaultDomains + #ret, error = server.runCommand(["getVariable", "BB_DEFAULT_TASK"]) + #raise Exception("updateConfig, options{%s], environment[%s], sys.argv[%s]" % (options, environment, sys.argv)) ret, error = server.runCommand(["updateConfig", options, environment, sys.argv]) if error: raise Exception("Unable to update the server configuration with local parameters: %s" % error)
New commit from RP fixes Toaster: https://patchwork.yoctoproject.org/project/bitbake/patch/20220310135344.612089-1-richard.purdie@linuxfoundation.org/ For unknown _reasons_ this is still failing on my local machine (Debian 11 and Python 3.9.2). However, I can confirm that a web UI build of quilt-native succeeds using master-next and the following toaster-container distro combinations: ubuntu-18.04 ubutnu-20.04 debian-10 debian-11 That is significant evidence that this issue has been fixed.
After rebooting the build machine and using a clean environment with only master-next, this is confirmed fixed.