| Summary: | bitbake cannot connect to xmlrpc socket (BBSERVER=0.0.0.0:port) | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] BitBake | Reporter: | Tim Orling <tim.orling> |
| Component: | bitbake | Assignee: | Richard Purdie <richard.purdie> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | major | ||
| Priority: | High | CC: | david.reyna, poky.bs.watcher, poky.watcher, randy.macleod |
| Version: | 4.1 | ||
| Target Milestone: | 4.0 M4 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | Don't know | |
|
Description
Tim Orling
2022-03-07 00:29:55 UTC
A test script I attempted to use with git bisect:
--------
#!/bin/bash
pokydir=<path/to>/poky
bitbake=${pokydir}/bitbake/bin/bitbake
builddir=<path/to/builds>/build-bitbake-xmlrpc
pushd <path/to/builds>
source ${pokydir}/oe-init-build-env ${builddir}
if BITBAKE_UI="knotty" ${bitbake} --server-only -B 0.0.0.0:0
then
port=$(cat ${builddir}/bitbake.lock | awk '{print $2}')
echo $port
BITBAKE_UI="toasterui" BBSERVER="${port}" ${bitbake} quilt-native --token="" 2>&1 >> ${builddir}/toaster-ui.log
status=$?
fi
if [ status != 0 ]
then
ps -ef | grep bitbake | awk '{print $2}' | xargs kill -9
rm -rf ${builddir}/bitbake.*
rm -rf ${builddir}/hashserve.sock
fi
exit $status
First reported in: https://github.com/crops/toaster-container/issues/32 Patch from RP fixes the bb.server.xmlrpcclient AttributeError:
diff --git a/bitbake/lib/bb/server/xmlrpcserver.py b/bitbake/lib/bb/server/xmlrpcserver.py
index 2fa71be667..01f55538ae 100644
--- a/bitbake/lib/bb/server/xmlrpcserver.py
+++ b/bitbake/lib/bb/server/xmlrpcserver.py
@@ -11,6 +11,7 @@ import hashlib
import time
import inspect
from xmlrpc.server import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler
+import bb.server.xmlrpcclient
import bb
Which brings master to the same state as before 3.4:
Traceback (most recent call last):
File "<path/to>/poky/bitbake/bin/bitbake", line 37, in <module>
sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv),
File "<path/to>/poky/bitbake/lib/bb/main.py", line 378, in bitbake_main
return ui_module.main(server_connection.connection, server_connection.events,
File "<path/to>/poky/bitbake/lib/bb/ui/toasterui.py", line 135, in main
params.updateToServer(server, os.environ.copy())
File "<path/to>/poky/bitbake/lib/bb/cookerdata.py", line 73, in updateToServer
ret, error = server.runCommand(["updateConfig", options, environment, sys.argv])
File "/usr/lib/python3.9/xmlrpc/client.py", line 1116, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python3.9/xmlrpc/client.py", line 1456, in __request
response = self.__transport.request(
File "/usr/lib/python3.9/xmlrpc/client.py", line 1160, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib/python3.9/xmlrpc/client.py", line 1176, in single_request
return self.parse_response(resp)
File "/usr/lib/python3.9/xmlrpc/client.py", line 1348, in parse_response
return u.close()
File "/usr/lib/python3.9/xmlrpc/client.py", line 662, in close
raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault 1: "<class 'xml.parsers.expat.ExpatError'>:not well-formed (invalid token): line 181, column 17">
With knotty ui:
Traceback (most recent call last):
File "<path/to>/poky/bitbake/bin/bitbake", line 37, in <module>
sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv),
File "<path/to>/poky/bitbake/lib/bb/main.py", line 378, in bitbake_main
return ui_module.main(server_connection.connection, server_connection.events,
File "<path/to>/poky/bitbake/lib/bb/ui/knotty.py", line 403, in main
params.updateToServer(server, os.environ.copy())
File "<path/to>/poky/bitbake/lib/bb/cookerdata.py", line 73, in updateToServer
ret, error = server.runCommand(["updateConfig", options, environment, sys.argv])
File "/usr/lib/python3.9/xmlrpc/client.py", line 1116, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python3.9/xmlrpc/client.py", line 1456, in __request
response = self.__transport.request(
File "/usr/lib/python3.9/xmlrpc/client.py", line 1160, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib/python3.9/xmlrpc/client.py", line 1176, in single_request
return self.parse_response(resp)
File "/usr/lib/python3.9/xmlrpc/client.py", line 1348, in parse_response
return u.close()
File "/usr/lib/python3.9/xmlrpc/client.py", line 662, in close
raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault 1: "<class 'xml.parsers.expat.ExpatError'>:not well-formed (invalid token): line 181, column 17">
The code path that is currently failing is: lib/bb/cookerdata.py.updateToServer sends an "updateConfig" command to the bitbake XMLRCP server [1] This command in turn calls "updateConfigOpts" [2][3] The suspected problem is that characters not permitted in XML 1.0 [4] need to be escaped in one of the payloads (options, environement, command line). Of these, environment is a likely culprit. An update to the expat library in Python itself is probably the root cause (all versions of Python 3.6+ have been getting expat security upgrades in dot releases). These are just hunches and not yet proven. [1] https://git.openembedded.org/bitbake/tree/lib/bb/cookerdata.py#n73 [2] https://git.openembedded.org/bitbake/tree/lib/bb/command.py#n286 [3] https://git.openembedded.org/bitbake/tree/lib/bb/cooker.py#n448 [4] https://en.wikipedia.org/wiki/Valid_characters_in_XML#XML_1.0 The diff below is one of the places I was investigating the payloads.
The first commented out line is a test that the XML RPC server itself is functional.
The second commented out line spits out the payload variables and halts execution by raising an exception.
The next steps mentally would be to see what expat does when converting those payloads to XML (this is a streaming on-the-fly operation as I understand it).
And then... assuming this highlights an invalid character... figure out how to escape said character and sanitize the payload before streaming it through expat to XML.
diff --git a/bitbake/lib/bb/cookerdata.py b/bitbake/lib/bb/cookerdata.py
index fe5696c7048..969f3e14399 100644
--- a/bitbake/lib/bb/cookerdata.py
+++ b/bitbake/lib/bb/cookerdata.py
@@ -70,6 +70,8 @@ class ConfigParameters(object):
options['default_loglevel'] = bb.msg.loggerDefaultLogLevel
options['debug_domains'] = bb.msg.loggerDefaultDomains
+ #ret, error = server.runCommand(["getVariable", "BB_DEFAULT_TASK"])
+ #raise Exception("updateConfig, options{%s], environment[%s], sys.argv[%s]" % (options, environment, sys.argv))
ret, error = server.runCommand(["updateConfig", options, environment, sys.argv])
if error:
raise Exception("Unable to update the server configuration with local parameters: %s" % error)
New commit from RP fixes Toaster: https://patchwork.yoctoproject.org/project/bitbake/patch/20220310135344.612089-1-richard.purdie@linuxfoundation.org/ For unknown _reasons_ this is still failing on my local machine (Debian 11 and Python 3.9.2). However, I can confirm that a web UI build of quilt-native succeeds using master-next and the following toaster-container distro combinations: ubuntu-18.04 ubutnu-20.04 debian-10 debian-11 That is significant evidence that this issue has been fixed. After rebooting the build machine and using a clean environment with only master-next, this is confirmed fixed. |