With python-3.14 in gentoo bitbake fails to start with PicklingError ERROR: Attempting to set server environment: Unable to update the server configuration with local parameters: Traceback (most recent call last): File "/OE/build/luneos-whinlatter/bitbake/lib/bb/command.py", line 91, in runCommand result = command_method(self, commandline) File "/OE/build/luneos-whinlatter/bitbake/lib/bb/command.py", line 291, in updateConfig command.cooker.updateConfigOpts(options, environment, cmdline) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/OE/build/luneos-whinlatter/bitbake/lib/bb/cooker.py", line 471, in updateConfigOpts self.reset() ~~~~~~~~~~^^ File "/OE/build/luneos-whinlatter/bitbake/lib/bb/cooker.py", line 1741, in reset self.handlePRServ() ~~~~~~~~~~~~~~~~~^^ File "/OE/build/luneos-whinlatter/bitbake/lib/bb/cooker.py", line 306, in handlePRServ self.prhost = prserv.serv.auto_start(self.data) ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/OE/build/luneos-whinlatter/bitbake/lib/prserv/serv.py", line 444, in auto_start singleton.start() ~~~~~~~~~~~~~~~^^ File "/OE/build/luneos-whinlatter/bitbake/lib/prserv/serv.py", line 246, in start self.process = self.prserv.serve_as_process(log_level=logging.WARNING) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/OE/build/luneos-whinlatter/bitbake/lib/bb/asyncrpc/serv.py", line 402, in serve_as_process self.process.start() ~~~~~~~~~~~~~~~~~~^^ File "/usr/lib/python3.14/multiprocessing/process.py", line 121, in start self._popen = self._Popen(self) ~~~~~~~~~~~^^^^^^ File "/usr/lib/python3.14/multiprocessing/context.py", line 224, in _Popen return _default_context.get_context().Process._Popen(process_obj) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^ File "/usr/lib/python3.14/multiprocessing/context.py", line 300, in _Popen return Popen(process_obj) File "/usr/lib/python3.14/multiprocessing/popen_forkserver.py", line 35, in __init__ super().__init__(process_obj) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^ File "/usr/lib/python3.14/multiprocessing/popen_fork.py", line 20, in __init__ self._launch(process_obj) ~~~~~~~~~~~~^^^^^^^^^^^^^ File "/usr/lib/python3.14/multiprocessing/popen_forkserver.py", line 47, in _launch reduction.dump(process_obj, buf) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.14/multiprocessing/reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^ _pickle.PicklingError: Can't pickle local object <function AsyncServer.serve_as_process.<locals>.run at 0x7fa4972eaf00> when serializing dict item '_target' when serializing multiprocessing.context.Process state when serializing multiprocessing.context.Process object
This is likely caused by Python 3.14 changing the default way that multiprocess starts child processes away from the 'fork' method (see https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods). Bitbake relies heavily on the 'fork' method to share non-pickleable data with the child processes, so we'll need to change this for Python 3.14 compatability
And by "change" I mean that we'll need to instruct the python multiprocess to continue using the 'fork' method using either get_context() or set_start_method() so that our code remains compatible
Joshua is right with the straightforward fix - just force 'fork' method on 3.14+. I've got a patch ready that handles this with proper fallback for platforms that don't support fork. Tested it by reproducing the original error (forcing spawn method) then confirming the fix prevents the PicklingError. This is my first contribution so I figured I'd start small. Appreciate it!
I also have a patch I've been testing for this
Well what should we do then? I have no idea how to navigate things like this.
Feel free to send yours to the mailing list; We can look both over and see which we like better. Thanks
Fixed in bitbake 62be9113d98fccb347c6aa0a10d5c4ee2857f8b6
Created attachment 5131 [details] log.do_populate_sysroot.333633 with the exception
Even with the mentioned patch applied I'm seeing PicklingErrors as shown in attachment (to preserve formatting).
Do we need to update meta/lib/oe/utils.py:import multiprocessing as well? now with new bitbake version required by oe-core we might be able to use it. It works for me, sent to ML: https://lists.openembedded.org/g/openembedded-core/message/221758
Yep, I forgot to loop back around and check OE after the bitbake fix. Thanks.