Bug 11929 - preparing tinfoil twice with config_only=False is failing
Summary: preparing tinfoil twice with config_only=False is failing
Status: RESOLVED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: 2.4
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 2.4 M4
Assignee: Paul Eggleton
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks: 10805
  Show dependency tree
 
Reported: 2017-08-14 17:12 UTC by Leonardo Sandoval Gonzalez
Modified: 2017-09-15 04:52 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Leonardo Sandoval Gonzalez 2017-08-14 17:12:09 UTC
Running the sample example [1] with config_only = False and preparing tinfoil twice in the same python process is causing the following error:

NOTE: Starting bitbake server...
NOTE: Starting bitbake server...
Parsing recipes: 100% |#########################################################################################################################################################################################################| Time: 0:00:02
Parsing of 809 .bb files complete (0 cached, 809 parsed). 1258 targets, 44 skipped, 0 masked, 0 errors.
TMPDIR is "/home/lsandov1/poky-testing/poky/b2/tmp"
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection... (Traceback (most recent call last):
  File "/home/lsandov1/poky-testing/poky/bitbake/lib/bb/main.py", line 451, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/home/lsandov1/poky-testing/poky/bitbake/lib/bb/server/process.py", line 448, in connectProcessServer
    server_connection.connection.updateFeatureSet(featureset)
  File "/home/lsandov1/poky-testing/poky/bitbake/lib/bb/server/process.py", line 322, in updateFeatureSet
    _, error = self.runCommand(["setFeatures", featureset])
  File "/home/lsandov1/poky-testing/poky/bitbake/lib/bb/server/process.py", line 319, in runCommand
    return self.recv.get()
  File "/home/lsandov1/poky-testing/poky/bitbake/lib/bb/server/process.py", line 562, in get
    res = self.reader.recv_bytes()
  File "/usr/lib64/python3.4/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/lib64/python3.4/multiprocessing/connection.py", line 416, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib64/python3.4/multiprocessing/connection.py", line 389, in _recv
    raise EOFError
EOFError
)
.
.
.



[1] https://wiki.yoctoproject.org/wiki/TipsAndTricks/Tinfoil
Comment 1 Leonardo Sandoval Gonzalez 2017-08-14 17:15:20 UTC
import os
import sys

# Set up sys.path to let us import tinfoil                                                                                                                                                                                                     
scripts_path = os.path.dirname(os.path.realpath(__file__))
lib_path = scripts_path + '/lib'
sys.path.insert(0, lib_path)
import scriptpath
scriptpath.add_bitbake_lib_path()

import bb.tinfoil

def auto(config_only):
    with bb.tinfoil.Tinfoil() as tinfoil:
        tinfoil.prepare(config_only=config_only)
        tmpdir = tinfoil.config_data.getVar('TMPDIR')
        print('TMPDIR is "%s"' % tmpdir)
        data = tinfoil.cooker.recipecaches[''].pkg_fn.items()
        # print(data)                                                   

    with bb.tinfoil.Tinfoil() as tinfoil:
        tinfoil.prepare(config_only=config_only)
        tmpdir = tinfoil.config_data.getVar('TMPDIR')
        print('TMPDIR is "%s"' % tmpdir)

config_only = False
auto(config_only=config_only)
Comment 2 Richard Purdie 2017-08-14 21:40:53 UTC
Its trying to connect two UIs to one server which isn't supported. I agree the error should be better.
Comment 3 Paul Eggleton 2017-08-22 02:28:00 UTC
This seems to be working fine with latest master.
Comment 4 Leonardo Sandoval Gonzalez 2017-08-22 13:45:42 UTC
(In reply to comment #3)
> This seems to be working fine with latest master.

The must be something different between your setup and mine.

Can you try the scritpt (tinfoil-example.py) contained in [1] ?

I still see the issue. 

[1] http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=lsandov1/tinfoil-example
Comment 5 Paul Eggleton 2017-08-22 20:31:02 UTC
Yep, just ran the script you linked to, same result i.e.

---------------- snip ----------------
$ tinfoil-example.py
NOTE: Starting bitbake server...
NOTE: Starting bitbake server...
Loading cache: 100% |############################################| Time: 0:00:00
Loaded 2108 entries from dependency cache.
Parsing recipes: 100% |##########################################| Time: 0:00:04
Parsing of 1514 .bb files complete (1430 cached, 84 parsed). 2108 targets, 102 skipped, 0 masked, 0 errors.
TMPDIR is "/home/paul/poky/poky/build/tmp"
NOTE: Starting bitbake server...
NOTE: Starting bitbake server...
Loading cache: 100% |############################################| Time: 0:00:00
Loaded 2108 entries from dependency cache.
TMPDIR is "/home/paul/poky/poky/build/tmp"
---------------- snip ----------------

I can try it on a different machine perhaps.
Comment 6 Leonardo Sandoval Gonzalez 2017-08-23 16:18:35 UTC
This patch fixes the problem

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/wip5&id=693f3528ffbcc4e4c54c7c362c62fe02f708ce01

so removing the socket **before** cooker's shutdown fixes the problem.

Paul, I let you double check this so you can decide what to do with the status but for me is resolved/fixed.
Comment 7 Paul Eggleton 2017-08-23 23:14:54 UTC
It does seem to resolve the EOFError, but there's still an awkward reconnect (and delay to retry) - so it's an improvement but not where we'd ideally like to get it to yet.
Comment 8 Paul Eggleton 2017-08-24 11:10:41 UTC
Richard's latest patches resolve the delay as well.
Comment 9 Richard Purdie 2017-08-24 13:05:42 UTC
I've merged http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=2b453483d4610945c6156cc7f472ffae803cb28a and http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=4152c472430db1b32c943026f8c976bb99b9717b which at least hopefully improves the situation (shorter retry and no tracebacks). Is it enough? Not sure...
Comment 10 Paul Eggleton 2017-09-15 04:52:16 UTC
I think we'll call this done for the moment, and if there are other behavioural improvements we can make we'll deal with them separately.