Bug 1017

As a data point, I tried building core-image-sato master today for atom-pc:
61018fda65fcb7a08d74b57abfe8e8809fcdaae0

I included meta and meta-yocto in meta layers. I deleted tmp and sstate-cache. 

BB_NUMBER_THREADS = "24"
PARALLEL_MAKE = "-j 24"

There was a lot of disk I/O from background rm processes.

It took my 5 runs to get it to complete parsing the recipes and start running tasks.

Crazy request but this code just changed heavily in master with changes from upstream. Could you retest please?

Will do so tonight.

Well, turns out I was a few commits behind on my build machine. Pulling the latest master (0788081ac7a04f94c349b2673ed6ae122aa7d1a8) and trying to build core-image-sato yields an immediate new kind of failure:


Pseudo is not present but is required, building this first before the main build
Traceback (most recent call last):
  File "/home/dvhart/source/poky.git/bitbake/bin/bitbake", line 34, in <module>
    import bb
  File "/home/dvhart/source/poky.git/bitbake/lib/bb/__init__.py", line 74, in <module>
    from bb import fetch2 as fetch
  File "/home/dvhart/source/poky.git/bitbake/lib/bb/fetch2/__init__.py", line 31, in <module>
    import bb.data, bb.persist_data, bb.utils
  File "/home/dvhart/source/poky.git/bitbake/lib/bb/persist_data.py", line 29, in <module>
    from bb.compat import total_ordering
ImportError: No module named compat
/home/dvhart/source/poky.git/bitbake/lib/bb/event.py:100: RuntimeWarning: Parent module 'bb' not found while handling absolute import
  from bb.msg import BBLogFormatter
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/home/dvhart/source/poky.git/bitbake/lib/bb/event.py", line 100, in print_ui_queue
    from bb.msg import BBLogFormatter
  File "/home/dvhart/source/poky.git/bitbake/lib/bb/__init__.py", line 74, in <module>
    from bb import fetch2 as fetch
  File "/home/dvhart/source/poky.git/bitbake/lib/bb/fetch2/__init__.py", line 31, in <module>
    import bb.data, bb.persist_data, bb.utils
  File "/home/dvhart/source/poky.git/bitbake/lib/bb/persist_data.py", line 29, in <module>
    from bb.compat import total_ordering
ImportError: No module named compat
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/home/dvhart/source/poky.git/bitbake/lib/bb/event.py", line 100, in print_ui_queue
    from bb.msg import BBLogFormatter
  File "/home/dvhart/source/poky.git/bitbake/lib/bb/__init__.py", line 74, in <module>
    from bb import fetch2 as fetch
  File "/home/dvhart/source/poky.git/bitbake/lib/bb/fetch2/__init__.py", line 31, in <module>
    import bb.data, bb.persist_data, bb.utils
  File "/home/dvhart/source/poky.git/bitbake/lib/bb/persist_data.py", line 29, in <module>
    from bb.compat import total_ordering
ImportError: No module named compat

Bisecting the last 45 patches unfortunately yielded a pair of different sorts of failures. The one reported above as well as:


Pseudo is not present but is required, building this first before the main build
Traceback (most recent call last):
  File "/home/dvhart/source/poky.git/bitbake/bin/bitbake", line 219, in <module>
    ret = main()
  File "/home/dvhart/source/poky.git/bitbake/bin/bitbake", line 193, in main
    cooker = bb.cooker.BBCooker(configuration, server)
  File "/home/dvhart/source/poky.git/bitbake/lib/bb/cooker.py", line 82, in __init__
    self.parseConfigurationFiles(self.configuration.file)
  File "/home/dvhart/source/poky.git/bitbake/lib/bb/cooker.py", line 628, in parseConfigurationFiles
    bb.fetch.fetcher_init(self.configuration.data)
  File "/home/dvhart/source/poky.git/bitbake/lib/bb/fetch2/__init__.py", line 225, in fetcher_init
    pd = persist_data.persist(d)
NameError: global name 'persist_data' is not defined


We should be ensuring our patch series are bisectable.

I think this is a separate issue. We're missing a module from upstream BitBake[1] which was added by a patch from Chris Larson[2].

1. http://git.openembedded.org/cgit.cgi/bitbake/tree/lib/bb/compat.py
2/ http://git.openembedded.org/cgit.cgi/bitbake/commit/lib/bb/compat.py?id=1190406c526c7bb7cf415867be83e0403812a7dd

compat.py is a separate issue which has been fixed. Changes should always be bisectable, I agree but in this case I failed to merge all the patches in the correct order and didn't realise until it was too late to be able to correct the series.

If you could test with the compat.py file added I'd appreciate it, thanks and sorry about this mess.

Sorry about the last two comments, those should have gone to the list, obviously not related.

With master as of:

commit 506241c914383c20561144069ba0b0d3850ce8cc
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date:   Sat May 7 08:49:28 2011 +0100

    bitbake: Add missing file


I did 5 clean recipe parsings in a row for each of qemux86 and n450:

$ rm -rf tmp; bitbake core-image-sato

And saw no "database is locked" errors. Perhaps this is behind us? I suggest we close with the understand that without an explicit fix we may have just changed the race window sufficiently to mask the issue for a time. We may need to reopen.

Closing as we're not seeing reports of these anymore. Please reopen or refile if these occur again.