Bug 1332 - pseudo segfault in the 2nd build from hob
Summary: pseudo segfault in the 2nd build from hob
Status: RESOLVED FIXED
Alias: None
Product: Pseudo
Classification: Yocto Project Subprojects
Component: pseudo (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: High critical
Target Milestone: 1.1
Assignee: Paul Eggleton
QA Contact:
URL:
Whiteboard: Fix posted for review (5 sep)
Depends on:
Blocks: 1285
  Show dependency tree
 
Reported: 2011-08-04 16:40 UTC by Jessica
Modified: 2011-09-05 14:44 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: ---


Attachments
Patch to aid debugging in hob (1.91 KB, patch)
2011-08-30 11:55 UTC, Joshua Lock - Disabled
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jessica 2011-08-04 16:40:12 UTC
When trying to run 2nd build from hob, got the following error:

/home/jzhang/yocto-latest/scripts/bitbake: line 86: 30647 Segmentation fault      PSEUDO_BINDIR=$PSEUDOBINDIR PSEUDO_LIBDIR=$PSEUDOBINDIR/../lib/pseudo/lib PSEUDO_PREFIX=$PSEUDOBINDIR/../../ PSEUDO_DISABLED=1 $PSEUDOBINDIR/pseudo $BITBAKE $@

I'm also attaching the core file with the bug.

To reproduce the bug, under your build directory instead of using bitbake, run hob.

After hob comes up, leave the default selections e.g. Machine: qemux86, and select Base image: core-image-minimal, after the packages are populated click on the "Bake" button. After the "Build complete" prompt out, select "Edit Image", which will bring u back to the main hob view and reselect "core-image-minimal" and click "Bake", after several tasks, you'll see the segfault.
Comment 1 Jessica 2011-08-04 16:45:00 UTC
seems the core file is too big to be attached.

Also, make sure you have following patch in the code:

$ git diff
diff --git a/lib/bb/ui/hob.py b/lib/bb/ui/hob.py
index b5c2342..8af7740 100644
--- a/lib/bb/ui/hob.py
+++ b/lib/bb/ui/hob.py
@@ -446,6 +446,7 @@ class MainWindow (gtk.Window):
         self.nb.set_current_page(0)

     def build_complete_cb(self, running_build):
+        self.handler.building = None
         self.stopping = False
         self.back.connect("clicked", self.back_button_clicked_cb)
         self.back.set_sensitive(True)
Comment 2 Mark Hatle 2011-08-11 11:23:51 UTC
This is not looking like a pseudo problems, at least not directly.

The cooker.log from the first "good" run:


OE Build Configuration:
BB_VERSION        = "1.13.3"
TARGET_ARCH       = "arm"
TARGET_OS         = "linux-gnueabi"
MACHINE           = "qemuarm"
DISTRO            = ""
DISTRO_VERSION    = ""
TUNE_FEATURES     = "armv5 dsp thumb arm926ejs"
TARGET_FPU        = "soft"
meta              = "mhatle/fix_1352:68661582ede47f7b70999764ea80fe3e5e6aa78d"


The log from the subsequent run:


pseudo: You must set the PSEUDO_PREFIX environment variable to run pseudo.
pseudo: You must set the PSEUDO_PREFIX environment variable to run pseudo.
pseudo: You must set the PSEUDO_PREFIX environment variable to run pseudo.

OE Build Configuration:
BB_VERSION        = "1.13.3"
TARGET_ARCH       = "arm"
TARGET_OS         = "linux-gnueabi"
MACHINE           = "qemuarm"
DISTRO            = ""
DISTRO_VERSION    = ""
TUNE_FEATURES     = "armv5 dsp thumb arm926ejs"
TARGET_FPU        = "soft"
meta              = "<unknown>:<unknown>"

The three pseudo messages indicate that the environment was cleared and pseudo was trying to be run way too early in the build.  The following messages indicate that the meta data couldn't be found correctly..

The core indicates the failure occurred in:  (note pseudo should be in memory, but inactive at this point)

python /home/mhatle/git/oss/oe-core/bitbake/bin/bitbake -R conf/hob.local.conf

The backtrace:


#0  0x0000003c534966f8 in tupleitem (a=0x3689890, i=0) at Objects/tupleobject.c:337
#1  0x0000003c534970e0 in tuplesubscript (self=0x3689890, item=<value optimized out>)
    at Objects/tupleobject.c:658
#2  0x0000003c534db99b in PyEval_EvalFrameEx (f=<value optimized out>, 
    throwflag=<value optimized out>) at Python/ceval.c:1294
#3  0x0000003c534df4e9 in PyEval_EvalCodeEx (co=0x1936990, 
    globals=<value optimized out>, locals=<value optimized out>, 
    args=<value optimized out>, argcount=2, kws=0x3725248, kwcount=0, defs=
    0x19fe1e8, defcount=1, closure=0x0) at Python/ceval.c:3026
#4  0x0000003c534dd897 in fast_function (f=<value optimized out>, 
    throwflag=<value optimized out>) at Python/ceval.c:3860
#5  call_function (f=<value optimized out>, throwflag=<value optimized out>)
    at Python/ceval.c:3785
#6  PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>)
    at Python/ceval.c:2445
#7  0x0000003c534de312 in fast_function (f=<value optimized out>, 
    throwflag=<value optimized out>) at Python/ceval.c:3850
#8  call_function (f=<value optimized out>, throwflag=<value optimized out>)
    at Python/ceval.c:3785
#9  PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>)
    at Python/ceval.c:2445
#10 0x0000003c534df4e9 in PyEval_EvalCodeEx (co=0x190e558, 
    globals=<value optimized out>, locals=<value optimized out>, 
    args=<value optimized out>, argcount=4, kws=0x0, kwcount=0, defs=0x0, 
    defcount=0, closure=0x0) at Python/ceval.c:3026
#11 0x0000003c5346df70 in function_call (func=<function at remote 0x191a0c8>, arg=
    (<HobHandler(building='image', generating=False, image_dir='/tmp/hob-images', current_command=None, server=<ServerProxy(_ServerProxy__transport=<Transport(_use_datetime=0, verbose=0) at remote 0x365e7a0>, _ServerProxy__handler='/', _ServerProxy__host='127.0.0.1:50989', _ServerProxy__verbose=0, _ServerProxy__allow_none=True, _ServerProxy__encoding=None) at remote 0x365ef38>, build_toolchain_headers=False, build_toolchain=False, image_output_types=['tar.bz2', 'ext3'], model=<TaskListModel(packages=<unknown at remote 0x7fdf74268690>, images=<unknown at remote 0x7fdf74268730>, tasks=<unknown at remote 0x7fdf74268820>, contents=<unknown at remote 0x7fdf74268780>, selected_image='core-image-sato') at remote 0x3059500>, build_queue=[], current_phase='building') at remote 0x30595a0>, <BBUIEventQueue(EventHandle=1, eventQueueNotify=<_Event(_Verbose__verbose=False, _Event__flag=False, _Event__cond=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x1844b10>, acquire=<built-in method acquire o...(truncated), kw=0x0) at Objects/funcobject.c:524
#12 0x0000003c53443db3 in PyObject_Call (func=<function at remote 0x191a0c8>, 
    arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:2492

#13 0x0000003c53458cff in instancemethod_call (func=<function at remote 0x191a0c8>, 
    arg=
    (<HobHandler(building='image', generating=False, image_dir='/tmp/hob-images', current_command=None, server=<ServerProxy(_ServerProxy__transport=<Transport(_use_datetime=0, verbose=0) at remote 0x365e7a0>, _ServerProxy__handler='/', _ServerProxy__host='127.0.0.1:50989', _ServerProxy__verbose=0, _ServerProxy__allow_none=True, _ServerProxy__encoding=None) at remote 0x365ef38>, build_toolchain_headers=False, build_toolchain=False, image_output_types=['tar.bz2', 'ext3'], model=<TaskListModel(packages=<unknown at remote 0x7fdf74268690>, images=<unknown at remote 0x7fdf74268730>, tasks=<unknown at remote 0x7fdf74268820>, contents=<unknown at remote 0x7fdf74268780>, selected_image='core-image-sato') at remote 0x3059500>, build_queue=[], current_phase='building') at remote 0x30595a0>, <BBUIEventQueue(EventHandle=1, eventQueueNotify=<_Event(_Verbose__verbose=False, _Event__flag=False, _Event__cond=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x1844b10>, acquire=<built-in method acquire o...(truncated), kw=0x0) at Objects/classobject.c:2579
#14 0x0000003c53443db3 in PyObject_Call (func=<instancemethod at remote 0x368b9b0>, 
    arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:2492
#15 0x0000003c534d7ba3 in PyEval_CallObjectWithKeywords (func=
    <instancemethod at remote 0x368b9b0>, arg=
    (<BBUIEventQueue(EventHandle=1, eventQueueNotify=<_Event(_Verbose__verbose=False, _Event__flag=False, _Event__cond=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x1844b10>, acquire=<built-in method acquire of thread.lock object at remote 0x1844b10>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x1844b10>) at remote 0x1c81110>) at remote 0x1b85650>, eventQueueLock=<thread.lock at remote 0x1844af8>, port=32778, host='127.0.0.1', t=<Thread(_Thread__ident=140597806331648, run=<instancemethod at remote 0x30591e0>, _Thread__block=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x1844b40>, acquire=<built-in method acquire of thread.lock object at remote 0x1844b40>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x1844b40>) at remote 0x32d3d10>, _Thread__name='Thread-1', _Thread__daemonic=True, _Thread__started=<_Event(_Verbose__verbose=False, _Event__flag=True, _Event__con...(truncated), kw=<value optimized out>) at Python/ceval.c:3633
#16 0x00007fdf7cdd51bb in ?? ()
#17 0x0000000001f38190 in ?? ()
#18 0x0000000000000000 in ?? ()

The code where the failure occured:

332     {
333             if (i < 0 || i >= Py_SIZE(a)) {
334                     PyErr_SetString(PyExc_IndexError, "tuple index out of range");
335                     return NULL;
336             }
337             Py_INCREF(a->ob_item[i]);
338             return a->ob_item[i];
339     }


a = 0x3689890, a->ob_item = 0x0, i = 0
Comment 3 Mark Hatle 2011-08-11 14:27:56 UTC
Reassigning to Joshua, he indicated on IRC he would investigate the hob refresh issue.  If this persists after that has been resolved, please reassign to me.
Comment 4 Joshua Lock - Disabled 2011-08-17 08:54:15 UTC
This is confusing to me because the only code we * explicitly* call from hob which affects the environment is the reparse command, however this is run at least once before the first build and doesn't have an adverse effect. I wonder if there's some tear down done on build completion that unsets parts of the environment?

The sequence of commands for a typical build will consist of 
i) buildTargetsTree
ii) resetCooker & reparseFiles (triggered at least once before the build when the custom recipe of the image contents is written to ensure it's included, may have been triggered more times by editing preferences etc)
iii) buildTargets

Commands are executed by the run_next_command() method of bb/ui/crumbs/hobeventhandler.py usually triggered by an explicit user action or a CommandCompleted event from the Cooker.

Once the build has completed no new command is run before the build again, which should treigger a reset & reparse before the next build - it's at this point that we see complaints from pseudo about the unset environment.

Debugging of this issue can be sped up with a full sstate cache and by turning do_rootfs into a noop for your test target, i.e:

diff --git a/meta/recipes-core/images/core-image-minimal.bb b/meta/recipes-core/images/core-image-minimal.bb
index 743e121..c339483 100644
--- a/meta/recipes-core/images/core-image-minimal.bb
+++ b/meta/recipes-core/images/core-image-minimal.bb
@@ -13,3 +13,7 @@ IMAGE_ROOTFS_SIZE = "8192"
 
 # remove not needed ipkg informations
 ROOTFS_POSTPROCESS_COMMAND += "remove_packaging_data_files ; "
+
+do_rootfs () {
+	exit 0
+}

Further a patch similar to the following will ensure that any errors are visible:
diff --git a/lib/bb/ui/hob.py b/lib/bb/ui/hob.py
index b0a98cb..999203f 100644
--- a/lib/bb/ui/hob.py
+++ b/lib/bb/ui/hob.py
@@ -134,6 +134,9 @@ class MainWindow (gtk.Window):
         self.set_busy_cursor(False)
         gtk.main_quit()
 
+    def command_failed_cb(self, handler, errormsg):
+        print(errormsg)
+
     def scroll_tv_cb(self, model, path, it, view):
         view.scroll_to_cell(path)
  
@@ -1003,6 +1007,7 @@ def main (server, eventHandler):
     configurator.connect("layers-changed", handler.reload_data)
     handler.connect("config-found", configurator.configFound)
     handler.connect("fatal-error", window.fatal_error_cb)
+    handler.connect("command-failed", window.command_failed_cb)
 
     try:
         # kick the while thing off
Comment 5 Joshua Lock - Disabled 2011-08-26 11:13:05 UTC
Further debugging hints gleaned through discussing the bug with various folk.

RP suggested the following may provide more insight:
* Dumping the environment (os.environ) before each build
* Blowing away the pseudo data dirs between builds
* Killing the pseudo process between builds (PID can be found in the pseudo.pid file)
Comment 6 Joshua Lock - Disabled 2011-08-30 11:53:46 UTC
Paul asked about how to get more debugging output from hob. attachment 201 [details] from bug 1285 includes some hints on how to print more debugging output.
Comment 7 Joshua Lock - Disabled 2011-08-30 11:55:04 UTC
Created attachment 217 [details]
Patch to aid debugging in hob

Based on attachment 201 [details] from bug 1285 an untested patch against BitBake master to add more debugging output.
Comment 8 Paul Eggleton 2011-08-31 08:06:35 UTC
So far I have been able to at least determine that pseudo isn't the cause of the crash - with pseudo hacked out of the bitbake script and the sanity.bbclass check commented out it still crashes. This would appear to suggest that we're somehow crashing the python interpreter, which is certainly unusual.
Comment 9 Paul Eggleton 2011-09-05 05:40:04 UTC
Found the issue - we weren't clearing out some internal lists between builds. I have posted a couple of patches to the bitbake-devel list for review.
Comment 10 Paul Eggleton 2011-09-05 14:44:21 UTC
Fixed in Poky commit a9325ecad889556b0043244509ed767eea363e1e.