13209 – wic hangs causing oe-selftest failures

Bug 13209 - wic hangs causing oe-selftest failures

Summary: wic hangs causing oe-selftest failures

Status:	RESOLVED FIXED

Alias:	None

Product:	OE-Core
Classification:	Build System, Metadata & Runtime
Component:	Scripts and Tools (show other bugs)
Version:	2.7
Hardware:	x86 Multiple

Importance:	Medium+ normal
Target Milestone:	2.7 M3
Assignee:	Richard Purdie
QA Contact:

URL:
Whiteboard:

Depends on:
Blocks:

Reported:	2019-02-28 09:29 UTC by Richard Purdie
Modified:	2019-03-07 15:30 UTC (History)
CC List:	2 users (show)

See Also:
OS type for building Yocto:	---
Type of Regression:	---
Verified:
Documentation change:	No (bug/feature does not impact docs)

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description Richard Purdie 2019-02-28 09:29:26 UTC

Having fixed other bitbake reconnection bugs, we're left with periodic wic hangs. Once it crashes, all further selftests within that build directory fail.

The best information about what is going on is from https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/285

where the server log says:

Exiting
Delaying shutdown due to active processes which appear to be holding bitbake.lock:
b'                     USER        PID ACCESS COMMAND\n/home/pokybuild/yocto-worker/oe-selftest/build/build-st-40683/bitbake.lock:\n                     pokybuild  42924 F.... python3\n'
Delaying shutdown due to active processes which appear to be holding bitbake.lock:

and this process 42924 doesn't go away. This would be enough to block the server exit which would stop any further server launching.

Logging into that worker later shows the process is still around, everything else has exited.

$ ps ax

 36419 ?        S<     0:09 [kworker/22:0H]
 41593 ?        S<     0:00 [kworker/8:0H]
 42924 ?        S      0:00 python3 /home/pokybuild/yocto-worker/oe-selftest/build/scripts/wic create wictestdisk --image-name=core-imag
 43430 ?        S      0:00 [kworker/8:2]
 43881 ?        S      0:00 [kworker/2:2]
 46806 ?        S      0:00 [kworker/26:0]
 46878 ?        S      0:12 [kworker/22:2]

$ cat /proc/42924/cmdline 
python3/home/pokybuild/yocto-worker/oe-selftest/build/scripts/wiccreatewictestdisk--image-name=core-image-minimal--debug-o/home/pokybuild/yocto-worker/oe-selftest/build/build-st-40683/wic-tmp

[pokybuild@centos7-ty-3 ~]$ ls /proc/42924/fd -la
total 0
dr-x------. 2 pokybuild pokybuild  0 Feb 28 09:13 .
dr-xr-xr-x. 9 pokybuild pokybuild  0 Feb 28 08:01 ..
lr-x------. 1 pokybuild pokybuild 64 Feb 28 09:23 0 -> /dev/null
lrwx------. 1 pokybuild pokybuild 64 Feb 28 09:23 1 -> /home/pokybuild/yocto-worker/oe-selftest/build/build-st-40683/bitbake-cookerdaemon.log (deleted)
lrwx------. 1 pokybuild pokybuild 64 Feb 28 09:23 2 -> /home/pokybuild/yocto-worker/oe-selftest/build/build-st-40683/bitbake-cookerdaemon.log (deleted)
lrwx------. 1 pokybuild pokybuild 64 Feb 28 09:23 3 -> /home/pokybuild/yocto-worker/oe-selftest/build/build-st-40683/bitbake.lock (deleted)
lrwx------. 1 pokybuild pokybuild 64 Feb 28 09:23 4 -> socket:[352130911]
lr-x------. 1 pokybuild pokybuild 64 Feb 28 09:23 5 -> anon_inode:inotify
l-wx------. 1 pokybuild pokybuild 64 Feb 28 09:23 6 -> pipe:[352130910]
lrwx------. 1 pokybuild pokybuild 64 Feb 28 09:23 7 -> /home/pokybuild/yocto-worker/oe-selftest/build/build-st-40683/bitbake-cookerdaemon.log (deleted)
lr-x------. 1 pokybuild pokybuild 64 Feb 28 09:23 8 -> anon_inode:inotify

[pokybuild@centos7-ty-3 ~]$ strace -p 42924
strace: Process 42924 attached
futex(0x7f5f57dac930, FUTEX_WAIT_PRIVATE, 2, NULL

Comment 1 Richard Purdie 2019-02-28 09:29:50 UTC

Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
__lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
135	2:	movl	%edx, %eax
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 expat-2.1.0-10.el7_3.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-37.el7_6.x86_64 libcom_err-1.42.9-13.el7.x86_64 libffi-3.0.13-18.el7.x86_64 libgcc-4.8.5-36.el7.x86_64 libselinux-2.5-14.1.el7.x86_64 openssl-libs-1.0.2k-16.el7.x86_64 pcre-8.32-17.el7.x86_64 sqlite-3.7.17-8.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) bt
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f5f5744dde6 in _L_lock_941 () from /lib64/libpthread.so.0
#2  0x00007f5f5744dcdf in __GI___pthread_mutex_lock (mutex=0x7f5f57dac930 <_rtld_local+2352>) at ../nptl/pthread_mutex_lock.c:113
#3  0x00007f5f57b95039 in _dl_add_to_namespace_list (new=0x21d1960, nsid=0) at dl-object.c:35
#4  0x00007f5f57b90824 in _dl_map_object_from_fd (
    name=name@entry=0x7f5f49535120 "/usr/lib64/python3.6/lib-dynload/mmap.cpython-36m-x86_64-linux-gnu.so", 
    origname=origname@entry=0x0, fd=-1, fbp=fbp@entry=0x7fff66f8c858, realname=<optimized out>, loader=loader@entry=0x0, 
    l_type=l_type@entry=2, mode=mode@entry=-1879048190, stack_endp=stack_endp@entry=0x7fff66f8c850, nsid=nsid@entry=0)
    at dl-load.c:1612
#5  0x00007f5f57b92576 in _dl_map_object (loader=0x0, loader@entry=0x7f5f57daa660, 
    name=name@entry=0x7f5f49535120 "/usr/lib64/python3.6/lib-dynload/mmap.cpython-36m-x86_64-linux-gnu.so", type=type@entry=2, 
    trace_mode=trace_mode@entry=0, mode=mode@entry=-1879048190, nsid=<optimized out>) at dl-load.c:2400
#6  0x00007f5f57b9e1d4 in dl_open_worker (a=a@entry=0x7fff66f8cdd8) at dl-open.c:231
#7  0x00007f5f57b99704 in _dl_catch_error (objname=objname@entry=0x7fff66f8cdc8, errstring=errstring@entry=0x7fff66f8cdd0, 
    mallocedp=mallocedp@entry=0x7fff66f8cdc0, operate=operate@entry=0x7f5f57b9e090 <dl_open_worker>, args=args@entry=0x7fff66f8cdd8)
    at dl-error.c:177
#8  0x00007f5f57b9dabb in _dl_open (file=0x7f5f49535120 "/usr/lib64/python3.6/lib-dynload/mmap.cpython-36m-x86_64-linux-gnu.so", 
    mode=-2147483646, caller_dlopen=<optimized out>, nsid=-2, argc=8, argv=0x7fff66f94b38, env=0x1fb9680) at dl-open.c:649
#9  0x00007f5f57240eeb in dlopen_doit (a=a@entry=0x7fff66f8cfe0) at dlopen.c:66
#10 0x00007f5f57b99704 in _dl_catch_error (objname=0x1c59120, errstring=0x1c59128, mallocedp=0x1c59118, 
    operate=0x7f5f57240e90 <dlopen_doit>, args=0x7fff66f8cfe0) at dl-error.c:177
#11 0x00007f5f572414ed in _dlerror_run (operate=operate@entry=0x7f5f57240e90 <dlopen_doit>, args=args@entry=0x7fff66f8cfe0)
    at dlerror.c:163
#12 0x00007f5f57240f81 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#13 0x00007f5f578421d6 in _PyImport_FindSharedFuncptr (prefix=prefix@entry=0x7f5f57890c56 "PyInit", 
    shortname=shortname@entry=0x7f5f495380f0 "mmap", 
    pathname=pathname@entry=0x7f5f49535120 "/usr/lib64/python3.6/lib-dynload/mmap.cpython-36m-x86_64-linux-gnu.so", fp=fp@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/dynload_shlib.c:95
#14 0x00007f5f57839d55 in _PyImport_LoadDynamicModuleWithSpec (spec=spec@entry=0x7f5f4943d748, fp=fp@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/importdl.c:129
#15 0x00007f5f578395f4 in _imp_create_dynamic_impl (module=<optimized out>, file=0x0, spec=0x7f5f4943d748)
    at /usr/src/debug/Python-3.6.6/Python/import.c:1994
#16 _imp_create_dynamic (module=<optimized out>, args=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/clinic/import.c.h:289
#17 0x00007f5f57748207 in PyCFunction_Call (func=func@entry=0x7f5f57d4cfc0, args=args@entry=0x7f5f4943d6d8, 
    kwds=kwds@entry=0x7f5f493c1bd0) at /usr/src/debug/Python-3.6.6/Objects/methodobject.c:139
#18 0x00007f5f577ae866 in do_call_core (kwdict=0x7f5f493c1bd0, callargs=0x7f5f4943d6d8, func=0x7f5f57d4cfc0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:5095
#19 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:3397
#20 0x00007f5f577b2fec in _PyEval_EvalCodeWithName (_co=0x7f5f57d86db0, globals=<optimized out>, locals=<optimized out>, 
    args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x7f5f57c75cd0, kwcount=<optimized out>, kwstep=1, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0, name=<optimized out>, qualname=0x7f5f57d30990)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4159
---Type <return> to continue, or q <return> to quit---
#21 0x00007f5f577b3e4a in fast_function (func=<optimized out>, stack=0x7f5f57c75cc0, nargs=2, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4971
#22 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f8d7b0, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#23 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#24 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#25 fast_function (func=<optimized out>, stack=0x7f5f4a29e550, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#26 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f8da00, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#27 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#28 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#29 fast_function (func=<optimized out>, stack=0x7f5f49395bd8, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#30 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f8dc50, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#31 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#32 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#33 fast_function (func=<optimized out>, stack=0x1ff5fe8, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#34 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f8dea0, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#35 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#36 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#37 fast_function (func=<optimized out>, stack=0x202f9f0, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#38 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f8e0f0, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#39 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#40 0x00007f5f577b51ba in _PyFunction_FastCall (globals=<optimized out>, nargs=2, args=<optimized out>, co=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#41 _PyFunction_FastCallDict (func=func@entry=0x7f5f57d50d90, args=args@entry=0x7fff66f8e300, nargs=<optimized out>, 
    kwargs=kwargs@entry=0x0) at /usr/src/debug/Python-3.6.6/Python/ceval.c:5014
---Type <return> to continue, or q <return> to quit---
#42 0x00007f5f57709e9e in _PyObject_FastCallDict (func=func@entry=0x7f5f57d50d90, args=args@entry=0x7fff66f8e300, 
    nargs=<optimized out>, kwargs=kwargs@entry=0x0) at /usr/src/debug/Python-3.6.6/Objects/abstract.c:2310
#43 0x00007f5f5770b1b4 in _PyObject_CallMethodIdObjArgs (callable=0x7f5f57d50d90, 
    name=name@entry=0x7f5f57b25fc0 <PyId__find_and_load.13318>) at /usr/src/debug/Python-3.6.6/Objects/abstract.c:2796
#44 0x00007f5f577ccd6f in PyImport_ImportModuleLevelObject (name=name@entry=0x7f5f4fe68110, globals=<optimized out>, 
    locals=<optimized out>, fromlist=fromlist@entry=0x7f5f57b0f190 <_Py_NoneStruct>, level=0)
    at /usr/src/debug/Python-3.6.6/Python/import.c:1578
#45 0x00007f5f577aadd5 in import_name (level=0x7f5f57b59600 <small_ints+160>, fromlist=0x7f5f57b0f190 <_Py_NoneStruct>, 
    name=0x7f5f4fe68110, f=0x7f5f493f7708) at /usr/src/debug/Python-3.6.6/Python/ceval.c:5224
#46 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:2892
#47 0x00007f5f577b43d2 in _PyEval_EvalCodeWithName (qualname=0x0, name=<optimized out>, closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, 
    kwstep=2, kwcount=<optimized out>, kwargs=<optimized out>, kwnames=0x0, argcount=0, args=0x0, locals=0x7f5f493bf510, 
    globals=0x7f5f493bf510, _co=0x7f5f495364b0) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4159
#48 PyEval_EvalCodeEx (_co=_co@entry=0x7f5f495364b0, globals=globals@entry=0x7f5f493bf510, locals=locals@entry=0x7f5f493bf510, 
    args=args@entry=0x0, argcount=argcount@entry=0, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, 
    defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4180
#49 0x00007f5f577b505b in PyEval_EvalCode (co=co@entry=0x7f5f495364b0, globals=globals@entry=0x7f5f493bf510, 
    locals=locals@entry=0x7f5f493bf510) at /usr/src/debug/Python-3.6.6/Python/ceval.c:731
#50 0x00007f5f577a737b in builtin_exec_impl (module=<optimized out>, locals=0x7f5f493bf510, globals=0x7f5f493bf510, 
    source=0x7f5f495364b0) at /usr/src/debug/Python-3.6.6/Python/bltinmodule.c:983
#51 builtin_exec (module=<optimized out>, args=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/clinic/bltinmodule.c.h:283
#52 0x00007f5f57748207 in PyCFunction_Call (func=func@entry=0x7f5f57d929d8, args=args@entry=0x7f5f49537208, 
    kwds=kwds@entry=0x7f5f493c1828) at /usr/src/debug/Python-3.6.6/Objects/methodobject.c:139
#53 0x00007f5f577ae866 in do_call_core (kwdict=0x7f5f493c1828, callargs=0x7f5f49537208, func=0x7f5f57d929d8)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:5095
#54 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:3397
#55 0x00007f5f577b2fec in _PyEval_EvalCodeWithName (_co=0x7f5f57d86db0, globals=<optimized out>, locals=<optimized out>, 
    args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x7f5f49428768, kwcount=<optimized out>, kwstep=1, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0, name=<optimized out>, qualname=0x7f5f57d30990)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4159
#56 0x00007f5f577b3e4a in fast_function (func=<optimized out>, stack=0x7f5f49428750, nargs=3, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4971
#57 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f8eb90, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#58 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#59 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#60 fast_function (func=<optimized out>, stack=0x7f5f493959d8, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#61 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f8ede0, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
---Type <return> to continue, or q <return> to quit---
#62 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#63 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#64 fast_function (func=<optimized out>, stack=0x21da6e8, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#65 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f8f030, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#66 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#67 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#68 fast_function (func=<optimized out>, stack=0x1ffff60, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#69 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f8f280, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#70 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#71 0x00007f5f577b51ba in _PyFunction_FastCall (globals=<optimized out>, nargs=2, args=<optimized out>, co=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#72 _PyFunction_FastCallDict (func=func@entry=0x7f5f57d50d90, args=args@entry=0x7fff66f8f490, nargs=<optimized out>, 
    kwargs=kwargs@entry=0x0) at /usr/src/debug/Python-3.6.6/Python/ceval.c:5014
#73 0x00007f5f57709e9e in _PyObject_FastCallDict (func=func@entry=0x7f5f57d50d90, args=args@entry=0x7fff66f8f490, 
    nargs=<optimized out>, kwargs=kwargs@entry=0x0) at /usr/src/debug/Python-3.6.6/Objects/abstract.c:2310
#74 0x00007f5f5770b1b4 in _PyObject_CallMethodIdObjArgs (callable=0x7f5f57d50d90, 
    name=name@entry=0x7f5f57b25fc0 <PyId__find_and_load.13318>) at /usr/src/debug/Python-3.6.6/Objects/abstract.c:2796
#75 0x00007f5f577ccd6f in PyImport_ImportModuleLevelObject (name=0x7f5f49523670, globals=<optimized out>, locals=<optimized out>, 
    fromlist=0x0, level=0) at /usr/src/debug/Python-3.6.6/Python/import.c:1578
#76 0x00007f5f577a646b in builtin___import__ (self=<optimized out>, args=<optimized out>, kwds=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/bltinmodule.c:238
#77 0x00007f5f57747f9f in _PyCFunction_FastCallDict (func_obj=func_obj@entry=0x7f5f57d92630, args=<optimized out>, nargs=1, kwargs=0x0)
    at /usr/src/debug/Python-3.6.6/Objects/methodobject.c:231
#78 0x00007f5f57748346 in _PyCFunction_FastCallKeywords (func=func@entry=0x7f5f57d92630, stack=<optimized out>, nargs=nargs@entry=1, 
    kwnames=kwnames@entry=0x0) at /usr/src/debug/Python-3.6.6/Objects/methodobject.c:294
#79 0x00007f5f577b3faf in call_function (pp_stack=pp_stack@entry=0x7fff66f8f810, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4830
#80 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#81 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#82 fast_function (func=<optimized out>, stack=0x7f5f49511a10, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
---Type <return> to continue, or q <return> to quit---
#83 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f8fa60, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#84 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#85 0x00007f5f577b43d2 in _PyEval_EvalCodeWithName (qualname=0x0, name=<optimized out>, closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, 
    kwstep=2, kwcount=<optimized out>, kwargs=<optimized out>, kwnames=0x0, argcount=0, args=0x0, locals=0x7f5f4951b620, 
    globals=0x7f5f493b8948, _co=0x7f5f4950fc90) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4159
#86 PyEval_EvalCodeEx (_co=_co@entry=0x7f5f4950fc90, globals=globals@entry=0x7f5f493b8948, locals=locals@entry=0x7f5f4951b620, 
    args=args@entry=0x0, argcount=argcount@entry=0, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, 
    defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4180
#87 0x00007f5f577b505b in PyEval_EvalCode (co=co@entry=0x7f5f4950fc90, globals=globals@entry=0x7f5f493b8948, 
    locals=locals@entry=0x7f5f4951b620) at /usr/src/debug/Python-3.6.6/Python/ceval.c:731
#88 0x00007f5f57832ce4 in builtin_eval_impl (module=<optimized out>, locals=0x7f5f4951b620, globals=0x7f5f493b8948, 
    source=0x7f5f4950fc90) at /usr/src/debug/Python-3.6.6/Python/bltinmodule.c:902
#89 builtin_eval (module=<optimized out>, args=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/clinic/bltinmodule.c.h:245
#90 0x00007f5f57747ee9 in _PyCFunction_FastCallDict (func_obj=func_obj@entry=0x7f5f57d92990, args=0x20a6950, nargs=3, 
    kwargs=<optimized out>) at /usr/src/debug/Python-3.6.6/Objects/methodobject.c:234
#91 0x00007f5f57748346 in _PyCFunction_FastCallKeywords (func=func@entry=0x7f5f57d92990, stack=<optimized out>, nargs=nargs@entry=3, 
    kwnames=kwnames@entry=0x0) at /usr/src/debug/Python-3.6.6/Objects/methodobject.c:294
#92 0x00007f5f577b3faf in call_function (pp_stack=pp_stack@entry=0x7fff66f8fe30, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4830
#93 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#94 0x00007f5f577b2fec in _PyEval_EvalCodeWithName (_co=0x7f5f4fe43d20, globals=<optimized out>, locals=<optimized out>, 
    args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x1ffd830, kwcount=<optimized out>, kwstep=1, 
    defs=0x7f5f4fbda220, defcount=1, kwdefs=0x0, closure=0x0, name=<optimized out>, qualname=0x7f5f4fe80330)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4159
#95 0x00007f5f577b3e4a in fast_function (func=<optimized out>, stack=0x1ffd818, nargs=3, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4971
#96 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f90130, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#97 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#98 0x00007f5f577b51ba in _PyFunction_FastCall (globals=<optimized out>, nargs=2, args=<optimized out>, co=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#99 _PyFunction_FastCallDict (func=func@entry=0x7f5f4a5760d0, args=args@entry=0x7fff66f90320, nargs=<optimized out>, 
    kwargs=kwargs@entry=0x0) at /usr/src/debug/Python-3.6.6/Python/ceval.c:5014
#100 0x00007f5f57709e9e in _PyObject_FastCallDict (func=func@entry=0x7f5f4a5760d0, args=args@entry=0x7fff66f90320, 
    nargs=nargs@entry=2, kwargs=kwargs@entry=0x0) at /usr/src/debug/Python-3.6.6/Objects/abstract.c:2310
#101 0x00007f5f57709fb1 in _PyObject_Call_Prepend (func=0x7f5f4a5760d0, obj=<optimized out>, args=0x7f5f493bbd30, kwargs=0x0)
    at /usr/src/debug/Python-3.6.6/Objects/abstract.c:2373
#102 0x00007f5f57709e20 in _PyObject_FastCallDict (func=func@entry=0x7f5f5012ffc8, args=args@entry=0x7fff66f90448, 
---Type <return> to continue, or q <return> to quit---
    nargs=nargs@entry=1, kwargs=kwargs@entry=0x0) at /usr/src/debug/Python-3.6.6/Objects/abstract.c:2331
#103 0x00007f5f577f5df3 in pattern_subx (self=self@entry=0x7f5f4a6ca100, ptemplate=<optimized out>, string=0x7f5f4938d8a0, count=0, 
    subn=subn@entry=0) at /usr/src/debug/Python-3.6.6/Modules/_sre.c:1160
#104 0x00007f5f577f6289 in _sre_SRE_Pattern_sub_impl (count=<optimized out>, string=<optimized out>, repl=<optimized out>, 
    self=0x7f5f4a6ca100) at /usr/src/debug/Python-3.6.6/Modules/_sre.c:1257
#105 _sre_SRE_Pattern_sub (self=self@entry=0x7f5f4a6ca100, args=<optimized out>, nargs=nargs@entry=2, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Modules/clinic/_sre.c.h:337
#106 0x00007f5f5774803a in _PyCFunction_FastCallDict (func_obj=func_obj@entry=0x7f5f49352fc0, args=0x1ff59a8, nargs=2, 
    kwargs=<optimized out>) at /usr/src/debug/Python-3.6.6/Objects/methodobject.c:250
#107 0x00007f5f57748346 in _PyCFunction_FastCallKeywords (func=func@entry=0x7f5f49352fc0, stack=<optimized out>, nargs=nargs@entry=2, 
    kwnames=kwnames@entry=0x0) at /usr/src/debug/Python-3.6.6/Objects/methodobject.c:294
#108 0x00007f5f577b3faf in call_function (pp_stack=pp_stack@entry=0x7fff66f90790, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4830
#109 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#110 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#111 fast_function (func=<optimized out>, stack=0x7f5f4938a730, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#112 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f909e0, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#113 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#114 0x00007f5f577b2fec in _PyEval_EvalCodeWithName (_co=0x7f5f4a79fc90, globals=<optimized out>, locals=<optimized out>, 
    args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x21a5320, kwcount=<optimized out>, kwstep=1, 
    defs=0x7f5f4a7a3b50, defcount=1, kwdefs=0x0, closure=0x0, name=<optimized out>, qualname=0x7f5f4a7a23d8)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4159
#115 0x00007f5f577b3e4a in fast_function (func=<optimized out>, stack=0x21a5308, nargs=3, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4971
#116 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f90ce0, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#117 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#118 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#119 fast_function (func=<optimized out>, stack=0x21de9c0, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#120 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f90f30, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#121 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#122 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
---Type <return> to continue, or q <return> to quit---
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#123 fast_function (func=<optimized out>, stack=0x2009bc0, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#124 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f91180, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#125 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#126 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#127 fast_function (func=<optimized out>, stack=0x200cce8, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#128 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f913d0, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#129 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#130 0x00007f5f577b2fec in _PyEval_EvalCodeWithName (_co=0x7f5f4a2b2270, globals=<optimized out>, locals=<optimized out>, 
    args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x200b1c8, kwcount=<optimized out>, kwstep=1, 
    defs=0x7f5f4a2ae798, defcount=1, kwdefs=0x0, closure=0x0, name=<optimized out>, qualname=0x7f5f50189500)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4159
#131 0x00007f5f577b3e4a in fast_function (func=<optimized out>, stack=0x200b1b0, nargs=3, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4971
#132 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f916d0, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#133 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#134 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#135 fast_function (func=<optimized out>, stack=0x21dfcf0, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#136 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f91920, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#137 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#138 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#139 fast_function (func=<optimized out>, stack=0x200bb20, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#140 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f91b70, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#141 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#142 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
---Type <return> to continue, or q <return> to quit---
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#143 fast_function (func=<optimized out>, stack=0x21a61c8, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#144 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f91dc0, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#145 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#146 0x00007f5f577b43d2 in _PyEval_EvalCodeWithName (qualname=0x0, name=<optimized out>, closure=0x0, kwdefs=0x0, defcount=0, 
    defs=0x0, kwstep=2, kwcount=<optimized out>, kwargs=<optimized out>, kwnames=0x0, argcount=2, args=0x21a6050, locals=0x1be7fa0, 
    globals=<optimized out>, _co=0x7f5f4fe91ae0) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4159
#147 PyEval_EvalCodeEx (_co=0x7f5f4fe91ae0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7f5f49354ba0, 
    argcount=<optimized out>, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, 
    kwdefs=kwdefs@entry=0x0, closure=0x0) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4180
#148 0x00007f5f57722093 in function_call (func=func@entry=0x7f5f4a2e9ea0, arg=arg@entry=0x7f5f49354b88, kw=kw@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Objects/funcobject.c:604
#149 0x00007f5f57709c23 in PyObject_Call (func=func@entry=0x7f5f4a2e9ea0, args=args@entry=0x7f5f49354b88, kwargs=kwargs@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Objects/abstract.c:2261
#150 0x00007f5f577aa05a in do_call_core (kwdict=0x0, callargs=0x7f5f49354b88, func=0x7f5f4a2e9ea0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:5099
#151 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:3397
#152 0x00007f5f577b31ca in _PyEval_EvalCodeWithName (_co=0x7f5f4fe84d20, globals=<optimized out>, locals=<optimized out>, 
    args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x21a2f38, kwcount=<optimized out>, kwstep=1, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x7f5f4a2cfda0, name=<optimized out>, qualname=0x7f5f4a585d30)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4159
#153 0x00007f5f577b3e4a in fast_function (func=<optimized out>, stack=0x21a2f28, nargs=2, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4971
#154 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f923e0, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#155 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#156 0x00007f5f577b2fec in _PyEval_EvalCodeWithName (_co=0x7f5f4a528810, globals=<optimized out>, locals=<optimized out>, 
    args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x21a2c28, kwcount=<optimized out>, kwstep=1, 
    defs=0x7f5f4a525648, defcount=1, kwdefs=0x0, closure=0x0, name=<optimized out>, qualname=0x7f5f4a5305d0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4159
#157 0x00007f5f577b3e4a in fast_function (func=<optimized out>, stack=0x21a2c10, nargs=3, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4971
#158 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f926e0, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#159 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#160 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
---Type <return> to continue, or q <return> to quit---
#161 fast_function (func=<optimized out>, stack=0x21f9938, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#162 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f92930, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#163 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#164 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#165 fast_function (func=<optimized out>, stack=0x21f3820, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#166 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f92b80, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#167 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#168 0x00007f5f577b2fec in _PyEval_EvalCodeWithName (_co=0x7f5f4a289300, globals=<optimized out>, locals=<optimized out>, 
    args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x0, kwcount=<optimized out>, kwstep=2, defs=0x7f5f4a2a0178, 
    defcount=1, kwdefs=0x0, closure=0x0, name=<optimized out>, qualname=0x7f5f4a2834f8)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4159
#169 0x00007f5f577b52ae in _PyFunction_FastCallDict (func=func@entry=0x7f5f49a89ea0, args=args@entry=0x7fff66f92e20, nargs=3, 
    kwargs=kwargs@entry=0x0) at /usr/src/debug/Python-3.6.6/Python/ceval.c:5063
#170 0x00007f5f57709e9e in _PyObject_FastCallDict (func=func@entry=0x7f5f49a89ea0, args=args@entry=0x7fff66f92e20, 
    nargs=nargs@entry=3, kwargs=kwargs@entry=0x0) at /usr/src/debug/Python-3.6.6/Objects/abstract.c:2310
#171 0x00007f5f57709fb1 in _PyObject_Call_Prepend (func=0x7f5f49a89ea0, obj=<optimized out>, args=0x7f5f49354548, kwargs=0x0)
    at /usr/src/debug/Python-3.6.6/Objects/abstract.c:2373
#172 0x00007f5f57709c23 in PyObject_Call (func=func@entry=0x7f5f57c72088, args=args@entry=0x7f5f49354548, kwargs=kwargs@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Objects/abstract.c:2261
#173 0x00007f5f5775dbd5 in slot_tp_init (self=self@entry=0x7f5f49356828, args=args@entry=0x7f5f49354548, kwds=kwds@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Objects/typeobject.c:6407
#174 0x00007f5f5775a5b2 in type_call (type=<optimized out>, type@entry=0x1fa2748, args=args@entry=0x7f5f49354548, kwds=kwds@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Objects/typeobject.c:915
#175 0x00007f5f57709e20 in _PyObject_FastCallDict (func=0x1fa2748, args=<optimized out>, nargs=<optimized out>, kwargs=0x0)
    at /usr/src/debug/Python-3.6.6/Objects/abstract.c:2331
#176 0x00007f5f577b415c in call_function (pp_stack=pp_stack@entry=0x7fff66f930b0, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4854
#177 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#178 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#179 fast_function (func=<optimized out>, stack=0x21f4e20, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#180 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f93300, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
---Type <return> to continue, or q <return> to quit---
#181 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#182 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#183 fast_function (func=<optimized out>, stack=0x21f4b68, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#184 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f93550, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#185 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#186 0x00007f5f577b51ba in _PyFunction_FastCall (globals=<optimized out>, nargs=5, args=<optimized out>, co=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#187 _PyFunction_FastCallDict (func=func@entry=0x7f5f49833bf8, args=args@entry=0x7fff66f93740, nargs=<optimized out>, 
    kwargs=kwargs@entry=0x0) at /usr/src/debug/Python-3.6.6/Python/ceval.c:5014
#188 0x00007f5f57709e9e in _PyObject_FastCallDict (func=func@entry=0x7f5f49833bf8, args=args@entry=0x7fff66f93740, 
    nargs=nargs@entry=5, kwargs=kwargs@entry=0x0) at /usr/src/debug/Python-3.6.6/Objects/abstract.c:2310
#189 0x00007f5f57709fb1 in _PyObject_Call_Prepend (func=0x7f5f49833bf8, obj=<optimized out>, args=0x7f5f49608e08, kwargs=0x0)
    at /usr/src/debug/Python-3.6.6/Objects/abstract.c:2373
#190 0x00007f5f57709c23 in PyObject_Call (func=func@entry=0x7f5f4af20088, args=args@entry=0x7f5f49608e08, kwargs=kwargs@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Objects/abstract.c:2261
#191 0x00007f5f5775dbd5 in slot_tp_init (self=self@entry=0x7f5f495cd048, args=args@entry=0x7f5f49608e08, kwds=kwds@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Objects/typeobject.c:6407
#192 0x00007f5f5775a5b2 in type_call (type=<optimized out>, type@entry=0x1fa5628, args=args@entry=0x7f5f49608e08, kwds=kwds@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Objects/typeobject.c:915
#193 0x00007f5f57709e20 in _PyObject_FastCallDict (func=0x1fa5628, args=<optimized out>, nargs=<optimized out>, kwargs=0x0)
    at /usr/src/debug/Python-3.6.6/Objects/abstract.c:2331
#194 0x00007f5f577b415c in call_function (pp_stack=pp_stack@entry=0x7fff66f939d0, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4854
#195 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#196 0x00007f5f577b2fec in _PyEval_EvalCodeWithName (_co=0x7f5f4a2dc9c0, globals=<optimized out>, locals=<optimized out>, 
    args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x2204f80, kwcount=<optimized out>, kwstep=1, 
    defs=0x7f5f4a273300, defcount=1, kwdefs=0x0, closure=0x0, name=<optimized out>, qualname=0x7f5f4a5f83b0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4159
#197 0x00007f5f577b3e4a in fast_function (func=<optimized out>, stack=0x2204f70, nargs=2, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4971
#198 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f93cd0, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#199 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#200 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
---Type <return> to continue, or q <return> to quit---bt
#201 fast_function (func=<optimized out>, stack=0x1fdafc8, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#202 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f93f20, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#203 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#204 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#205 fast_function (func=<optimized out>, stack=0x1fb7458, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#206 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f94170, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#207 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#208 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#209 fast_function (func=<optimized out>, stack=0x1ce5c38, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#210 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f943c0, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#211 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#212 0x00007f5f577b3d9a in _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4912
#213 fast_function (func=<optimized out>, stack=0x1c2e3d0, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4947
#214 0x00007f5f577b40d3 in call_function (pp_stack=pp_stack@entry=0x7fff66f94610, oparg=<optimized out>, kwnames=kwnames@entry=0x0)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:4851
#215 0x00007f5f577a8377 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /usr/src/debug/Python-3.6.6/Python/ceval.c:3328
#216 0x00007f5f577b43d2 in _PyEval_EvalCodeWithName (qualname=0x0, name=<optimized out>, closure=0x0, kwdefs=0x0, defcount=0, 
    defs=0x0, kwstep=2, kwcount=<optimized out>, kwargs=<optimized out>, kwnames=0x0, argcount=0, args=0x0, locals=0x7f5f57d121f8, 
    globals=0x7f5f57d121f8, _co=0x7f5f57c35540) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4159
#217 PyEval_EvalCodeEx (_co=_co@entry=0x7f5f57c35540, globals=globals@entry=0x7f5f57d121f8, locals=locals@entry=0x7f5f57d121f8, 
    args=args@entry=0x0, argcount=argcount@entry=0, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, 
    defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0) at /usr/src/debug/Python-3.6.6/Python/ceval.c:4180
#218 0x00007f5f577b505b in PyEval_EvalCode (co=co@entry=0x7f5f57c35540, globals=globals@entry=0x7f5f57d121f8, 
    locals=locals@entry=0x7f5f57d121f8) at /usr/src/debug/Python-3.6.6/Python/ceval.c:731
#219 0x00007f5f5783d63e in run_mod (mod=mod@entry=0x1caec28, filename=filename@entry=0x7f5f57bec730, 
    globals=globals@entry=0x7f5f57d121f8, locals=locals@entry=0x7f5f57d121f8, flags=flags@entry=0x7fff66f94910, 
    arena=arena@entry=0x7f5f57d2c270) at /usr/src/debug/Python-3.6.6/Python/pythonrun.c:1025
#220 0x00007f5f576e7fd2 in PyRun_FileExFlags (fp=fp@entry=0x1c7bf10, 
---Type <return> to continue, or q <return> to quit---
    filename_str=filename_str@entry=0x7f5f57c8db90 "/home/pokybuild/yocto-worker/oe-selftest/build/scripts/wic", 
    start=start@entry=257, globals=globals@entry=0x7f5f57d121f8, locals=locals@entry=0x7f5f57d121f8, closeit=closeit@entry=1, 
    flags=flags@entry=0x7fff66f94910) at /usr/src/debug/Python-3.6.6/Python/pythonrun.c:978
#221 0x00007f5f576e8392 in PyRun_SimpleFileExFlags (fp=fp@entry=0x1c7bf10, filename=<optimized out>, closeit=closeit@entry=1, 
    flags=flags@entry=0x7fff66f94910) at /usr/src/debug/Python-3.6.6/Python/pythonrun.c:420
#222 0x00007f5f576e896b in PyRun_AnyFileExFlags (fp=fp@entry=0x1c7bf10, filename=<optimized out>, closeit=closeit@entry=1, 
    flags=flags@entry=0x7fff66f94910) at /usr/src/debug/Python-3.6.6/Python/pythonrun.c:81
#223 0x00007f5f57843f0c in run_file (p_cf=0x7fff66f94910, 
    filename=0x1be72c0 L"/home/pokybuild/yocto-worker/oe-selftest/build/scripts/wic", fp=0x1c7bf10)
    at /usr/src/debug/Python-3.6.6/Modules/main.c:344
#224 Py_Main (argc=argc@entry=8, argv=argv@entry=0x1be6010) at /usr/src/debug/Python-3.6.6/Modules/main.c:814
#225 0x0000000000400a99 in main (argc=8, argv=<optimized out>) at /usr/src/debug/Python-3.6.6/Programs/python.c:102

Comment 2 Richard Purdie 2019-02-28 11:18:20 UTC

Similarly we had an earlier build fail on centos7-ty-1:

[root@centos7-ty-1 rpurdie]# cat /proc/145941/cmdline 
python3/home/pokybuild/yocto-worker/oe-selftest/build/scripts/wiccreatetemp.wks-ecore-image-minimal-o/home/pokybuild/yocto-worker/oe-selftest/build/build-st-17119/wic-tmp/

#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007ff5a9c4fde6 in _L_lock_941 () from /lib64/libpthread.so.0
#2  0x00007ff5a9c4fcdf in __GI___pthread_mutex_lock (mutex=0x7ff5aa5ae930 <_rtld_local+2352>) at ../nptl/pthread_mutex_lock.c:113
#3  0x00007ff5aa397039 in _dl_add_to_namespace_list (new=0x2997500, nsid=0) at dl-object.c:35
#4  0x00007ff5aa392824 in _dl_map_object_from_fd (
    name=name@entry=0x7ff59bd37120 "/usr/lib64/python3.6/lib-dynload/mmap.cpython-36m-x86_64-linux-gnu.so", 
    origname=origname@entry=0x0, fd=-1, fbp=fbp@entry=0x7ffd5a0b0648, realname=<optimized out>, loader=loader@entry=0x0, 
    l_type=l_type@entry=2, mode=mode@entry=-1879048190, stack_endp=stack_endp@entry=0x7ffd5a0b0640, nsid=nsid@entry=0)
    at dl-load.c:1612
#5  0x00007ff5aa394576 in _dl_map_object (loader=0x0, loader@entry=0x7ff5aa5ac660, 
    name=name@entry=0x7ff59bd37120 "/usr/lib64/python3.6/lib-dynload/mmap.cpython-36m-x86_64-linux-gnu.so", type=type@entry=2, 
    trace_mode=trace_mode@entry=0, mode=mode@entry=-1879048190, nsid=<optimized out>) at dl-load.c:2400
#6  0x00007ff5aa3a01d4 in dl_open_worker (a=a@entry=0x7ffd5a0b0bc8) at dl-open.c:231
#7  0x00007ff5aa39b704 in _dl_catch_error (objname=objname@entry=0x7ffd5a0b0bb8, errstring=errstring@entry=0x7ffd5a0b0bc0, 
    mallocedp=mallocedp@entry=0x7ffd5a0b0bb0, operate=operate@entry=0x7ff5aa3a0090 <dl_open_worker>, args=args@entry=0x7ffd5a0b0bc8)
    at dl-error.c:177
#8  0x00007ff5aa39fabb in _dl_open (file=0x7ff59bd37120 "/usr/lib64/python3.6/lib-dynload/mmap.cpython-36m-x86_64-linux-gnu.so", 
    mode=-2147483646, caller_dlopen=<optimized out>, nsid=-2, argc=8, argv=0x7ffd5a0b8928, env=0x2700b50) at dl-open.c:649
#9  0x00007ff5a9a42eeb in dlopen_doit (a=a@entry=0x7ffd5a0b0dd0) at dlopen.c:66
#10 0x00007ff5aa39b704 in _dl_catch_error (objname=0x241d530, errstring=0x241d538, mallocedp=0x241d528, 
    operate=0x7ff5a9a42e90 <dlopen_doit>, args=0x7ffd5a0b0dd0) at dl-error.c:177
#11 0x00007ff5a9a434ed in _dlerror_run (operate=operate@entry=0x7ff5a9a42e90 <dlopen_doit>, args=args@entry=0x7ffd5a0b0dd0)
    at dlerror.c:163


so the same hang, in trying to import the mmap module by the looks of it.

Comment 3 Richard Purdie 2019-02-28 11:31:17 UTC

Running the failed wic command under strace, the "import mmap" in lib/oe/package.py appears to be the one that triggers this, with imports in four different pids:

[pid 30593] stat("/usr/lib/python3.6/lib-dynload/mmap.cpython-36m-x86_64-linux-gnu.so", {st_mode=S_IFREG|0644, st_size=25392, ...}) = 0
[pid 30593] openat(AT_FDCWD, "/usr/lib/python3.6/lib-dynload/mmap.cpython-36m-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 9
[pid 30612] stat("/usr/lib/python3.6/lib-dynload/mmap.cpython-36m-x86_64-linux-gnu.so", {st_mode=S_IFREG|0644, st_size=25392, ...}) = 0
[pid 30612] openat(AT_FDCWD, "/usr/lib/python3.6/lib-dynload/mmap.cpython-36m-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 9
[pid 30643] stat("/usr/lib/python3.6/lib-dynload/mmap.cpython-36m-x86_64-linux-gnu.so",  <unfinished ...>
[pid 30643] openat(AT_FDCWD, "/usr/lib/python3.6/lib-dynload/mmap.cpython-36m-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 30649] stat("/usr/lib/python3.6/lib-dynload/mmap.cpython-36m-x86_64-linux-gnu.so", {st_mode=S_IFREG|0644, st_size=25392, ...}) = 0
[pid 30649] openat(AT_FDCWD, "/usr/lib/python3.6/lib-dynload/mmap.cpython-36m-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 9

Comment 4 Richard Purdie 2019-02-28 13:49:18 UTC

The import is from lib/oe/package.py. One is from bitbake server started by wic through bitbake_main(), another is from a worker started by that all. The others appear to be from bitbake -e calls by wic.

Mixing bitbake_main and "bitbake -e" calls within the same python process is probably a locking nightmare.

The bug we're hitting looks like a python interpreter/pthread problem with the python 3.6 install from EPLP on Centos.

The easiest workaround is probably to just call "bitbake XXX" from wic in all cases, there doesn't appear to be a good reason not to. We'll try and that see if this problem reoccurs.

Comment 5 Richard Purdie 2019-03-02 22:54:20 UTC

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=3d59df60a6d5831a1cf27858f08ae24ba0d7419d

Comment 6 Armin Kuster 2019-03-07 15:30:15 UTC

with backport?