Bug 13709 - Buildhistory race when building sdk and eSDK in parallel
Summary: Buildhistory race when building sdk and eSDK in parallel
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: 3.0
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 3.2 M2
Assignee: Chee Yang
QA Contact:
URL:
Whiteboard: autobuilder NEWCOMER AB-INT
Depends on:
Blocks:
 
Reported: 2019-12-16 14:40 UTC by Richard Purdie
Modified: 2020-07-25 07:14 UTC (History)
6 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Richard Purdie 2019-12-16 14:40:36 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/1356

Looks like a buildhistory race when sdk and eSDK are running in parallel:

NOTE: recipe core-image-sato-1.0-r0: task do_sdk_depends: Succeeded
NOTE: Running task 8949 of 8949 (/home/pokybuild/yocto-worker/qemux86-64/build/meta/recipes-sato/images/core-image-sato.bb:do_populate_sdk_ext)
NOTE: recipe core-image-sato-1.0-r0: task do_populate_sdk_ext: Started
ERROR: core-image-sato-1.0-r0 do_populate_sdk: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:do_populate_sdk(d)
     0003:
File: '/home/pokybuild/yocto-worker/qemux86-64/build/meta/classes/populate_sdk_base.bbclass', lineno: 169, function: do_populate_sdk
     0165:
     0166:    populate_sdk(d)
     0167:
     0168:fakeroot python do_populate_sdk() {
 *** 0169:    populate_sdk_common(d)
     0170:}
     0171:SSTATETASKS += "do_populate_sdk"
     0172:SSTATE_SKIP_CREATION_task-populate-sdk = '1'
     0173:do_populate_sdk[cleandirs] = "${SDKDEPLOYDIR}"
File: '/home/pokybuild/yocto-worker/qemux86-64/build/meta/classes/populate_sdk_base.bbclass', lineno: 166, function: populate_sdk_common
     0162:                    manifest_type=Manifest.MANIFEST_TYPE_SDK_HOST)
     0163:    create_manifest(d, manifest_dir=d.getVar('SDK_DIR'),
     0164:                    manifest_type=Manifest.MANIFEST_TYPE_SDK_TARGET)
     0165:
 *** 0166:    populate_sdk(d)
     0167:
     0168:fakeroot python do_populate_sdk() {
     0169:    populate_sdk_common(d)
     0170:}
File: '/home/pokybuild/yocto-worker/qemux86-64/build/meta/lib/oe/sdk.py', lineno: 413, function: populate_sdk
     0409:    env_bkp = os.environ.copy()
     0410:
     0411:    img_type = d.getVar('IMAGE_PKGTYPE')
     0412:    if img_type == "rpm":
 *** 0413:        RpmSdk(d, manifest_dir).populate()
     0414:    elif img_type == "ipk":
     0415:        OpkgSdk(d, manifest_dir).populate()
     0416:    elif img_type == "deb":
     0417:        DpkgSdk(d, manifest_dir).populate()
File: '/home/pokybuild/yocto-worker/qemux86-64/build/meta/lib/oe/sdk.py', lineno: 42, function: populate
     0038:    def populate(self):
     0039:        self.mkdirhier(self.sdk_output)
     0040:
     0041:        # call backend dependent implementation
 *** 0042:        self._populate()
     0043:
     0044:        # Don't ship any libGL in the SDK
     0045:        self.remove(os.path.join(self.sdk_output, self.sdk_native_path,
     0046:                         self.d.getVar('libdir_nativesdk').strip('/'),
File: '/home/pokybuild/yocto-worker/qemux86-64/build/meta/lib/oe/sdk.py', lineno: 182, function: _populate
     0178:        self.install_locales(self.host_pm)
     0179:
     0180:        self.host_pm.run_intercepts(populate_sdk='host')
     0181:
 *** 0182:        execute_pre_post_process(self.d, self.d.getVar("POPULATE_SDK_POST_HOST_COMMAND"))
     0183:
     0184:        if not bb.utils.contains("SDKIMAGE_FEATURES", "package-management", True, False, self.d):
     0185:            self.host_pm.remove_packaging_data()
     0186:
File: '/home/pokybuild/yocto-worker/qemux86-64/build/meta/lib/oe/utils.py', lineno: 260, function: execute_pre_post_process
     0256:    for cmd in cmds.strip().split(';'):
     0257:        cmd = cmd.strip()
     0258:        if cmd != '':
     0259:            bb.note("Executing %s ..." % cmd)
 *** 0260:            bb.build.exec_func(cmd, d)
     0261:
     0262:# For each item in items, call the function 'target' with item as the first
     0263:# argument, extraargs as the other arguments and handle any exceptions in the
     0264:# parent thread
File: '/home/pokybuild/yocto-worker/qemux86-64/build/bitbake/lib/bb/build.py', lineno: 250, function: exec_func
     0246:    with bb.utils.fileslocked(lockfiles):
     0247:        if ispython:
     0248:            exec_func_python(func, d, runfile, cwd=adir)
     0249:        else:
 *** 0250:            exec_func_shell(func, d, runfile, cwd=adir)
     0251:
     0252:    try:
     0253:        curcwd = os.getcwd()
     0254:    except:
File: '/home/pokybuild/yocto-worker/qemux86-64/build/bitbake/lib/bb/build.py', lineno: 451, function: exec_func_shell
     0447:    with open(fifopath, 'r+b', buffering=0) as fifo:
     0448:        try:
     0449:            bb.debug(2, "Executing shell function %s" % func)
     0450:            with open(os.devnull, 'r+') as stdin, logfile:
 *** 0451:                bb.process.run(cmd, shell=False, stdin=stdin, log=logfile, extrafiles=[(fifo,readfifo)])
     0452:        finally:
     0453:            os.unlink(fifopath)
     0454:
     0455:    bb.debug(2, "Shell function %s finished" % func)
File: '/home/pokybuild/yocto-worker/qemux86-64/build/bitbake/lib/bb/process.py', lineno: 182, function: run
     0178:        if not stderr is None:
     0179:            stderr = stderr.decode("utf-8")
     0180:
     0181:    if pipe.returncode != 0:
 *** 0182:        raise ExecutionError(cmd, pipe.returncode, stdout, stderr)
     0183:    return stdout, stderr
Exception: bb.process.ExecutionError: Execution of '/home/pokybuild/yocto-worker/qemux86-64/build/build/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r0/temp/run.buildhistory_get_sdk_installed_host.9350' failed with exit code 1:
rm: cannot remove ‘/home/pokybuild/yocto-worker/qemux86-64/build/build/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r0/bh_installed_pkgs.txt’: No such file or directory
WARNING: exit code 1 from a shell command.


ERROR: Logfile of failure stored in: /home/pokybuild/yocto-worker/qemux86-64/build/build/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r0/temp/log.do_populate_sdk.9350
NOTE: recipe core-image-sato-1.0-r0: task do_populate_sdk: Failed
ERROR: Task (/home/pokybuild/yocto-worker/qemux86-64/build/meta/recipes-sato/images/core-image-sato.bb:do_populate_sdk) failed with exit code '1'
NOTE: recipe core-image-sato-1.0-r0: task do_populate_sdk_ext: Succeeded
Comment 1 Richard Purdie 2020-03-23 10:29:07 UTC
Happened again: https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/1713
Comment 4 Chee Yang 2020-07-21 07:34:18 UTC
for eSDK, buildhistory are by default disabled in SDK_INHERIT_BLACKLIST. 
it seems like in these build eSDK do INHERIT buildhistory, is this expected?
Comment 5 Richard Purdie 2020-07-22 14:42:53 UTC
(In reply to comment #4)
> for eSDK, buildhistory are by default disabled in SDK_INHERIT_BLACKLIST. 
> it seems like in these build eSDK do INHERIT buildhistory, is this expected?

Those would be applied when the eSDK is running but not while eSDK itself is being built. The error happens when the eSDK is being built at the same time as an SDK.