Failed builds:

https://autobuilder.yocto.io/builders/nightly-oe-selftest/builds/491
https://autobuilder.yocto.io/builders/nightly-oe-selftest/builds/490

issue reproduced locally on the same distro (ubuntu 17.04). Also happening on debian 9 and pyro [1]

[1] https://autobuilder.yocto.io/builders/nightly-oe-selftest/builds/496/steps/Running%20oe-selftest/logs/stdio

This is work in-progress with no definite root cause found. This is affecting mainly ubuntu 17.04 but due to the amount of users using the latter, it may become a blocker for release. Will talk to key maintainers so decide what to do with this (move it to 2.5M1?)

Workaround:

if task do_package_write_rpm mutually excludes between recipes, the issue is not present, meaning that the following workaround would make the test pass:

diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 1deaf832da..9a65389984 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -713,6 +713,9 @@ do_package_write_rpm[sstate-outputdirs] = "${DEPLOY_DIR_RPM}"
 # but we need to stop the rootfs/solver from running while we do...
 do_package_write_rpm[sstate-lockfile-shared] += "${DEPLOY_DIR_RPM}/rpm.lock"
 
+# due the signing serially
+do_package_write_rpm[lockfiles] += "${TMPDIR}/gpg.lock"
+
 python do_package_write_rpm_setscene () {
     sstate_setscene(d)
 }

although the latter is not a patch for upstream (we do not have to create rpms serially), it confirms that it is an issue related to the signing tool itself. This problem is present en 2.1.X and 2.2.X. 


Errors:

The errors observed coming from the rpmsign are two:

 gpg: signing failed: Cannot allocate memory
 gpg: signing failed: End of file

where in both cases, causes the rpmsign to hang, thus related bitbake workers. For the first error (memory allocation) I monitor the mem usage with top cmd, and there are some big steps in memory consumption, for example, with a sample rate of 1s, these are two seconds in a row:

KiB Mem : 32943196 total,  1021076 free,  1453984 used, 30468136 buff/cache
KiB Mem : 32943196 total,   246016 free,  2424848 used, 30272332 buff/cache

so 1g of memory consumed in the process of signing rpm packages. I am not sure which process or processes are consuming such amount of memory but something is  not quite right in the memory allocation process.

Reproducibility:

this is happening 100% or gpg 2.1.15 version, which is the default version for ubuntu 15.04. I tried with the latest version from gnupg and the memory issue remains (the  end of file was not observed). As a way to report the problem to corresponding upstream community, I tried the following code but unfortunately pinentry was asking for the passphrase so at the end I was not able to reproduce what is observed in oe-core. I pasted the code so it can be used in further debugging

#!/usr/bin/env python3                                                                                                                                                                                                                         

import multiprocessing
import subprocess
import argparse
import glob
import time

gpg_path = ''

def sign(chunk):
    action = "--addsign"
    cmd = "rpm %s \                                                                                                                                                                                                                            
    --define '_gpg_name testuser'  \                                                                                                                                                                                                           
    --define '_gpg_sign_cmd_extra_args --no-permission-warning --batch --passphrase=test123 --pinentry-mode=loopback' \                                                                                                                        
    --define '_binary_filedigest_algorithm 8' \                                                                                                                                                                                                
    --define '__gpg /usr/bin/gpg' \                                                                                                                                                                                                            
    --define '_gpg_path %s' %s" % (action, gpg_path, chunk)

    status, stdout = subprocess.getstatusoutput(cmd)
    if status:
        print ("Failed to sign RPM packages: %s" % stdout)
    else:
        print('cmd %s\nOutput %s' % (cmd, stdout))

def sign_error(a):
    print('error %s' % a)

def sign_pass(a):
    print('pass %s' % a)

def sign_repo(repo, gpg_path, pub_key_path, secret_key_path, pool_size, chunk_size):
    cmd = 'gpg --debug-all --batch --passphrase=test123 --homedir %s --import %s %s' % (gpg_path, pub_key_path, secret_key_path)
    status, stdout = subprocess.getstatusoutput(cmd)

    if status:
        print('Error while launching gpg %s:\n%s' % (cmd,stdout))
        return

    time.sleep(5)

    rpms = glob.glob(repo + '*.rpm')
    chunks = [' '.join(rpms[i:i+chunk_size]) for i in range(0, len(rpms), chunk_size)]

    print('%s rpms will be signed with a pool of %s length' % (len(rpms), pool_size))
    with multiprocessing.Pool(pool_size) as pool:
        pool.map(sign, chunks)

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description='Stress rpm signing')
    parser.add_argument('repo', help='Repository path')
    parser.add_argument('gpg_path', help='gpg directory path')
    parser.add_argument('pub_key_path', help='public key filename')
    parser.add_argument('secret_key_path', help='secret key filename')
    parser.add_argument('--pool-size', type=int, default=20)
    parser.add_argument('--chunk-size', type=int, default=20)

    args =  parser.parse_args()

    gpg_path = args.gpg_path

    sign_repo(args.repo, args.gpg_path, args.pub_key_path, args.secret_key_path, args.pool_size, args.chunk_size)

> 
> Reproducibility:
> 
> this is happening 100% or gpg 2.1.15 version, which is the default version
> for ubuntu 15.04. I tried with the latest version from gnupg and the memory

I mean ubuntu 17.04, meaning the latest version of ubuntu distro at this time

Workaround was accepted but issue is still present and must be fix during M1, thus moving to the latter milestone.

*** Bug 11258 has been marked as a duplicate of this bug. ***

*** Bug 11368 has been marked as a duplicate of this bug. ***

*** Bug 12073 has been marked as a duplicate of this bug. ***

gnupg 2.2.4 has a fix for this:

https://dev.gnupg.org/T3530

So we need to 
a) upgrade the gnupg recipe to at least that (and required libgcrypt 1.8.2)
b) enable and default to gnupg-native, instead of host gnupg
c) patch the gnupg source code (start_new_gpg_agent() in particular) to pass --auto-expand-secmem to gpg-agent instances.

oh, and 

d) make sure the native gpg launches native gpg-agent (and not the host one)

Fix merged to master:
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=0f49d9182f9a6204399c0946b9e9ae64ad3c0838