Bug 15690

oe-selftest-centos alma9-vk-1 master completed at 2024-12-08T02:22:17Z
https://valkyrie.yoctoproject.org/#/builders/76/builds/566/steps/14/logs/stdio

gpg can run as a server so that may be a factor.
There were also changes in the gpg recipe.

Wait to see if the happens again.

Bulk move of all unassigned 5.2 medium importance bugs to 5.3.

oe-selftest-fedora fedora39-vk-2 master-next completed at 2025-06-03T16:11:02Z
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/1624/steps/14/logs/stdio

Our team has been seeing a similar intermittent issue for both Kirkstone and Scarthgap:
"warning: Could not set GPG_TTY to stdin: Inappropriate ioctl for device"

If we keep reattempting the build, it eventually succeeds.

Sample with redactions via [...]:
ERROR: [...] do_package_write_rpm: Error executing a python function in exec_func_python() autogenerated:
    
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>    
     0001:
 *** 0002:sign_rpm(d)
     0003:
File: '[...]/openembedded-core/meta/classes/sign_rpm.bbclass', lineno: 59, function: sign_rpm
     0055:
     0056:    signer = get_signer(d, d.getVar('RPM_GPG_BACKEND'))
     0057:    rpms = glob.glob(d.getVar('RPM_PKGWRITEDIR') + '/*')
     0058:
 *** 0059:    signer.sign_rpms(rpms,
     0060:                     d.getVar('RPM_GPG_NAME'),
     0061:                     d.getVar('RPM_GPG_PASSPHRASE'),
     0062:                     d.getVar('RPM_FILE_CHECKSUM_DIGEST'),
     0063:                     int(d.getVar('RPM_GPG_SIGN_CHUNK')),
File: '[...]/openembedded-core/meta/lib/oe/gpg_sign.py', lineno: 59, function: sign_rpms
     0055:                cmd += "--define '_file_signing_key_password %s' " % fsk_password
     0056:
     0057:        # Sign in chunks
     0058:        for i in range(0, len(files), sign_chunk):
 *** 0059:            subprocess.check_output(shlex.split(cmd + ' '.join(files[i:i+sign_chunk])), stderr=subprocess.STDOUT)
     0060:
     0061:    def detach_sign(self, input_file, keyid, passphrase_file, passphrase=None, armor=True, output_suffix=None, use_sha256=False):
     0062:        """Create a detached signature of a file"""
     0063:
File: '/usr/lib/python3.10/subprocess.py', lineno: 421, function: check_output
     0417:        else:
     0418:            empty = b'' 
     0419:        kwargs['input'] = empty
     0420:
 *** 0421:    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
     0422:               **kwargs).stdout
     0423:
     0424:
     0425:class CompletedProcess(object):
File: '/usr/lib/python3.10/subprocess.py', lineno: 526, function: run 
     0522:            # We don't call process.wait() as .__exit__ does that for us. 
     0523:            raise
     0524:        retcode = process.poll()
     0525:        if check and retcode: 
 *** 0526:            raise CalledProcessError(retcode, process.args,
     0527:                                     output=stdout, stderr=stderr)
     0528:    return CompletedProcess(process.args, retcode, stdout, stderr)
     0529:
     0530:
Exception: subprocess.CalledProcessError: Command '['[...]/build/tmp-beaglebone-glibc/work/beaglebone-[...]-linux-gnueabi/packagegroup-[...]-security/1.0-r0/recipe-sysroot-native/usr/bin/rpmsign', '--addsign', '--define', '_gpg_name [...]', '--define', '_gpg_sign_cmd_extra_args --no-permission-warning --batch --passphrase=[...] --agent-program=[...]/build/tmp-beaglebone-glibc/work/beaglebone-[...]-linux-gnueabi/packagegroup-[...]-security/1.0-r0/recipe-sysroot-native/usr/bin/gpg-agent|--auto-expand-secmem --pinentry-mode=loopback', '--define', '_binary_filedigest_algorithm 8', '--define', '__gpg [...]/build/tmp-beaglebone-glibc/work/beaglebone-[...]-linux-gnueabi/packagegroup-[...]-security/1.0-r0/recipe-sysroot-native/usr/bin/gpg', '--define', '_gpg_path [...]', '[...]/build/tmp-beaglebone-glibc/work/beaglebone-[...]-linux-gnueabi/packagegroup-[...]-security/1.0-r0/deploy-rpms/beaglebone/packagegroup-[...]-security-dev-1.0-r0.beaglebone.rpm', '[...]/build/tmp-beaglebone-glibc/work/beaglebone-[...]-linux-gnueabi/packagegroup-[...]-security/1.0-r0/deploy-rpms/beaglebone/packagegroup-[...]-security-dbg-1.0-r0.beaglebone.rpm', '[...]/build/tmp-beaglebone-glibc/work/beaglebone-[...]-linux-gnueabi/packagegroup-[...]-security/1.0-r0/deploy-rpms/beaglebone/packagegroup-[...]-security-1.0-r0.beaglebone.rpm']' returned non-zero exit status 1.

Subprocess output:
warning: Could not set GPG_TTY to stdin: Inappropriate ioctl for device
warning: Could not set GPG_TTY to stdin: Inappropriate ioctl for device
warning: Could not set GPG_TTY to stdin: Inappropriate ioctl for device
gpg: signing failed: Connection reset by peer
gpg: signing failed: Connection reset by peer
error: gpg exec failed (2) 
[...]/build/tmp-beaglebone-glibc/work/beaglebone-[...]-linux-gnueabi/packagegroup-[...]-security/1.0-r0/deploy-rpms/beaglebone/packagegroup-[...]-security-dev-1.0-r0.beaglebone.rpm:
[...]/build/tmp-beaglebone-glibc/work/beaglebone-[...]-linux-gnueabi/packagegroup-[...]-security/1.0-r0/deploy-rpms/beaglebone/packagegroup-[...]-security-dbg-1.0-r0.beaglebone.rpm:
[...]/build/tmp-beaglebone-glibc/work/beaglebone-[...]-linux-gnueabi/packagegroup-[...]-security/1.0-r0/deploy-rpms/beaglebone/packagegroup-[...]-security-1.0-r0.beaglebone.rpm:

ERROR: Logfile of failure stored in: [...]/build/tmp-beaglebone-glibc/work/beaglebone-[...]-linux-gnueabi/packagegroup-21sw-security/1.0-r0/temp/log.do_package_write_rpm.1067898
ERROR: Task ([...]/meta-[...]/recipes-core/packagegroups/packagegroup-[...]-security.bb:do_package_write_rpm) failed with exit code '1'

Clayton,

How often does it happen out of say 10 or 100 builds for you?
What is your build host distro and HW (cpus, mem, storage)?

Have you tried stracing all or part of the failure and
comparing it to when the build works? :-)

Hi, Randy

We've seen this on multiple Ubuntu systems (typically 22.04) including:
Ryzen 7000 Series, 96 GB RAM, PCIe 4.0 NVMe storage
Intel 14th gen mobile, 96 GB RAM, PCIe 4.0 NVMe storage

I haven't seen this for a while, though signed builds are ad-hoc (partially due to this issue). From my memory, it could happen 6 build attempts in a row (or work just fine).

Debugging attempts have mostly been focused on GPG (no luck); we haven't tried strace (but will keep it in mind for the future).

I'm wondering if this problem surfaces with rpm-sequoia: https://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/rpm-sequoia?h=walnascar

I have been investigating this issue, but I'm stumped.

I noticed that the command generated by the `LocalSigner.sign_rpms()' method in `openembedded-core: gpg_sign.py` was erroneous:


--agent-program=[...]/build/tmp-beaglebone-glibc/work/armv7at2hf-neon-21software-linux-gnueabi/p11-kit/0.25.3/recipe-sysroot-native/usr/bin/gpg-agent|--auto-expand-secmem

Upon replacing the | character with a space in the script, the same error was still intermittently observed.

I took an `strace` of our build process, but the only relevant logging I could find was in `bitbake-cookerdaemon.log`:


0:openat(AT_FDCWD,
         "[...]/build/bitbake-cookerdaemon.log",
         O_RDWR|O_CREAT|O_APPEND|O_CLOEXEC, 0666) = 6

1580578:11:fstat(6, {st_mode=S_IFREG|0664, st_size=411537764, ...}) = 0

1580578:12:lseek(6, 0, SEEK_END) = 411537764

1580578:13:ioctl(6, TCGETS, 0x7fff62fbd650) = -1 ENOTTY (Inappropriate ioctl for device)

Upon investigating `bitbake-cookerdaemon.log`:

11270 12:59:30.506658 Exception in server main event loop running command [] (Traceback (most recent call last):
  File "[...]/bitbake/lib/bb/server/process.py", line 298, in main
    self.command_channel_reply.send(reply)
  File "[...]/bitbake/lib/bb/server/process.py", line 890, in send
    self._send(obj)
  File "[...]/bitbake/lib/bb/server/process.py", line 869, in _send
    self.writer.send_bytes(obj)
  File "/usr/lib/python3.12/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/lib/python3.12/multiprocessing/connection.py", line 427, in _send_bytes
    self._send(header + buf)
  File "/usr/lib/python3.12/multiprocessing/connection.py", line 384, in _send
    n = write(self._handle, buf)
        ^^^^^^^^^^^^^^^^^^^^^^^^
BrokenPipeError: [Errno 32] Broken pipe

Tracing code execution, the only method that appears to be capable of throwing a BrokenPipeError is `ConnectionWriter._send` in `bitbake/lib/bb/server/process.py`. Thinking perhaps the issue is being caused by concurrent signing tasks during the build process, I tried serializing the `do_package_write_rpm` tasks, but the same issue intermittently persists.

After backporting rpm-sequoia to a local Scarthgap build, there are still intermittent failures.

The stack trace is the same as with rpm.bb's "internal-openpgp".

However, there is a new line at the start of the subprocess output:

couldn't allocate absolute path for 'null'.
warning: Could not set GPG_TTY to stdin: Inappropriate ioctl for device
gpg: problem with fast path key listing: End of file - ignored
gpg: keydb_search failed: Broken pipe
gpg: skipped "[REDACTED_KEY]": Broken pipe
gpg: signing failed: Broken pipe
error: gpg exec failed (2)

NB: the specific subprocess output varies and we've seen this one before (with "internal-openpgp").

https://rpm.org/releases/6.0.0:
"rpmsign can use either GnuPG or Sequoia-sq for signing"

"[PATCH v8 7/9] rpm: 4.20.1 -> 6.0.1":
https://lists.openembedded.org/g/openembedded-core/message/232961

Hi Clayton,

We discussed in YP Bug review call, we are no longer seeing the AB-INT issue in the Yocto autobuilder.

There appear to be two different failure modes:
- On the autobuilder we saw "gpg: signing failed: Corrupted protection"
- Clayton saw "signing failed: Connection reset by peer" and "signing failed: Broken pipe"

I don't think the GPG_TTY warning is likely to be the cause of the failure.

We're going to close this issue as the original failure is resolved.

Clayton, please re-open a new bug if you're still seeing failures. We've made a lot of fixes in pseudo which hopefully will resolve your issue if you update to the latest scarthgap commit.