Build Configuration: BB_VERSION = "1.37.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "fedora-27" TARGET_SYS = "aarch64-poky-linux" MACHINE = "qemuarm64" DISTRO = "poky" DISTRO_VERSION = "2.4+snapshot-20180423" TUNE_FEATURES = "aarch64" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "master:f49ee61422c867516db252054e993df29f775136" ... ... ... NOTE: Running task 5790 of 5843 (virtual:nativesdk:/home/ab/yocto-worker/nightly-arm64/build/meta/recipes-core/libxml/libxml2_2.9.7.bb:do_package_write_rpm) NOTE: recipe nativesdk-opkg-1_0.3.6-r0: task do_compile: Started NOTE: recipe nativesdk-shared-mime-info-1.9-r0: task do_package: Started NOTE: recipe nativesdk-libxml2-2.9.7-r0: task do_package_write_rpm: Started NOTE: recipe nativesdk-libcomps-0.1.8+gitAUTOINC+01a4759894-r0: task do_package: Started ERROR: Execution of event handler 'runqueue_stats' failed Traceback (most recent call last): File "/home/ab/yocto-worker/nightly-arm64/build/meta/classes/buildstats.bbclass", line 212, in runqueue_stats(e=<bb.event.HeartbeatEvent object at 0x7f367e536438>): done = isinstance(e, bb.event.BuildCompleted) > system_stats.sample(e, force=done) if done: File "/home/ab/yocto-worker/nightly-arm64/build/meta/lib/buildstats.py", line 148, in SystemStats.sample(event=<bb.event.HeartbeatEvent object at 0x7f367e536438>, force=False) OSError: [Errno 30] Read-only file system command timed out: 16200 seconds without output running ['/home/ab/yocto-worker/nightly-arm64/yocto-autobuilder-helper/scripts/run-config', 'nightly-arm64', '/home/ab/yocto-worker/nightly-arm64/build/build', 'master', 'git://git.yoctoproject.org/poky', '', 'None', 'None', 'http://centaurus02.png.intel.com:8010/#builders/19/builds/12'], attempting to kill process killed by signal 9 program finished with exit code -1 After performing the QA test on nightly-arm64, the Fedora27 KVM distribution file-system becomes read-only and once the KVM has been re instantiated, it went into emergency mode and unable to boot Kernel.
So unless I'm missing something this is an infrastructure issue - right? The file it's writing to is under BUILDSTATS_BASE which is under TMPDIR and thus shouldn't ever be read-only for the build system.
See Paul's question
Hi Paul, We are trying to resolve this as the entire KVM Fedora27 file-system became read-only and system ran into emergency mode after we ran oe-selftest. I am trying to get Alim to recommission the entire Distro and rerun again.
What could we be doing in the tests though that would cause the system to go into read-only mode?
There are 2 occurrence which results in read-only file system during bitbake on fedora27. nightly-arm64 (first occurance) ------------- Link: http://centaurus02.png.intel.com:8010/#/builders/19/builds/12 NOTE: recipe nativesdk-libcomps-0.1.8+gitAUTOINC+01a4759894-r0: task do_package: Started ERROR: Execution of event handler 'runqueue_stats' failed Traceback (most recent call last): File "/home/ab/yocto-worker/nightly-arm64/build/meta/classes/buildstats.bbclass", line 212, in runqueue_stats(e=<bb.event.HeartbeatEvent object at 0x7f367e536438>): done = isinstance(e, bb.event.BuildCompleted) > system_stats.sample(e, force=done) if done: File "/home/ab/yocto-worker/nightly-arm64/build/meta/lib/buildstats.py", line 148, in SystemStats.sample(event=<bb.event.HeartbeatEvent object at 0x7f367e536438>, force=False) OSError: [Errno 30] Read-only file system nightly-x86_64 (2nd occurance) -------------- Link: http://centaurus02.png.intel.com:8010/#/builders/24/builds/4 NOTE: recipe core-image-sato-1.0-r0: task do_image_wic: Started ERROR: Execution of event handler 'runqueue_stats' failed Traceback (most recent call last): File "/home/ab/yocto-worker/nightly-x86-64/build/meta/classes/buildstats.bbclass", line 212, in runqueue_stats(e=<bb.event.HeartbeatEvent object at 0x7fc4211c61d0>): done = isinstance(e, bb.event.BuildCompleted) > system_stats.sample(e, force=done) if done: File "/home/ab/yocto-worker/nightly-x86-64/build/meta/lib/buildstats.py", line 148, in SystemStats.sample(event=<bb.event.HeartbeatEvent object at 0x7fc4211c61d0>, force=False): data + > b'\n') self.last_proc = now OSError: [Errno 30] Read-only file system We have completed recommission the entire KVM Distro Fedora27 assuming this is an Infrastructure issue, it shouldn't reoccur again as we seen such abnormalities in other distribution QA cycle but was never recorded anywhere. This ticket is to track this kind of abnormalities which may or may not be caused by bitbake or whatever. We ran a sanity check with running Oe-selftest excluding Bitbake and it PASSED successfully. Link: http://centaurus02.png.intel.com:8010/#/builders/18/builds/25
Created attachment 4292 [details] nightly-arm-fedora27 read-only file system
Created attachment 4293 [details] nightly-x86_64-fedora27 read-only file system
I fixed this issue by recomission new KVM for Fedora 27. I also tested on 2.5 M4 rc1 and didn't see any issue regarding to read-only file system.