10896 2017-01-07 23:28:23 +0000 bad magic number in 'oeqa': b'\x03\xf3\r\n' 2017-12-06 17:22:15 +0000 1 1 10 QA/Testing Runtime Testing general 2.2.1 x86 Multiple RESOLVED NOTABUG Medium normal 2.4 M2 1 akuster richard.purdie benjamin.esquivel brian.avery joshuagloe ross.burton --- --- Don't know oldest_to_newest 69612 0 akuster 2017-01-07 23:28:23 +0000 File: '/home/akuster/oss/maint/poky/meta/classes/testimage.bbclass', lineno: 126, function: testimage_main 0122: 0123:def testimage_main(d): 0124: import unittest 0125: import os *** 0126: import oeqa.runtime 0127: import time 0128: import signal 0129: from oeqa.oetest import ImageTestContext 0130: from oeqa.targetcontrol import get_target_controller Exception: ImportError: bad magic number in 'oeqa': b'\x03\xf3\r\n' I have only tried Morty. I initially saw this on a Ubuntu 16.0x system. My system was updated via Ubuntu's 'do release update' process that failed so I wrote this error off do to that. Over Xmas break I updated the system to Fedora Core 25 and am still seeing the issue. I finally had some time to poke at it and it looks like a pythong2 / python3 issue. as simple test python -c 'import runtime' python3 -c 'import runtime' Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: bad magic number in 'runtime': b'\x03\xf3\r\n' I noticed the runtime dir had a lot of *.pyc files, I deleted those and the problem went away. I now think this is an issue if someone builds krogoth then Morty within the same repo. Is this an FAQ, doc, wiki note or just an Old man thing? 69832 1 benjamin.esquivel 2017-01-16 17:28:09 +0000 I've seen this in the past and I did the same workaround that you did, erasing the .pyc files. I believe your analysis is pointing towards .pyc files that are generated by Python2 and then tried to be used by Python3. If this is indeed the case, it means you could reproduce it by running tests on YP releases pre/post Python2/3 change. I would not consider this a case a lot of users are going to find so a wiki note with the workaround might do the trick. A better solution would be to handle the exception with an error indicating what we know about it and how could it be fixed (workaround again here) 70754 2 joshuagloe 2017-02-16 16:43:22 +0000 Python 3 bytecode files are usually stored in a __pycache__ directory, so it's somewhat of a surprise that we're trying to use the Python 2.7 pyc files that live alongside the source. According to PEP-3147[1]: "Python will ignore all legacy pyc files when a source file exists next to it. In other words, if a foo.pyc file exists next to the foo.py file, the pyc file will be ignored in all cases " It sounds like we have a bug here. Why are we trying to load the 2.7 bytecode in Python3 when PEP-3147 states that shouldn't happen? Are we somehow telling Python to do that when mangling load paths to load from a layers meta/lib? 1. https://www.python.org/dev/peps/pep-3147/ 2. http://raulcd.com/how-python-caches-compiled-bytecode.html 70755 3 brian.avery 2017-02-16 16:46:54 +0000 On both a clean crops/poky:ubuntu-16.04 and crops/poky:fedora-25 I did the following: # apt-get/dnf install python-pip # apt-get/dnf install python3-pip # pip install runtime # pip3 install runtime greenlet $ python -c 'import runtime' $ python3 -c 'import runtime' ---- There were no errors, so it seems likely that something else is [ea]ffecting this. The above container have the minimal requirements for oe/poky builds, which is why I had to install pip and pip3 by hand. ---- 73734 4 ross.burton 2017-06-06 11:20:39 +0000 Made it happen! $ cd poky/meta/lib $ git checkout origin/krogoth $ PYTHONPATH=. python >>> import oeqa.runtime $ git checkout origin/master $ PYTHONPATH=. python3 >>> import oeqa Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: bad magic number in 'oeqa': b'\x03\xf3\r\n' 73735 5 ross.burton 2017-06-06 11:21:58 +0000 getcwd("/home/ross/Yocto/poky/meta/lib", 1026) = 31 stat("/home/ross/Yocto/poky/meta/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.cpython-34m-x86_64-linux-gnu.so", 0x7ffcdb0331c0) = -1 ENOENT (No such file or directory) stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.cpython-34m.so", 0x7ffcdb0331c0) = -1 ENOENT (No such file or directory) stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.abi3.so", 0x7ffcdb0331c0) = -1 ENOENT (No such file or directory) stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.so", 0x7ffcdb0331c0) = -1 ENOENT (No such file or directory) stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.py", 0x7ffcdb0331c0) = -1 ENOENT (No such file or directory) stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.pyc", {st_mode=S_IFREG|0644, st_size=103, ...}) = 0 getcwd("/home/ross/Yocto/poky/meta/lib", 1026) = 31 stat("/home/ross/Yocto/poky/meta/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.cpython-34m-x86_64-linux-gnu.so", 0x7ffcdb032c20) = -1 ENOENT (No such file or directory) stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.cpython-34m.so", 0x7ffcdb032c20) = -1 ENOENT (No such file or directory) stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.abi3.so", 0x7ffcdb032c20) = -1 ENOENT (No such file or directory) stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.so", 0x7ffcdb032c20) = -1 ENOENT (No such file or directory) stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.py", 0x7ffcdb032c20) = -1 ENOENT (No such file or directory) stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.pyc", {st_mode=S_IFREG|0644, st_size=103, ...}) = 0 open("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.pyc", O_RDONLY|O_CLOEXEC) = 3 73736 6 ross.burton 2017-06-06 11:23:43 +0000 So why is it looking for .pyc before __pycache__? 73740 7 ross.burton 2017-06-06 11:35:58 +0000 So py3 looks for __init__.py but that isn't present as it's optional in py3, but then finds the .pyc and fails. Stupid python. I think we just need to add back the missing empty __init__.py files. 73744 8 ross.burton 2017-06-06 11:56:19 +0000 Ok, so I have a patch which adds back about ten empty __init__.py files into meta/lib/oeqa. Or we tell people that this is what happens if you switch a branch between versions without doing a clean first. 73880 9 ross.burton 2017-06-08 13:26:24 +0000 Personally I think that this is a subtle gotcha but one that we can't neatly handle: module paths change over time and we'd need to maintain empty __init__.py in every directory. I prefer the solution of "run git clean" to remove these files if you switch between py2 and py3 branches in the same repository. Or, use git-worktree.