Bug 10896

I've seen this in the past and I did the same workaround that you did, erasing the .pyc files. I believe your analysis is pointing towards .pyc files that are generated by Python2 and then tried to be used by Python3. If this is indeed the case, it means you could reproduce it by running tests on YP releases pre/post Python2/3 change.

I would not consider this a case a lot of users are going to find so a wiki note with the workaround might do the trick. A better solution would be to handle the exception with an error indicating what we know about it and how could it be fixed (workaround again here)

Python 3 bytecode files are usually stored in a __pycache__ directory, so it's somewhat of a surprise that we're trying to use the Python 2.7 pyc files that live alongside the source.

According to PEP-3147[1]:

"Python will ignore all legacy pyc files when a source file exists next to it. In other words, if a foo.pyc file exists next to the foo.py file, the pyc file will be ignored in all cases "

It sounds like we have a bug here. Why are we trying to load the 2.7 bytecode in Python3 when PEP-3147 states that shouldn't happen? Are we somehow telling Python to do that when mangling load paths to load from a layers meta/lib?

1. https://www.python.org/dev/peps/pep-3147/
2. http://raulcd.com/how-python-caches-compiled-bytecode.html

On both a clean crops/poky:ubuntu-16.04 and crops/poky:fedora-25 I did the following:

# apt-get/dnf install python-pip
# apt-get/dnf install python3-pip
# pip install runtime
# pip3 install runtime greenlet

$ python -c 'import runtime'
$ python3 -c 'import runtime'

----
There were no errors, so it seems likely that something else is [ea]ffecting this. The above container have the minimal requirements for oe/poky builds, which is why I had to install  pip and pip3 by hand.

----

Made it happen!

$ cd poky/meta/lib
$ git checkout origin/krogoth
$ PYTHONPATH=. python
>>> import oeqa.runtime
$ git checkout origin/master
$ PYTHONPATH=. python3
>>> import oeqa
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: bad magic number in 'oeqa': b'\x03\xf3\r\n'

getcwd("/home/ross/Yocto/poky/meta/lib", 1026) = 31
stat("/home/ross/Yocto/poky/meta/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.cpython-34m-x86_64-linux-gnu.so", 0x7ffcdb0331c0) = -1 ENOENT (No such file or directory)
stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.cpython-34m.so", 0x7ffcdb0331c0) = -1 ENOENT (No such file or directory)
stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.abi3.so", 0x7ffcdb0331c0) = -1 ENOENT (No such file or directory)
stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.so", 0x7ffcdb0331c0) = -1 ENOENT (No such file or directory)
stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.py", 0x7ffcdb0331c0) = -1 ENOENT (No such file or directory)
stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.pyc", {st_mode=S_IFREG|0644, st_size=103, ...}) = 0
getcwd("/home/ross/Yocto/poky/meta/lib", 1026) = 31
stat("/home/ross/Yocto/poky/meta/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.cpython-34m-x86_64-linux-gnu.so", 0x7ffcdb032c20) = -1 ENOENT (No such file or directory)
stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.cpython-34m.so", 0x7ffcdb032c20) = -1 ENOENT (No such file or directory)
stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.abi3.so", 0x7ffcdb032c20) = -1 ENOENT (No such file or directory)
stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.so", 0x7ffcdb032c20) = -1 ENOENT (No such file or directory)
stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.py", 0x7ffcdb032c20) = -1 ENOENT (No such file or directory)
stat("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.pyc", {st_mode=S_IFREG|0644, st_size=103, ...}) = 0
open("/home/ross/Yocto/poky/meta/lib/oeqa/__init__.pyc", O_RDONLY|O_CLOEXEC) = 3

So why is it looking for .pyc before __pycache__?

So py3 looks for __init__.py but that isn't present as it's optional in py3, but then finds the .pyc and fails.

Stupid python.

I think we just need to add back the missing empty __init__.py files.

Ok, so I have a patch which adds back about ten empty __init__.py files into meta/lib/oeqa.

Or we tell people that this is what happens if you switch a branch between versions without doing a clean first.

Personally I think that this is a subtle gotcha but one that we can't neatly handle: module paths change over time and we'd need to maintain empty __init__.py in every directory.

I prefer the solution of "run git clean" to remove these files if you switch between py2 and py3 branches in the same repository.  Or, use git-worktree.