Toaster is not able to start in dunfell branch The trace listing: The system will start. Traceback (most recent call last): File "/home/bogdan/WORK/yocto-x86_64/poky.dunfell/bitbake/bin/../lib/toaster/manage.py", line 14, in <module> execute_from_command_line(sys.argv) File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line utility.execute() File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 356, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 283, in run_from_argv self.execute(*args, **cmd_options) File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 327, in execute self.check() File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 356, in check all_issues = self._run_checks( File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 346, in _run_checks return checks.run_checks(**kwargs) File "/usr/local/lib/python3.8/site-packages/django/core/checks/registry.py", line 81, in run_checks new_errors = check(app_configs=app_configs) File "/usr/local/lib/python3.8/site-packages/django/core/checks/urls.py", line 16, in check_url_config return check_resolver(resolver) File "/usr/local/lib/python3.8/site-packages/django/core/checks/urls.py", line 26, in check_resolver return check_method() File "/usr/local/lib/python3.8/site-packages/django/urls/resolvers.py", line 256, in check for pattern in self.url_patterns: File "/usr/local/lib/python3.8/site-packages/django/utils/functional.py", line 35, in __get__ res = instance.__dict__[self.name] = self.func(instance) File "/usr/local/lib/python3.8/site-packages/django/urls/resolvers.py", line 407, in url_patterns patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module) File "/usr/local/lib/python3.8/site-packages/django/utils/functional.py", line 35, in __get__ res = instance.__dict__[self.name] = self.func(instance) File "/usr/local/lib/python3.8/site-packages/django/urls/resolvers.py", line 400, in urlconf_module return import_module(self.urlconf_name) File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1014, in _gcd_import File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 671, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 783, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/bogdan/WORK/yocto-x86_64/poky.dunfell/bitbake/lib/toaster/toastermain/urls.py", line 72, in <module> if p.pattern.regex.pattern == '^' + modulename + '/': AttributeError: 'RegexURLResolver' object has no attribute 'pattern'
Reproduced on Ubuntu 18.04 with latest updates and python 3.6.9. tony@tony-tower-bionic:~/development/poky$ pip3 list installed --local DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning. beautifulsoup4 (4.9.1) certifi (2019.6.16) chardet (3.0.4) Django (1.11.29) idna (2.8) pytz (2020.1) requests (2.22.0) soupsieve (2.0.1) urllib3 (1.25.3)
I think this issue may have been introduced with this commit: https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/bitbake/lib/toaster/toastermain/urls.py?h=dunfell&id=4dabdbe11ddc47a885c7f8cc6b3f26439d40d77c I tried changing p.pattern.regex.pattern -> p.regex.pattern and it seems to resolve the issue.
Hi, I believe that this problem is because I ported Toaster to Django-2.x, but missed committing the new toaster-requirements file. It appears that you have the older Django, which will fail with the updated toaster code. The file "toaster-requirements.txt should have: Django>2.2.0 beautifulsoup4>=4.4.0 pytz This does imply that you would need to update your Django to the latest version. This is because Django-1.x is now deprecated, and is indeed failing on all newer hosts (plus there are problems with the recent SQLITE and Python-3.7 that force an upgrade). If you have an older host, the pre-patched version of Toaster should work with YP-3.1, as you are observing. - David
Thanks for the quick response! I tried resetting my local clone and bumping the Django version. tony@linux-8ava:~/development/poky> git --no-pager diff diff --git a/bitbake/toaster-requirements.txt b/bitbake/toaster-requirements.txt index a682b085d8..4b8de5276f 100644 --- a/bitbake/toaster-requirements.txt +++ b/bitbake/toaster-requirements.txt @@ -1,3 +1,3 @@ -Django>1.8,<1.12 +Django>2.2.0 beautifulsoup4>=4.4.0 pytz Then I cleared out ~/.local, did the pip3 install again and checked the list. tony@linux-8ava:~/development/poky> rm -rf ~/.local tony@linux-8ava:~/development/poky> pip3 install --user -r bitbake/toaster-requirements.txt Collecting Django>2.2.0 Using cached Django-3.0.6-py3-none-any.whl (7.5 MB) Collecting beautifulsoup4>=4.4.0 Using cached beautifulsoup4-4.9.1-py3-none-any.whl (115 kB) Requirement already satisfied: pytz in /usr/lib/python3.6/site-packages (from -r bitbake/toaster-requirements.txt (line 3)) (2018.5) Collecting asgiref~=3.2 Using cached asgiref-3.2.7-py2.py3-none-any.whl (19 kB) Collecting sqlparse>=0.2.2 Using cached sqlparse-0.3.1-py2.py3-none-any.whl (40 kB) Collecting soupsieve>1.2 Using cached soupsieve-2.0.1-py3-none-any.whl (32 kB) Installing collected packages: asgiref, sqlparse, Django, soupsieve, beautifulsoup4 WARNING: The script sqlformat is installed in '/home/tony/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The script django-admin is installed in '/home/tony/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. Successfully installed Django-3.0.6 asgiref-3.2.7 beautifulsoup4-4.9.1 soupsieve-2.0.1 sqlparse-0.3.1 tony@linux-8ava:~/development/poky> pip3 list installed --local Package Version ------------------- --------- apparmor 2.12.3 appdirs 1.4.3 asgiref 3.2.7 asn1crypto 0.24.0 Babel 2.5.1 beautifulsoup4 4.9.1 Brlapi 0.6.6 certifi 2018.1.18 cffi 1.11.2 chardet 3.0.4 chrome-gnome-shell 0.0.0 cmdln 2.0.0 configobj 5.0.6 cryptography 2.1.4 cupshelpers 1.0 decorator 4.2.1 Django 3.0.6 gitdb 4.0.5 GitPython 3.1.2 idna 2.6 isc 2.0 Jinja2 2.10.1 LibAppArmor 2.12.3 louis 3.3.0 MarkupSafe 1.0 ndg-httpsclient 0.4.0 packaging 16.8 pexpect 4.8.0 pip 20.1.1 ply 3.10 ptyprocess 0.5.2 py 1.8.1 pyasn1 0.4.2 pycairo 1.15.1 pycparser 2.17 pycups 1.9.74 pycurl 7.43.0.2 pygobject 3.26.1 pyOpenSSL 17.5.0 pyparsing 2.2.0 pysmbc 1.0.15.8 PySocks 1.6.8 python-linux-procfs 0.6 pytz 2018.5 pyudev 0.21.0 requests 2.20.1 rpm 4.14.1 setuptools 40.5.0 six 1.11.0 slip 0.6.5 slip.dbus 0.6.5 smmap 3.0.4 soupsieve 2.0.1 sqlparse 0.3.1 urllib3 1.24 urlscan 0.9.2 urwid 2.0.1 Then try to start toaster. tony@linux-8ava:~/development/poky> source oe-init-build-env ### Shell environment set up for builds. ### You can now run 'bitbake <target>' Common targets are: core-image-minimal core-image-sato meta-toolchain meta-ide-support You can also run generated qemu images with a command like 'runqemu qemux86' Other commonly useful commands are: - 'devtool' and 'recipetool' handle common recipe tasks - 'bitbake-layers' handles common layer tasks - 'oe-pkgdata-util' handles common target package tasks tony@linux-8ava:~/development/poky/build> source toaster start The system will start. Traceback (most recent call last): File "/home/tony/development/poky/bitbake/bin/../lib/toaster/manage.py", line 14, in <module> execute_from_command_line(sys.argv) File "/home/tony/.local/lib/python3.6/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line utility.execute() File "/home/tony/.local/lib/python3.6/site-packages/django/core/management/__init__.py", line 377, in execute django.setup() File "/home/tony/.local/lib/python3.6/site-packages/django/__init__.py", line 24, in setup apps.populate(settings.INSTALLED_APPS) File "/home/tony/.local/lib/python3.6/site-packages/django/apps/registry.py", line 114, in populate app_config.import_models() File "/home/tony/.local/lib/python3.6/site-packages/django/apps/config.py", line 211, in import_models self.models_module = import_module(models_module_name) File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 994, in _gcd_import File "<frozen importlib._bootstrap>", line 971, in _find_and_load File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 665, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 678, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/tony/development/poky/bitbake/lib/toaster/bldcontrol/models.py", line 8, in <module> from orm.models import Project, Build, Layer_Version File "/home/tony/development/poky/bitbake/lib/toaster/orm/models.py", line 61, in <module> from django.utils import six ImportError: cannot import name 'six' This seems weird because six is in the installed list, but I'm not a python developer so I'm not sure how to interpret a missing import error. Please let me know if there is something further I can try.
OS/Python versions for previous comment tony@linux-8ava:~/development/poky/build> cat /etc/os-release NAME="openSUSE Leap" VERSION="15.1" ID="opensuse-leap" ID_LIKE="suse opensuse" VERSION_ID="15.1" PRETTY_NAME="openSUSE Leap 15.1" ANSI_COLOR="0;32" CPE_NAME="cpe:/o:opensuse:leap:15.1" BUG_REPORT_URL="https://bugs.opensuse.org" HOME_URL="https://www.opensuse.org/" tony@linux-8ava:~/development/poky/build> python3 --version Python 3.6.10 tony@linux-8ava:~/development/poky/build> pip3 --version pip 20.1.1 from /usr/lib/python3.6/site-packages/pip (python 3.6)
Here is an update: 1) The patch for the old version number for Django (to Django-2.x) mentioned above was submitted and is now part of YP-3.1.1 LTS. This means that Toaster is supported on the newer hosts which typically include Django-2, including for example Ubuntu-19.04. 2) The issue for this defect arises because the user's host has Django-3, which had not yet been validated with Toaster. That validation is in process. Specifically, Django-3 no longer supports the package "six". As it turns out, the "six" package is only included for Python-2/Python-3 support, and is only used in one location (the instance in "models.py" as per the backtrace above). Since Toaster (and YP in general) no longer uses Python-2, this package is not longer needed. (a) Here is the relevant "six" documentation: https://six.readthedocs.io/ method: six.reraise(exc_type, exc_value, exc_traceback=None) (b) Here is the code in Toaster: exc_info = sys.exc_info() ... six.reraise(*exc_info) (c) The fix is to simply use the native Python-3 call to re-raise the exception. (d) That code (plus other patches needed to support Django-3) are in progress. - David
David, Do/Should we state in documentation that we recommend using django in a virtualenv?
> David, Do/Should we state in documentation that we recommend using django in a virtualenv? The Django has been updating very quickly recently, much more so than in the previous 5 years, leaving a wide variety of versions of Django on the popular and recent hosts. This makes it difficult to have one definitive version of Toaster at one time even though the rest of Toaster has been quite stable. To address this I will indeed add documentation on how to set up virtualenv for Django for hosts that need it, and how to test for that condition. - David
I have posted a patch to the Toaster mailing list to add support for Django-3. - The module 'six' was for Python 2/3 compatibility. That package is both no longer needed (since Toaster is Py3 only) and is also deprecated on recent hosts, so explicitly replace 'reraise' with the Python-3 version. - The 'autocommits_when_autocommit_is_off' attribute is deprecated in Django-3. It was used for 'sqlite3 in Python < 3.6 which doesn't handle transactions and savepoints properly when autocommit is off' - Toaster requirements file updated to include Django 3.0*.
Tim Orling has pushed a patch set to upgrade support to Django 3. toaster-requirements.txt: Django 3.2 LTS toaster: Update docs links in templates toaster: orm/models -- drop django.utils.six toaster: set DEFAULT_AUTO_FIELD toaster: migratation for models.BigAutoField bitbake: buildinfohelper.py fix for Django 3.2 toaster: drop landing_not_managed template