I checkout oe-core run . ./oe-init-build-env ../build/ ../bitbake/ export TOASTER_CONF=../openembedded-core/.templateconf source toaster this gets me a clean start with very very recent master. I connect a browser and click new project, I enter "gnuradio" toaster replies "Fields missing: projectversion" This is an improvement, still not good enough :)
Found on master:b3afe7d9bd25a943e947de3ec064ea245173c5a8
The steps I took to attempt to reproduce this: 1. Enter the virtualenv I use for Toaster. This has Python 2.7.5 and the required pip packages to Toaster (as in bitbake/toaster-requirements.txt). 2. Using http://openembedded.org/wiki/OE-Core_Standalone_Setup for instructions, I did: git clone git://git.openembedded.org/openembedded-core oe-core cd oe-core git clone git://git.openembedded.org/bitbake bitbake My oe-core clone is at 1241481 3. Still in the oe-core directory: . ./oe-init-build-env ./build/ ./bitbake/ 4. Now I'm in the build/ directory. export TOASTER_CONF=../.templateconf 5. source toaster start 6. Navigate to http://localhost:8000 in Chrome 7. Click "New project" 8. Enter "gnuradio" as the project name and leave the release set to "OpenEmbedded master". The project seems to be created correctly for me (see screenshot). I did then try to do a build of mpfr-native, which failed, but I did at least manage to create the project. I'll have a look at why the builds are failing next.
Created attachment 3124 [details] Toaster screenshot showing project created
Is the difference where the bitbake directory is located? ./openembedded-core ./bitbake versus ./openembedded-core ./openembedded-core/bitbake Not everyone has their bitbake checkout in the oe-core directory. I'll start over and see what I get.
Yes, I think we have stuff hard-coded in Toaster which shouldn't be. Brian has said he'll let me have the details so I can hopefully fix this.
We can check the COREBASE variable for the path to the OE-Core layer rather than trying to determine its location based on the location of bitbake: $ bitbake core-image-minimal -e | grep ^COREBASE COREBASE_FILES=" oe-init-build-env oe-init-build-env-memres scripts LICENSE .templateconf " COREBASE="/home/joshuagl/Projects/openembedded-core"
Do you guys need me to try anything else?
I'll try to reproduce by using something closer to a non-standard directory setup. I'm also waiting on Brian, as he has some ideas about how to fix this.
I moved my bitbake directory out of my oe-core directory, so I have: oe-core/ bitbake/ Then did: cd oe-core . ./oe-init-build-env ../build/ ../bitbake/ The from my build/ directory, did: export TOASTER_CONF=../oe-core/.templateconf Then: source toaster start I just want to confirm that I'm seeing the same thing as you: when toaster starts like this, I get two error messages: bash: /home/ell/dev/bitbake/bin/../../.templateconf: No such file or directory and Importing file: ../oe-core/.templateconf Failure while trying to import the toaster config file ../oe-core/.templateconf: No JSON object could be decoded Do you see those?
I've ended up with a .templateconf in the oe-core directory. Maybe from hacking around trying to get it working. I thought I retried with a clean setup and it was still there. It looks like I copied my export TAOSTER_CONF line wrong. Maybe check for existence of the .templateconf directory. I'm not familiar enough with toaster internals to understand who creates it and where it belongs.
OK, I can see your "Fields missing: projectversion" error now that I have my directories set up the same way as yours. I will investigate further. Do you see the errors about .templateconf I mentioned?
I made some progress, insofar as I can now create a project. The projectversion error is due to there being no releases in the db. I populated the db with releases using: python ../bitbake/lib/toaster/manage.py loadconf ../oe-core/meta/conf/toasterconf.json (from my build/ directory). I can at least create a project once I have this config. However, trying to build from Toaster produces: ----------------- git clone -b "master" "" "/home/ell/dev/_toaster_clones/_git___git.openembedded.org_openembedded-core_master/bitbake" 2016-04-14 16:18:21,833 WARNING localhostbecontroller: shellcmd error command: git clone -b "master" "" "/home/ell/dev/_toaster_clones/_git___git.openembedded.org_openembedded-core_master/bitbake" fatal: repository '' does not exist 2016-04-14 16:18:21,833 ERROR runbuilds: Error launching build command: git clone -b "master" "" "/home/ell/dev/_toaster_clones/_git___git.openembedded.org_openembedded-core_master/bitbake" fatal: repository '' does not exist Traceback (most recent call last): File "/home/ell/dev/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py", line 67, in schedule "%d:%d" % (br.pk, bec.be.pk)) File "/home/ell/dev/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py", line 255, in triggerBuild layers = self.setLayers(bitbake, layers, targets) File "/home/ell/dev/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py", line 171, in setLayers self._shellcmd("git clone -b \"%s\" \"%s\" \"%s\" " % (bitbake.commit, bitbake.giturl, os.path.join(self.pokydirname, 'bitbake'))) File "/home/ell/dev/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py", line 70, in _shellcmd raise ShellCmdException(err) ShellCmdException: command: git clone -b "master" "" "/home/ell/dev/_toaster_clones/_git___git.openembedded.org_openembedded-core_master/bitbake" fatal: repository '' does not exist 2016-04-14 16:18:22,847 DEBUG Saving error command: git clone -b "master" "" "/home/ell/dev/_toaster_clones/_git___git.openembedded.org_openembedded-core_master/bitbake" fatal: repository '' does not exist (test (Release master (master), BBV master (Branch: master)) failed) ----------------- I will continue to investigate.
I have some tentative commits which (I think) fix this. The attached patch 9455.patch and the instructions below explain how to test the solution. Note that I use a virtualenv for Toaster. This has Python 2.7.5 and the required pip packages for Toaster (as in bitbake/toaster-requirements.txt). 1. Using http://openembedded.org/wiki/OE-Core_Standalone_Setup for hints, I did: cd ~/dev/toaster_oecore_only git clone git://git.openembedded.org/openembedded-core oe-core git clone git://git.openembedded.org/bitbake bitbake (I think this approximately matches the bug reporter's original setup.) 2. Apply 9455.patch (attached to this bug): patch -p1 < 9455.patch 4. Go to the oe-core directory and initialise the bitbake env: cd oe-core . ./oe-init-build-env ../build/ ../bitbake/ 5. Now I'm in the build/ directory; tell toaster where conf files are: export TOASTER_CONF=../oe-core/.templateconf 6. Start toaster: source toaster start 7. Navigate to http://localhost:8000 in Chrome. 8. Click "New project". 9. Enter "gnuradio" as the project name and leave the release set to "OpenEmbedded master". (This is where the original failure occurred.) 10. Do a build of mpfr-native. This was successful for me, but may still fail for other directory layouts. Though I have tested it with the poky-style layout suggested in the Toaster manual, and it works in that situation as well. Please leave any comments if you do/don't get this to work.
Created attachment 3131 [details] 9455.patch - bitbake patches to get Toaster working in a non-poky setup This is a patch for comments before I submit it to the main Toaster mailing list.
Created attachment 3141 [details] Notes about localhostbecontroller.py from initial code investigations
I made some tentative attempts to fix this bug in poky-contrib, elliot/toaster/fix_oe-core_usage-9455 but the work isn't complete.
There are several facets to this issue, all of which interlock and cause problems which are difficult to trace. Below is my current understanding of where the problems are, but there may be things I've missed or misinterpreted. 1. We don't use TOASTER_DIR consistently. It appears that this directory is used to identify the location of the pid and log files for the running server; but also used to identify the location of the bitbake source directory. Changing it can therefore cause several knock-on problems when Toaster tries to check out bitbake. bitbake/bin/toaster also uses BUILDDIR where it should be using TOASTER_DIR. 2. checksettings.py uses TOASTER_DIR + '/build' as the setting for the builddir for the BuildEnvironment object. It should be BUILDDIR, but changing it to this might cause other issues elsewhere in the code. 3. There are several assumptions in the code that the directory structure looks like this: poky (git repo) build bitbake In a non-poky (open-embedded style) setup, it may look more like this: oe-core (git repo) bitbake (git repo) build By default, checksettings.py sets BuildEnvironment.sourcedir to TOASTER_DIR, based on an assumption that TOASTER_DIR is inside the poky git repo. This causes problems in localhostbecontroller, where git repos for layers are cloned. Because of Toaster's assumptions about directory structures, changing TOASTER_DIR (or trying to use BBBASEDIR instead) results in breakages in either the poky or oe setup. For example, if a user changes TOASTER_DIR to a different directory (which isn't inside the bitbake source directory), localhostbecontroller becomes unable to clone the bitbake directory. There is also an assumption in localhostbecontroller that the bitbake directory will be located in <git clone>/bitbake, where the <git clone> is Toaster's clone of the poky directory. In non-poky setups, we don't have a poky directory above the bitbake directory: the bitbake directory is a git repo itself. This also causes problems when we're trying to create a clone of the bitbake repo, as we attempt to clone the directory above the bitbake directory: again, this is fine in the poky case, but breaks in the oe case.
Thanks for looking into this. The OpenEmbedded ecosystem extends far beyond just poky :)
For this we are rewriting the way in which toaster is configured to remove path guessing and any hard coded values so it's going to take a bit longer unfortunately. Though not recommended you can enter the django admin[1] interface on toaster (e.g. http://localhost:8000/admin/) where you can configure a number of these values, be warned however it's not for the faint-hearted! [1] https://docs.djangoproject.com/en/1.8/ref/django-admin/#django-admin-createsuperuser