Just a quick note so that I don't forget: if we don't manage to do this before 2.0, we'll need to put some hack in place to hide the start builds and create projects functionality in analysis mode.

Notes from other people for me to remember:

Analysis Mode: Sadly, it appears that master in analysis mode is totally broken. at least when all it sees is setscene repopulates. trying the full build next. Full build on master doesn't fix anything. pretty much totally broken.
	•	directory structure not there when build image
	•	under configuration it shows : Layers __FIXME__unidentified_layer
	•	under task-> bash - layer show __FIXME__unidentified_layer and Packages show 0
	•	Packages list is empty.
	•	Performance cpu usage empty
	•	Performance Disk io empty

-----

The only real difference is that analysis mode starts up a bitbake memres server from the startup and listens for events.
in managed mode the localbuilder starts a bitbake server on demand, and listens to events. 
To squash all we really need is to have the local guy check and see if he's getting build events and wait until done to launch his own.
If the memres server were stable we could probably just use the one the whole time but I fear it's not quite there yet. but, we can take it down and put up a local buil guy and then whent thta build is done, pop up another memres guy.

So, when we start in analysis mode we have a bb env we inherit based on the user having previously run oe-init-build-env. then sourcing bitbake/bit/toaster start   starts up:
1) bitbake --postread conf/toaster.conf --server-only -t xmlrpc -B 0.0.0.0:0
2) bitbake --observe-only -u toasterui
where toaster.conf has : INHERIT+="toaster buildhistory"

the bitbake observer listens to events and feed them into toaster. the important environment variable set is BBSERVER=0.0.0.0:-1
bitbake commands you run by hand use the BBSERVER var to know to communicate with the memres server.

When we start in managed mode:
In the managed mode  case. we run oe-init-build-env-memres which starts the xmlrpc bitbake server and sets the important BBSERVER environment variable. 

we start up on demand (aka when an entry appears in the build request table):
1)bitbake --read toaster-pre.conf --postread toaster.conf --server-only -t xmlrpc -B 0.0.0.0:0
2)bitbake --observe-only -u toasterui --remote-server=0.0.0.0:-1 -t xmlrpc
percent has in it the default toaster setup:
IMAGE_INSTALL_append=""
PACKAGE_CLASSES="package_rpm"
MACHINE="qemux86"
SDKMACHINE="x86_64"
DISTRO="poky"
IMAGE_FSTYPES="ext3 jffs2 tar.bz2"
TOASTER_BRBE ="1:1"


toaster.conf has:
INHERIT+="toaster buildhistory"

——

The only other difference is that the managed mode adds the following env variables:
DATABASE_URL - tells toaster what db to use.
and TOASTER_BRBE which is a way to connect a build request to a build environment.  we will only have 1 build env in the 2.0 (and probably going forward as well).

so, at first look over. they are quite similar assuming we are running toaster collocated w/ bitable; which is our 2.0 use case.

I have tested and confirmed that bitbake is happy running with —read toaster-pre.conf when toaster-pre.conf is an empty file.  This means we can default to having it read the toaster-pre.conf and just put the toaster stuff in when we are controlling the build.  note, I am assuming/enforcing 1 build at a time…  bitable itself enforces the same thing effectively.

 we will only have 1 build env in the 2.0 (and probably going
> forward as well).

I quite disagree with the 'going forward' bit, and I think I can provide evidence from user interviews as to why. 

If Toaster is ever going to be anything more than a toy or a good demo, it must be able to play nice with big build infrastructure. And that involves being able to run as a single instance with multiple users, and deal with more than one build environment so that those users can start their builds without queuing.  

We can't simply disregard everything we've learnt from users over the past 3 years and make decisions about Toaster based on personal opinions.

Fix has been sent for review: https://lists.yoctoproject.org/pipermail/toaster/2015-October/002946.html

looks like I misunderstood the scope of the bug. I thought it's only about running bitbake the same way, but it's more than that.

Here is the list of tasks Brian pointed out to on #toaster:

1) we need to make sure that sourcing allows us to pull in layers etc.
2) we need to make sure that we use the bb variables to tell us where things are.  
3) we need to make sure the builds we kick off use the existing bb server, or kill it gracefully and start a new one.  This is impt because it's not clear how stable the memres server is so it may be safer in the short term to make a new one. 
4) we probably need a toaster_pid dir to keep the pids of all the pieces we want to kill when we are stopped. there are ~ 4 processes right now we need to kill on exit: bbserver, bbobserver, the webserver, and the bldcontrol

Send first version of the patchset for review: https://lists.yoctoproject.org/pipermail/toaster/2015-October/003083.html

I made command line builds to use the same way to run bitbake server and toasterui observer, but the issues mentioned in this bug are still there. I'll continue working on them.

(In reply to comment #7)
> Send first version of the patchset for review:
> https://lists.yoctoproject.org/pipermail/toaster/2015-October/003083.html
> 
> I made command line builds to use the same way to run bitbake server and
> toasterui observer, but the issues mentioned in this bug are still there.
> I'll continue working on them.

I ran a CLI build with this first version, and I can still see some data collection issues. I built core-image-minimal on a clean build directory. I got:

* variables information
* tasks information 
* time per task information
* a list of recipes built

I am missing:

* layer information (they all appear as __FIXME__unidentified_layer)
* package information (both built and installed)
* file information (directory structure is empty)
* CPU and disk I/O information per task

From the missing information, the most important is the layers, packages and files.

So then I started a build from Toaster, and we are missing different information. We have:

1. variable history
2. layer names and commits in the configuration history page
3. layer names and commits in the recipes table
4. layer name in the packages built table
5. task information
6. time per task
7. CPU usage per task
8. Disk I/O per task
9. File information (directory structure page shows data)

We don't have:

1. Layer branches in the configuration history page
2. Recipe dependency information
3. Recipe details are empty if you navigate to them from the recipes table (but populated except for layer branch and commit if you navigate to them from the tasks table or the package details pages)
4. Layer branch and commit in the packages tables (built and installed)

Now, some of the issues above, at least number 3 in the 'we don't have' section, I thought they were fixed by some of recent Michael's patches, but I am not sure if those patches are not in the ed/toaster/8279-v2 branch right now, of if they are back.

It turned out that most of missing information caused by bitbake bug and should be fixed by this commit: http://lists.openembedded.org/pipermail/bitbake-devel/2015-October/006595.html

Belen, please confirm that neither of mentioned issues are present in toaster-next anymore or point out to those that still there.

(In reply to comment #10)
> It turned out that most of missing information caused by bitbake bug and
> should be fixed by this commit:
> http://lists.openembedded.org/pipermail/bitbake-devel/2015-October/006595.
> html
> 
> Belen, please confirm that neither of mentioned issues are present in
> toaster-next anymore or point out to those that still there.

Right now, in toaster-next:

* In build mode, we are only missing the layer branch information (layer names and commits are there)

* In analysis mode, we are still missing tons of information, but it looks like we are missing this patch of yours

http://lists.openembedded.org/pipermail/bitbake-devel/2015-October/006596.html

which fixes the INHERIT issue.

Moving to 2.1 as we decided it's too risky to have it in 2.0

(In reply to comment #11)
> Right now, in toaster-next:
> 
> * In build mode, we are only missing the layer branch information (layer
> names and commits are there)
> 
> * In analysis mode, we are still missing tons of information, but it looks
> like we are missing this patch of yours
> 
> http://lists.openembedded.org/pipermail/bitbake-devel/2015-October/006596.
> html
> 
> which fixes the INHERIT issue.

Pushed fix for INHERIT issue to toaster-next.

Now all information seems to come in for command line builds, but the layer branch information is till missing for builds started from Toaster. I'll open a bug for that.

After discussing my RFC with toaster team I decided to go ahead and rebase my implementation and send it for final review. Please let me know if you have any objections.

sent for review: https://lists.yoctoproject.org/pipermail/toaster/2015-October/003209.html

Working on fixing bugs mentioned in review comments:
1. If you want to start a build from Toaster, you need to remove the
SDKMACHINE variable from the BitBake variables page in the project
configuration. Otherwise builds fail. We need to make sure we remove that
variable from the toasterconf.json files (as separate patches I guess).

2. For builds started from CLI, all data is collected. But for the build I
started from Toaster, the following data was missing:

* The recipes table is empty (http://i.imgur.com/sxqeqxp.png), but if you
navigate to a recipe from the package details page, the data is there
(http://i.imgur.com/FTcbwA2.png)

* The layer branch and commit are missing from packages tables and recipe
details pages (http://i.imgur.com/6agaG6L.png). But they show in the build
configuration page (http://i.imgur.com/Bt1XFxv.png)

fixed review comment 1: https://lists.yoctoproject.org/pipermail/toaster/2015-November/003254.html

Issues I came across:
Biggish:
1) brian's bug 7880 is causing problems here. We need to enable toaster to use diff build dirs on a per project basis and share nothing but sstate and downloads (configurable from the ui)
      -- for instance, if I start bitbake with . <path>/oe-init-build-env build-cli toaster cannot manage builds since it is assuming a build dir named "build".
      -- this would also address Belen’s issue where a cli build is running and we simultaneously start a toaster-build.
     — this could be done after the merge for bug 8279
2)  if we die early (in parse where it says nothing provides alsalib for instance) all subsequent builds fail and I couldn’t get builds to work again from toaster w/o a restart of the whole env/ (i didn’t try part restarts)
    — I feel we need to fix this b4 we can merge

especially because of #2, I would end up in a position where I needed to kill all the bb processes by hand.

Middling:
1) Our event handling needs to include the parse stage.  This isn’t directly related to this enhancement but it was highlighted by this work since it let me trip across it a lot .
     —  if the cli build doesn’t need to actually execute any tasks (e.g. is set scenes from sstate, we don’t see it in toaster - according to Belen, always been true but parse would make it appear
     — if we die early (in parse where it says nothing provides alsalib for instance) our build doesn’t know we stopped  we just see a 0% build…
     — this could be done after the merge for bug 8279

2) Our stopping needs work 
      — toaster stop needs to unset the env variables so that we can restart if we change something things like TOASTER_DIR DATABASE_URL BUILDDIR
      — toaster stop needs to clean up the bb processes it may have started. I ended up with an awk killall script because I kept having to kill all the bb processes by hand 
      — I feel we need to fix this b4 we can merge

(In reply to comment #19)

> 
> Middling:
> 1) Our event handling needs to include the parse stage.  This isn’t directly
> related to this enhancement but it was highlighted by this work since it let
> me trip across it a lot .
>      —  if the cli build doesn’t need to actually execute any tasks (e.g. is
> set scenes from sstate, we don’t see it in toaster - according to Belen,
> always been true but parse would make it appear
>      — if we die early (in parse where it says nothing provides alsalib for
> instance) our build doesn’t know we stopped  we just see a 0% build…

No no, what has always been true is builds started from cli that die at the parse stage not appearing in Toaster. Builds running only setscene tasks did appear before.

Basically that (not getting cli builds that failed at parse time) was always true because we were not setting the event filter from toasterui.py.  Now that we are starting to collect events from managed builds at the ParseStarted event we should also collect builds from the cli at ParseStarted so they are the same.  I sent a patch to Elliot to turn this on.

I'll try to fix 3 issues mentioned above:

- if we die early (in parse where it says nothing provides alsalib for instance) all subsequent builds fail and I couldn’t get builds to work again from toaster w/o a restart of the whole env/ (i didn’t try part restarts)

— toaster stop needs to unset the env variables so that we can restart if we change something things like TOASTER_DIR DATABASE_URL BUILDDIR

— toaster stop needs to clean up the bb processes it may have started. I ended up with an awk killall script because I kept having to kill all the bb processes by hand 


Please let me know if something else has to be fixed to merge my patchset.

I can't reproduce the issue number one. Here is what I did:

1. Added 'DEPENDS += "bla"' line to zlib recipe

2. started command line build and it failed:
$ bitbake busybox
WARNING: Host distribution "openSUSE-13.2" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
Parsing recipes: 100% |########################################################################################################| Time: 00:00:20
Parsing of 887 .bb files complete (0 cached, 887 parsed). 1311 targets, 37 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
ERROR: Nothing PROVIDES 'bla-native' (but virtual:native:/home/ed/git/yocto/poky/_toaster_clones/_git___git.yoctoproject.org_poky_master/meta/recipes-core/zlib/zlib_1.2.8.bb DEPENDS on or otherwise requires it). Close matches:
  vala-native
  bjam-native
  db-native
ERROR: Required build target 'busybox' has no buildable providers.
Missing or unbuildable dependency chain was: ['busybox', 'rpm-native', 'zlib-native', 'bla-native']

3. reverted the change - removed previously added line from zlib recipe.

4. run the build again and it succeeded:
$ bitbake busybox
WARNING: Host distribution "openSUSE-13.2" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
Parsing recipes: 100% |########################################################################################################| Time: 00:00:12
Parsing of 887 .bb files complete (0 cached, 887 parsed). 1311 targets, 37 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION        = "1.28.0"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "openSUSE-13.2"
TARGET_SYS        = "x86_64-poky-linux"
MACHINE           = "qemux86-64"
DISTRO            = "poky"
DISTRO_VERSION    = "2.0"
TUNE_FEATURES     = "m64 core2"
TARGET_FPU        = ""
meta-yocto
meta-yocto-bsp    = "master:9bd13bac55faf37ba84da6769f32a5d584bd83cc"
meta              = "ed/toaster/8279-v3:1b25a70991589ed1f123015c16ee4806c46e3199"

NOTE: Preparing RunQueue
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Tasks Summary: Attempted 694 tasks of which 693 didn't need to be rerun and all succeeded.

Summary: There was 1 WARNING message shown.

5. Saw the build in UI.

6. Started new build from UI.

7. Saw it succeeded.

8. Checked running bitbake processes - both server and observer processes are running:
$ ps aux | grep bitbake
ed        1876  0.0  0.1 122612 33404 pts/1    S    12:47   0:00 python ../bitbake/bin/../lib/toaster/manage.py runserver 0.0.0.0:8080
ed        1886  1.4  0.2 659992 76408 pts/1    Sl   12:47   0:20 /usr/bin/python ../bitbake/bin/../lib/toaster/manage.py runserver 0.0.0.0:8080
ed        1922  0.9  0.2 178628 68228 pts/1    S    12:47   0:13 python ../bitbake/bin/../lib/toaster/manage.py runbuilds
ed        5091  8.8  0.5 508772 190388 ?       Sl   13:07   0:16 python /home/ed/git/yocto/poky/bitbake/bin/bitbake --postread conf/toaster.conf --server-only -t xmlrpc -B 0.0.0.0:0
ed        5104  7.1  0.2 263264 78744 pts/1    Sl   13:07   0:13 python /home/ed/git/yocto/poky/bitbake/bin/bitbake --observe-only -u toasterui --remote-server=0.0.0.0:-1 -t xmlrpc
ed        5835  0.0  0.0  10520  1600 pts/1    S+   13:11   0:00 grep --color=auto bitbake


Please, advice how to reproduce this issue.

Sent updated patchset: https://lists.yoctoproject.org/pipermail/toaster/2015-November/003369.html

please review.

The patchset contains only one additional commit - unsetting environment variables.

Regarding issue number 3 - I need more details on that as current code seems to kill all processes. BTW, I'm not sure it's related to this bug at all.

Heres how to reproduce my "getting stuck" issue:
screenshots available -> https://www.dropbox.com/sh/hnzb6b2cltxdbvz/AAAVK4DYRnZjk7P0K7hE_dvZa?dl=0
1) new proj on master , branch ed/8279-V5
2) build zlib 
  -- done from toaster ui
  -- this works fine
3) build alsalib 
  -- done from toaster ui
  -- this fails. note there is an alsa-lib but i deliberately mistyped alsalib
4) build alsa-lib
  -- done from toaster ui
  this *shoul* work but instead nothing happens.
5) look at toaster_ui.log 
  -- you see ERROR: UNSAVED log: Nothing PROVIDES 'alsalib'
  -- and it doesn't notice the alsa-lib request
  -- also, the bovserver seems to have exited:
     -- [2]-  Done                    bitbake --observe-only -u toasterui --remote-server=$BBSERVER -t xmlrpc >> ${BUILDDIR}/toaster_ui.log 2>&1


This is the bug I think we need to fix before we merge.

Fixed issue number one. Brian, thank you for the explanations!

Please, review: https://lists.yoctoproject.org/pipermail/toaster/2015-November/003407.html

it's in good enough shape to merge.

The only way I could trick it is the following:
1) build bash from the ui
2) build frog from the cli : bitbake frog
 -- bitbake frog fails.  That failure is noted by the UI but the failure is applied to the previously successful bash build which now turns red.
3) further successful builds from either the ui or cli seem unaffected.

patchset has been accepted upstream: http://cgit.openembedded.org/bitbake/commit/?id=7a2e9745ff3ef6d6360f74b222248d4bd35e6a08

There are 2 commits that has been rejected by upstream though. I'm working on that.

Reworked rejected patches: http://lists.openembedded.org/pipermail/bitbake-devel/2015-December/006745.html

2nd patchset has been accepted upstream: http://cgit.openembedded.org/bitbake/commit/?id=d7819508dac488a64be3caec88db285cda9599ab

Since Jethro there is a single way to start Toaster.
Put this in verified.