Bug 8209 - Impossible to download cooker log from Toaster
Summary: Impossible to download cooker log from Toaster
Status: VERIFIED FIXED
Alias: None
Product: Toaster
Classification: Build System, Metadata & Runtime
Component: toaster (show other bugs)
Version: 1.8
Hardware: x86 Multiple
: Medium normal
Target Milestone: 2.0
Assignee: Elliot Smith
QA Contact: Alexandru Roman
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2015-08-26 09:48 UTC by Belen Barros Pena
Modified: 2015-11-09 10:35 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Yes (doc changes required)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Belen Barros Pena 2015-08-26 09:48:52 UTC
Found in master:393bd7496d71eb101f21234c1233a2d18fd2c73e

For any builds you run, you are available to download the following artifacts from Toaster:

* Image files
* Other artifacts
* License manifest file
* Task logs

However, when you try to download the cooker log, you get the "The build artifact you are trying to download no longer exists." message.
Comment 1 Elliot Smith 2015-09-07 13:33:38 UTC
The cooker log for a build is copied by the runbuilds command from the "real" build environment to a toaster artifacts directory, keyed by the ID of the build.

e.g. artifacts for build 1 go into toaster_build_artifacts/1 (NB toaster_build_artifacts is a customisable directory name).

runbuilds is run once per second (!) by bitbake/bin/toaster. runbuilds initiates the copying of build artifacts from bitbake to the toaster artifacts directory for the build. Only build requests with status BuildRequest.REQ_ARCHIVE are queued up to have their logs copied. One possibility is that build requests don't have or get this status (perhaps they used to), so runbuilds ignores them.

The output for runbuilds goes into /tmp/toaster_N/, where N is the process ID of the original bitbake/bin/toaster command.

runbuilds.py (in bitbake/lib/toaster/bldcontrol/management/commands) contains the command which does the copying, archive().

archive() constructs a new cooker log filename of the form toaster_build_artifacts/$build number$/cooker_log.txt, then writes the "real" log file's content into that file.

Note that the build "knows" where its log file is (Build.cooker_log_path), so I'm not sure why we're not just using that. Perhaps because that would require the user to keep their bitbake logs. This might be a problem if a user cleared out any build detritus periodically (e.g. by deleting their old build directories), but we want toaster to be able to keep its logs. However, it would make toaster a lot simpler if it just pointed to the artifacts already in bitbake.

This new log filename is the one which toaster tries to read (to send the log back to the user when they click the link in the UI).

The view which handles requests for build artifacts is build_artifact() (in toastergui/views.py). The template used is unavailable_artifact.html (in toastergui/templates).

When I tried to reproduce the bug, I modified the code so that the real error is shown by the view's template when the download fails. This reveals that the cooker_log.txt file is not being correctly created, so the expected file doesn't exist. This is why the view fails and shows the error message. I checked the filesystem and there was no sign of the cooker_log.txt file where it was supposed to be. The directory for the build (toaster_build_artifacts/1) was there, however.
Comment 2 Elliot Smith 2015-09-07 16:07:46 UTC
When a build request is successfully completed, its status is set to 3 (BuildRequest.REQ_COMPLETED). It looks like this means it won't be queued up to have its log copied to the toaster build artifacts directory, as its status never gets to BuildRequest.REQ_ARCHIVE (and runbuilds looks for BuildRequest objects with status BuildRequest.REQ_ARCHIVE).

The only place I can find where the BuildRequest's status is set to REQ_ARCHIVE is in bitbake/lib/bb/ui/buildinfohelper.py, in the BuildInfoHelper._store_build_done() method.

This method is invoked from BuildInfoHelper.close(), which appears to store done build requests, but only if the BuildInfoHelper has a brbe property set.

This brbe property can be set from the TOASTER_BRBE environment variable, which in turn is set in the bitbake/bin/toaster script. However, it also appears to be set automatically in runbuilds if it is not in the environment. This variable is saved into the database.

(It's not clear what brbe actually stands for; "build request build environment"? In runbuilds.py it's called the "build identification variable", and has the format "$build request id$:$build environment controller.build environment$".)
Comment 3 Elliot Smith 2015-09-07 18:40:57 UTC
After more digging: BuildInfoHelper._get_build_information() sets the cooker_log_path for a build from the BB_CONSOLELOG variable.

The time stamped into the value for this variable seems to be about a minute outside the correct time. For example, the log file which is supposed to exist is called:

/home/ell/dev/toaster/poky-contrib/build/tmp/log/cooker/qemux86/20150907181435.log

But the nearest log file is actually:

/home/ell/dev/toaster/poky-contrib/build/tmp/log/cooker/qemux86/20150907181339.log

Note that the timestamp is 56 seconds before the expected timestamp. It seems as though the actual log file name is slightly different from the expected one produced by bitbake.
Comment 4 Belen Barros Pena 2015-09-08 09:28:00 UTC
> 
> Note that the build "knows" where its log file is (Build.cooker_log_path),
> so I'm not sure why we're not just using that. Perhaps because that would
> require the user to keep their bitbake logs. This might be a problem if a
> user cleared out any build detritus periodically (e.g. by deleting their old
> build directories), but we want toaster to be able to keep its logs.
> However, it would make toaster a lot simpler if it just pointed to the
> artifacts already in bitbake.

This brings up the question of what happens to the task logs. Are they also stored in that artifacts directory? Or is Toaster just pointing to the logs created by BitBake?

In any case, I think it should be fine to just use what's created by BitBake instead of the black magic going on at the moment. Yes, you will lose the logs if you delete the build directory, but that is what currently happens with all other artifacts any way.
Comment 5 Elliot Smith 2015-09-10 10:50:40 UTC
Submitted to toaster mailing list for review.
Comment 6 Elliot Smith 2015-09-11 14:39:49 UTC
Making modifications after review by Brian Avery:

This fixed it for the managed build case but it failed for the
analysis mode case.

Environment:


Request Method: GET
Request URL: http://192.168.168.66:8000/toastergui/build/1/artifact/cookerlog/id/1

Django Version: 1.6
Python Version: 2.7.6
Installed Applications:
('django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.messages',
 'django.contrib.sessions',
 'django.contrib.admin',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'bldcollector',
 'toastermain',
 'south',
 'toastergui',
 'bldcontrol',
 'orm')
Installed Middleware:
('django.middleware.common.CommonMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware')


Traceback:
File "/big/src/intel/yocto-builds/TOASTER/myBugs/t-myBugs/local/lib/python2.7/site-packages/django/core/handlers/base.py"
in get_response
  114.                     response = wrapped_callback(request,
*callback_args, **callback_kwargs)
File "/big/src/intel/myBugs/poky/bitbake/lib/toaster/toastergui/views.py"
in build_artifact
  2745.                 build_artifact_storage_dir =
os.path.join(ToasterSetting.objects.get(name="ARTIFACTS_STORAGE_DIR").value,
"%d" % int(build_id))
File "/big/src/intel/yocto-builds/TOASTER/myBugs/t-myBugs/local/lib/python2.7/site-packages/django/db/models/manager.py"
in get
  151.         return self.get_queryset().get(*args, **kwargs)
File "/big/src/intel/yocto-builds/TOASTER/myBugs/t-myBugs/local/lib/python2.7/site-packages/django/db/models/query.py"
in get
  307.                 self.model._meta.object_name)

Exception Type: DoesNotExist at /toastergui/build/1/artifact/cookerlog/id/1
Exception Value: ToasterSetting matching query does not exist.
Comment 7 Belen Barros Pena 2015-09-18 11:40:31 UTC
Changing the documentation flag to 'yes': we need to remove the layer checkout directory step from the set up process descriptions.
Comment 8 Elliot Smith 2015-09-21 15:59:56 UTC
Merged into bitbake master.
Comment 9 Alexandru Roman 2015-11-09 10:35:35 UTC
Verified on master: fc45deac89ef63ca1c44e763c38ced7dfd72cbe1