Bug 8225

Summary: nightly-arm-lsb SendErrorReport failed with INTERNAL SERVER ERROR
Product: [Infrastructure] AutoBuilder Reporter: Jussi Kukkonen <jku>
Component: autobuilderAssignee: Mariano Lopez <mariano.lopez>
Status: VERIFIED FIXED QA Contact: Bogdan Alexandru Voiculescu <bogdanx.a.voiculescu>
Severity: normal    
Priority: Medium+ CC: bluelightning, bogdanx.a.voiculescu, elizabeth.flanagan, infras.ab.watcher, Infras.watcher, mariano.lopez, richard.purdie
Version: unspecified   
Target Milestone: 2.0   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Bug Depends on:    
Bug Blocks: 8249    
Attachments:
Description Flags
failing error report file
none
systemd error report none

Description Jussi Kukkonen 2015-08-31 10:44:34 UTC
https://autobuilder.yoctoproject.org/main/builders/nightly-arm-lsb/builds/458/steps/SendErrorReport/logs/stdio

--- 
### Shell environment set up for builds. ###

You can now run 'bitbake <target>'

Common targets are:
    core-image-minimal
    core-image-sato
    meta-toolchain
    adt-installer
    meta-ide-support

You can also run generated qemu images with a command like 'runqemu qemux86'
ERROR: INTERNAL SERVER ERROR
Preparing to send errors to: errors.yoctoproject.org
program finished with exit code 1
---
Comment 1 Beth Flanagan 2015-08-31 11:49:48 UTC
Reassigning to halstead as this is something infrastructure wise. My guess is the error reporting server was down or somehow unavailable.
Comment 2 Richard Purdie 2015-09-09 13:21:25 UTC
Has anyone tried saving the file that caused this problem and manually submitting it?

I seriously doubt this is an infraastructure issue, we also see "internal server error" when something breaks in submission (invalid characters, log too long, etc).
Comment 3 Richard Purdie 2015-09-12 07:27:39 UTC
Created attachment 2729 [details]
failing error report file

poky$ ./scripts/send-error-report -y  -l http://autobuilder.yoctoproject.org/main/builders/nightly-arm-lsb/builds/479 /home/richard/error_report_20150912005909.txt -n yocto-autobuilder-autogenerated-ubuntu1204
Preparing to send errors to: errors.yoctoproject.org
ERROR: INTERNAL SERVER ERROR
 (reproduces with the attached file, saved from the autobuilder)
Comment 4 Paul Eggleton 2015-09-23 13:00:32 UTC
*** Bug 8382 has been marked as a duplicate of this bug. ***
Comment 5 Paul Eggleton 2015-09-23 13:03:26 UTC
Somewhat concerning as well is that the output order in the autobuilder log is incorrect - we're seeing the INTERNAL SERVER ERROR line then the one about sending the error report to the server. Is this perhaps the result of receiving stdout and stderr separately and then concatenating them?
Comment 6 Richard Purdie 2015-09-23 14:20:02 UTC
I think these is from "interesting" characters in the log output, e.g. from systemd in the lastest failure. We're likely not escaping/encapsulating these correctly when sending to the server.
Comment 7 Mariano Lopez 2015-09-23 14:44:14 UTC
Created attachment 2765 [details]
systemd error report

This attachment is the error that Richard mentioned in the previous comment.
Comment 8 Mariano Lopez 2015-09-23 15:10:50 UTC
It seems the error report web server doesn't support Unicode characters
Comment 9 Mariano Lopez 2015-09-25 13:48:01 UTC
The error report class take the output from the bitbake log, and there is some output (qemu output and  in there that are not sanitized and contains control characters and sometimes Unicode characters. So far the only Unicode character used is the copyright symbol. So I will sanitize the qemu output and the error report web won't have any problem.
Comment 10 Mariano Lopez 2015-09-25 14:45:04 UTC
Sent patch for review
Comment 11 Mariano Lopez 2015-09-30 14:38:56 UTC
The patch to sanitize the output from qemu is in master-next at the moment.

I sent another patch to support Unicode logs in the Error Report Web and has been applied in tree and the live server:

http://git.yoctoproject.org/cgit/cgit.cgi/error-report-web/commit/?id=758ddda65e30eb1fd27477092da4c2bedc42fbb0

I will work on sanitizing the output from qemu without removing Unicode characters
Comment 12 Mariano Lopez 2015-10-01 18:51:22 UTC
Patches sent to the mailing list.
Comment 14 Bogdan Alexandru Voiculescu 2015-10-23 12:21:14 UTC
Error has not been reproduced after update
Checked on another failed build: 
https://autobuilder.yoctoproject.org/main/builders/nightly-arm-lsb/builds/458/steps/SendErrorReport/logs/stdio

.......................
You can also run generated qemu images with a command like 'runqemu qemux86'
Preparing to send errors to: errors.yoctoproject.org
Your entry can be found here: http://errors.yoctoproject.org/Errors/Build/9175/
Preparing to send errors to: errors.yoctoproject.org
Your entry can be found here: http://errors.yoctoproject.org/Errors/Build/9176/
.........................