Bug 15933

Summary: Improving logs in CI builds
Product: [Build System, Metadata & Runtime] BitBake Reporter: Marc Ferland <marc.ferland>
Component: bitbakeAssignee: Marc Ferland <marc.ferland>
Status: RESOLVED NOTABUG QA Contact:
Severity: enhancement    
Priority: Low CC: JPEWhacker, poky.bs.watcher, poky.watcher
Version: unspecified   
Target Milestone: 5.99   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know

Description Marc Ferland 2025-07-14 19:05:18 UTC
Hi,

I'm currently working with the scarthgap release and running Yocto builds in a GitLab CI environment. One issue I'm encountering is that the runners occasionally truncate the build output logs when they exceed a certain size (around 4 MB, from what I can tell). Note: I'm talking about the "realtime" output log viewer from the gitlab web interface.

To mitigate this, I've started using the --quiet option with bitbake, which significantly reduces the volume of output. However, the downside is that it also removes all progress feedback, making it difficult to monitor build status or diagnose issues when builds take a long time.

It would be extremely helpful to have a CI-friendly quiet mode, something like --quiet, but that still gives a clue about the build progress.

Happy to discuss!

Regards,

Marc
Comment 1 Joshua Watt 2025-07-17 14:43:45 UTC
What sort of output are your seeing? If it's a bunch of '#' characters from the progress bars, it's usually because bitbake thinks your terminal is interactive. The usual way to work around that if you can't change your CI terminal is to do `bitbake ... | cat` to force stdout to be a pipe instead of a terminal.

Otherwise, when bitbake is running in a non-interactive terminal, it already prints a "start" and "end" message when each task is run. If this is what you are seeing, I'm not sure there is much we can do that would be a level between print the progress as we do now and not printing it at all.

Can you please clarify what you are seeing and what sort of output you would expect?
Comment 2 Marc Ferland 2025-07-18 13:07:45 UTC
Sure!

I'm currently starting my builds like this:

    $ kas build ${KAS_JOB}:${KAS_COMMON} -- --quiet

The GitLab console is non-interactive (I’ve confirmed this), and
here's the kind of output I see:

    $ kas build ${KAS_JOB}:${KAS_COMMON} -- --quiet
    2025-07-17 19:08:54 - INFO     - kas 4.7 started
    2025-07-17 19:08:55 - INFO     - Running on GitLab CI
    <KAS LAYER CLONING OUTPUT REMOVED FOR BREVITY>
    2025-07-17 19:09:24 - INFO     - Loading cache...done.
    2025-07-17 19:09:31 - INFO     - Parsing recipes...done.
    2025-07-17 19:10:47 - INFO     - Initialising tasks...done.
    <NO OUTPUT FOR A LONG TIME...>
    Uploading artifacts for successful job 01:36
    Uploading artifacts...

As expected, once the build starts (right after the “Initialising
tasks” line), there’s no further output until the end, since --quiet
suppresses progress messages.

What I think would be useful in CI environments like this is an
alternative to --quiet, something like a --non-interactive-progress
option. It would keep output minimal to avoid log truncation but still
print occasional progress updates. For example:

    2025-07-17 19:10:47 - INFO     - Running tasks 10% (58/584)
    2025-07-17 19:12:27 - INFO     - Running tasks 20% (116/584)
    2025-07-17 19:13:13 - INFO     - Running tasks 30% (175/584)
    2025-07-17 19:15:33 - INFO     - Running tasks 40% (233/584)

Basically, it would behave like a non-interactive version of the
progress bar shown in TTYs, giving just enough feedback for humans to
know the build is alive without overwhelming the log.
Comment 3 Joshua Watt 2025-07-31 14:51:14 UTC
I'm pretty sure this is because you are running in a kas container, which is configured to make bitbake think it is running in an interactive terminal (this can be a common problem with containers). As such, please take this up with the kas project instead.

If you can reproduce outside of kas, feel free to reopen this bug
Comment 4 Joshua Watt 2025-07-31 14:53:44 UTC
For reference, Pyrex (another container system for bitbake) does this: https://github.com/garmin/pyrex/blob/87910c0db40e19a5cd2ca34361eebcf70bea0551/pyrex.py#L545 which helps to prevent these sorts for problems.