Bug 14364 - S3 fetcher: missing ProgressHandler for S3 fetcher
Summary: S3 fetcher: missing ProgressHandler for S3 fetcher
Status: RESOLVED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ enhancement
Target Milestone: 3.4 M2
Assignee: Richard Purdie
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2021-04-26 06:56 UTC by Przemyslaw Gorszkowski
Modified: 2021-05-02 22:54 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Przemyslaw Gorszkowski 2021-04-26 06:56:45 UTC 
If there is a S3 source for fetch jobs there is no progress bar at all.The reason of that is missing ProgressHandler for S3 fetcher (lib/bb/fetch2/s3.py).
Additionally due to the fact that aws-cli progress output ends with "\r" there is also missing support for such case in LineFilterProgressHandler (https://github.com/openembedded/bitbake/blob/master/lib/bb/progress.py#L79)

aws-cli progress output implementation:
https://github.com/aws/aws-cli/blob/develop/awscli/customizations/s3/results.py#L520
Comment 1 Przemyslaw Gorszkowski 2021-04-26 07:51:05 UTC 
I created a fix for the problem with LineFilterProgressHandler:
https://github.com/openembedded/bitbake/pull/29
Comment 2 Przemyslaw Gorszkowski 2021-04-26 08:25:14 UTC 
I added also a progress handler for S3 cp command to enable progress when we fetch a data from S3 source:
https://github.com/openembedded/bitbake/pull/30
Comment 3 Randy MacLeod 2021-04-29 15:02:31 UTC 
Patch on the list
Comment 4 Richard Purdie 2021-05-02 22:54:28 UTC 
Thanks for the patches, they have merged now: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=6da327c788b5fc2f50897771eeb82de1cc08e4e7