Bug 14364

Summary:	S3 fetcher: missing ProgressHandler for S3 fetcher
Product:	[Build System, Metadata & Runtime] BitBake	Reporter:	Przemyslaw Gorszkowski <pgorszkowski>
Component:	bitbake	Assignee:	Richard Purdie <richard.purdie>
Status:	RESOLVED FIXED	QA Contact:
Severity:	enhancement
Priority:	Medium+	CC:	poky.bs.watcher, poky.watcher, randy.macleod
Version:	unspecified
Target Milestone:	3.4 M2
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	Don't know

Description Przemyslaw Gorszkowski 2021-04-26 06:56:45 UTC

If there is a S3 source for fetch jobs there is no progress bar at all.The reason of that is missing ProgressHandler for S3 fetcher (lib/bb/fetch2/s3.py).
Additionally due to the fact that aws-cli progress output ends with "\r" there is also missing support for such case in LineFilterProgressHandler (https://github.com/openembedded/bitbake/blob/master/lib/bb/progress.py#L79)

aws-cli progress output implementation:
https://github.com/aws/aws-cli/blob/develop/awscli/customizations/s3/results.py#L520

Comment 1 Przemyslaw Gorszkowski 2021-04-26 07:51:05 UTC

I created a fix for the problem with LineFilterProgressHandler:
https://github.com/openembedded/bitbake/pull/29

Comment 2 Przemyslaw Gorszkowski 2021-04-26 08:25:14 UTC

I added also a progress handler for S3 cp command to enable progress when we fetch a data from S3 source:
https://github.com/openembedded/bitbake/pull/30

Comment 3 Randy MacLeod 2021-04-29 15:02:31 UTC

Patch on the list

Comment 4 Richard Purdie 2021-05-02 22:54:28 UTC

Thanks for the patches, they have merged now: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=6da327c788b5fc2f50897771eeb82de1cc08e4e7