Bug 1590

Summary: Archive work directory
Product: [Build System, Metadata & Runtime] Meta-yocto Reporter: Richard Purdie <richard.purdie>
Component: meta-yoctoAssignee: Wenzong.Fan <wenzong.fan>
Status: RESOLVED DUPLICATE QA Contact:
Severity: enhancement    
Priority: Low CC: jessica.zhang, liezhi.yang, mark.hatle, poky.bs.watcher, poky.watcher, richard.purdie, sgw, song.liu, Xiaofeng.yan, yi.zhao
Version: unspecified   
Target Milestone: 1.2 M3   
Hardware: x86   
OS: Multiple   
Whiteboard: (P2) (Test Case Ready) See 1977
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: ---

Description Richard Purdie 2011-10-04 07:12:18 UTC
Some legal departments believe a complete archive of the work directory is required for certain license compliance issues. We could therefore do with a class which archives up the work directories at the end of each build and provides them so those legal departments can be happy.
Comment 1 Wenzong.Fan 2011-12-07 01:16:30 UTC
Hi Richard,

I need some more details about this requirement, just copy my questions here:
1) What stuffs we should archive up?
The whole 'work/' dir include all binaries/logs...? Or just the sources code for all built packages would be OK.

2) What type of build will trigger the archive process?
I think for 'each build' is a little redundancy, should we just run it for 'core-image*' or any others?

If you have any advises/suggestions please tell me kindly, I suspect you would be the reviewer for the requirement :)

Thanks
Wenzong
Comment 2 Richard Purdie 2011-12-12 16:08:06 UTC
This request is one I transfered into the bugzilla from other sources. I believe the request was for all the sources in WORKDIR (after patching) along with the scripts within the temp directory that were used to build it. The presence of the binary components of the build is not required and would waste space although including them would make creating the achieve easier.

The process needs to be triggered when each workdir is finished with and it would be tarballed at this point.

The functionality should be in a separate class which the user can optionally include.
Comment 3 Wenzong.Fan 2011-12-12 17:54:54 UTC
(In reply to comment #2)
> This request is one I transfered into the bugzilla from other sources. I
> believe the request was for all the sources in WORKDIR (after patching) along
> with the scripts within the temp directory that were used to build it. The
> presence of the binary components of the build is not required and would waste
> space although including them would make creating the achieve easier.
> 
> The process needs to be triggered when each workdir is finished with and it
> would be tarballed at this point.
> 
> The functionality should be in a separate class which the user can optionally
> include.

So it will be a optional functionality, not default to any builds.

That's clear for me, thanks Richard.
Comment 4 Robert Yang 2011-12-14 22:50:30 UTC
*** Bug 1655 has been marked as a duplicate of this bug. ***
Comment 5 Wenzong.Fan 2011-12-15 22:54:16 UTC
Design Doc for this requirement:
--------------------------------
* Purpose:

Export/archive the patches sources to assist in license compliance by the end
user or legal departments. 

* Analysis:
1) What should be archive up?

All sources in $WORKDIR(after patching) and the build scripts which under
'$WROKDIR/temp' dir should be archive up;

2) When the archive process should be trigger?

The archive task should be triggered at the end of each build, that means
all required patches have been applied to source code and all build scripts
have been generated.

3) Where the functionality should be put to?

The functionality should be in a separate class which the user can optionally
include.

* Implement:
Add a new class 'export_sources.bbclass' to provide task 'do_export_sources()',
and get it called after 'do_install()' before 'do_populate_sysroot()'.

The Sources that need to be archived up:
$S, $T/run.*

The Destinations/tarballs: 
${TMPDIR}/deploy/sources/${MULTIMACH_TARGET_SYS}/${PF}.tar.bz2

* Usage:

If any packages need to be archived up, just make sure inherit the 'export_sources' class their bb files.
Comment 6 Wenzong.Fan 2011-12-19 00:04:47 UTC
(In reply to comment #5)
> Design Doc for this requirement:
> --------------------------------
> * Purpose:
> 
> Export/archive the patches sources to assist in license compliance by the end
> user or legal departments. 
> 
> * Analysis:
> 1) What should be archive up?
> 
> All sources in $WORKDIR(after patching) and the build scripts which under
> '$WROKDIR/temp' dir should be archive up;
> 
> 2) When the archive process should be trigger?
> 
> The archive task should be triggered at the end of each build, that means
> all required patches have been applied to source code and all build scripts
> have been generated.
> 
> 3) Where the functionality should be put to?
> 
> The functionality should be in a separate class which the user can optionally
> include.
> 
> * Implement:
> Add a new class 'export_sources.bbclass' to provide task 'do_export_sources()',
> and get it called after 'do_install()' before 'do_populate_sysroot()'.
> 
> The Sources that need to be archived up:
> $S, $T/run.*
> 
> The Destinations/tarballs: 
> ${TMPDIR}/deploy/sources/${MULTIMACH_TARGET_SYS}/${PF}.tar.bz2

Updates:

The #1655 has been re-opened as it's not duplicate with this requirement completely. So only consider for this requirement, following updates would be appropriate:

1) Change the bbclass name to 'archive.bbclass' and task name as 'do_archive()';
2) Put the destinations to: "$TMPDIR/archives/${MULTIMACH_TARGET_SYS}/${PF}.tar.bz2"

Thanks
Wenzong
> 
> * Usage:
> 
> If any packages need to be archived up, just make sure inherit the
> 'export_sources' class their bb files.
Comment 7 Wenzong.Fan 2011-12-30 18:37:58 UTC
Hi Richard,

I'm trying to find out the best point of when start the 'do_archive' task, say where it should be put to:

add do_archive after xxx before xxx

We hope to archive the patched sources and build scripts. For getting the build scripts as many as possible, I put it before "do_populate_sysroot", but this will ignore many packages if they don't run the task 'do_populate_sysroot'.

Maybe the best point is after 'do_patch', but this will lost much build scripts that after 'do_patch'. 

Any suggestions about this?

Thanks
Wenzong
Comment 8 Xiaofeng Yan 2012-02-09 22:11:37 UTC
This bug has been marked with new bug 1977 because of existing similar function with other bb class (sourcepkg.bbclass in meta/class, bug 1655, src_distribute.bbclass) 

We need merge functions in these bb class into a new bb class(archiver.bbclass) for avoiding repetitive work in different bb class. Also sourcepkg.bbclass has some bugs and can't be used in the current version.

http://bugzilla.pokylinux.org/show_bug.cgi?id=1977
Comment 9 Saul Wold 2012-02-21 14:05:12 UTC
Second Archive improvement that will be address by the new source archiver see bug 1977

*** This bug has been marked as a duplicate of bug 1977 ***