Bug 10605

Summary:	[Build-Perf] split perf test scripts from poky into another git repository
Product:	[Build System, Metadata & Runtime] OE-Core	Reporter:	Jianxun Zhang <jianxun.zhang>
Component:	Scripts and Tools	Assignee:	Markus Lehtonen <markus.lehtonen>
Status:	RESOLVED WONTFIX	QA Contact:
Severity:	normal
Priority:	Medium	CC:	markus.lehtonen, richard.purdie, ross.burton, sgw
Version:	2.2
Target Milestone:	Future
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Jianxun Zhang 2016-11-04 18:50:12 UTC

This is a candidate task for build performance area. Note we could close this issue as invalid if team doesn't like this idea.

Problem Statement:
Now the perf test scripts are a part of poky git. This layout potentially invalidate a bisect procedure when the section of git history contains changes to the perf test scripts themselves. We need to use same tool revisions to do benchmark against different commit tips in the source code project so that the data acquired are comparable. Now I have to manipulate the tree or copy the perf test out of tree to achieve this goal. These add another change for human errors. And actually such changes are not noticeable in a bisecting test.

Proposed Solution:
I suggest we put perf test scripts themselves into another git repository to address this concern.

Comment 1 Markus Lehtonen 2016-11-07 10:24:08 UTC

Assigning this to myself.

You have a valid point and it make sense to split it out. Any suggestions where to move it? I wouldn't necessarily create a separate layer just for build perf tests.

Comment 2 Richard Purdie 2016-11-10 15:51:46 UTC

Whilst I see the usecase for this, I also see various problems. My personal instinct is not to do this, we have other more important things to deal with.

Comment 3 Markus Lehtonen 2016-11-10 16:34:04 UTC

After sleeping over it I also start to think it might have more problems than upsides. First of all, it would be required to make it self-contained, we couldn't rely on any utilities/library functions in oe-core. Nevertheless, I still see the bisect usecase.

Comment 4 Jianxun Zhang 2016-11-10 17:48:35 UTC

I don't want to argue for urgency point. Making perf self-contained is what I believe (Thanks for Markus to make it clear) too.

I have said team can close this if you guys don't like this idea.

Comment 5 Jianxun Zhang 2016-11-11 16:23:58 UTC

Refer to our previous comment.