| Summary: | Best practices for multi-machine builds | ||
|---|---|---|---|
| Product: | [Documentation] Development Manual | Reporter: | Martin Jansa <Martin.Jansa> |
| Component: | development | Assignee: | Scott Rifenbark <srifenbark> |
| Status: | RESOLVED FIXED | QA Contact: | Alexandru Georgescu <alexandru.c.georgescu> |
| Severity: | enhancement | ||
| Priority: | Medium+ | CC: | alexandru.c.georgescu, bluelightning, richard.purdie, sgw, sjolley.yp.pm, srifenbark |
| Version: | unspecified | ||
| Target Milestone: | 1.8 M1 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | 24 November 2014: RESOLVED | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | Done (doc changes complete) | |
This topic would fit nicely in the "Common Tasks" section of the YP Development Manual - http://www.yoctoproject.org/docs/1.6/dev-manual/dev-manual.html#extendpoky. I will work with someone to get a section created. Scott Enhancement - moving to 1.7. Changing to NEEDINFO and setting for 1.7 M3. I don't know who to work with on this one. Scott Martin, Can you provide more information for Scott on this? Thanks, > I will work with someone to get a section created.
When we discussed this on ELCE 2013 with Richard, he suggested to find someone in your office to implement this faster.
What I wanted to cover is in the description, but maybe someone else will have different opinion/priorities.
Since Martin can't help, reassign to Richard (who also doesn't have time) Multi-Machine Builds, Package Architectures and Performance =========================================================== (bug 5400) When faced with several different pieces of hardware (machines) all using the same software environment, its very tempting to max out the optimisation flags for each one, optimising each platform exactly to the processor it contains. This would be done with very specific tunes being selected for each machine. This is perfectly fine however if build speed or maintenance of the package feeds are a consideration, there are some other things that can be considered. Firstly, its advised the builds are performed in the same TMPDIR where possible. Switching between different MACHINE values in the same TMPDIR is well supported and regularly used by developers. This is in contrast to DISTRO settings since changing distro related settings like DISTRO_FEATURES is not supported without a clean TMPDIR. Using the same TMPDIR for multiple machines means the existing native and often cross recipes can be reused, speeding up the build. By default, there are three levels of package group enabled, 'all' architecture, 'tune' or 'package' architecture and 'machine' architecture. Any given recipe will usually select one of these package types for its output. A recipe just generating scripts can be "all" archiecture for example since there isn't anything binary in there. 'machine' specific recipes will ge generated for each machine that is built and are usualy hardware specific. The middle package type is 'tune' or 'package' archietcture is usually controlled by the 'tune' selected. Some tunes are more generic and can run on multiple targets, e.g. an armv5 set of packages could run on armv6 and armv7 processors in most cases and i486 binaries could work on i586 and higher processors. If the same tune is selected for several different machines, software will be built in the first build howeber in subsequent builds, rather than rebuild it, it will be reused. A new sysroot for the machine will be generated however the software itself will not recompile and there is only one package feed. This means that the build of a second machine sharing a tune with the first will be extremly fast. Sometimes there are cases where injecting another level of package type into this hierarchy could be useful. An example is the emgd graphics stack in the meta-intel layer. Here, there is a subset of the software which are compiled against something different from the rest of the generic packages. The key code is in classes/emgd-gl.bbclass and for a specific set of packages, it redefines PACKAGE_ARCH. PACKAGE_EXTRA_ARCHS is then appended with this extra tune name in meta-intel-emgd.inc so when searching for packages, it becomes a 4 level search and packages in this new level will be preferred compared to the standard tune. This means that most software is reused with the common tune but in the cases were needed something specific can be done. In some cases, sstate may not be reused between machines when you'd expect it to. There are some tools which can help debug these kinds of problems, such as sstate-diff-machines.sh and the "-S printdiff" option to bitbake. They are usually due to references to variables like MACHINE in code which is supposed to only be 'tune' specific and these tools can help track down where this is occurring. Patches to fix any issues identified would be most welcome as they occasionaly do occur. Richard, I have some questions and some observations for Comment 7. I talked with Paul a bit to try and get some background information and we could not immediately finish. So, I am putting some questions here and some notes to myself for what I think is going on. Could you look at these comments and let me know where my gaps or mis-understandings are? ------------------------------ When faced with several different pieces of hardware (machines) all using the same software environment, its very tempting to max out the optimisation flags for each one, optimising each platform exactly to the processor it contains. This would be done with very specific tunes being selected for each machine. This is perfectly fine however if build speed or maintenance of the package feeds are a consideration, there are some other things that can be considered. - what specifically does the user do to “max out the optimization for each one”? - How do you select specific tunes for a given machine? Firstly, its advised the builds are performed in the same TMPDIR where possible. Switching between different MACHINE values in the same TMPDIR is well supported and regularly used by developers. This is in contrast to DISTRO settings since changing distro related settings like DISTRO_FEATURES is not supported without a clean TMPDIR. Using the same TMPDIR for multiple machines means the existing native and often cross recipes can be reused, speeding up the build. - How does the build system handle it if the user does not purposely change the TMPDIR but has changed a distro-related setting that apparently is is not supported? - Is the DISTRO_FEATURES setting the only distro related setting not supported without a clean TMPDIR? What other common distro-related settings could the user set that would force the build system into a state where it could not use the same TMPDIR? - (Note to myself) To use a different TMPDIR, the user sets TMPDIR to a location by defining the TMPDIR variable in the local.conf file. See http://www.yoctoproject.org/docs/1.7/mega-manual/mega-manual.html#var-TMPDIR for an explanation of the TMPDIR variable. By default, there are three levels of package group enabled, 'all' architecture, 'tune' or 'package' architecture and 'machine' architecture. Any given recipe will usually select one of these package types for its output. A recipe just generating scripts can be "all" archiecture for example since there isn't anything binary in there. 'machine' specific recipes will ge generated for each machine that is built and are usualy hardware specific. The middle package type is 'tune' or 'package' archietcture is usually controlled by the 'tune' selected. - “package group” would be clearer if stated as “package architectures”. - I don't understand the user role associated with “Any given recipe will usually select one of these package types for its output.” Does that mean the user has to select it in the recipe by specifically setting it? And, if so, then the user needs to understand if the recipe is going to just be generating scripts and would select “all” and machine-specific recipes when building for specific hardware and would select “machine”? From the preceding paragraph I don't know when the user would select “tune” or “package” as the architecture. - If the user does not specifically select one of the three package architectures then how does the build system do it? Some tunes are more generic and can run on multiple targets, e.g. an armv5 set of packages could run on armv6 and armv7 processors in most cases and i486 binaries could work on i586 and higher processors. If the same tune is selected for several different machines, software will be built in the first build however in subsequent builds, rather than rebuild it, it will be reused. A new sysroot for the machine will be generated however the software itself will not recompile and there is only one package feed. This means that the build of a second machine sharing a tune with the first will be extremly fast. - Is there any guidance we can give the user regarding tunes that are generic in nature and can be used across different machines other than what is stated here? Sometimes there are cases where injecting another level of package type into this hierarchy could be useful. An example is the emgd graphics stack in the meta-intel layer. Here, there is a subset of the software which are compiled against something different from the rest of the generic packages. The key code is in classes/emgd-gl.bbclass and for a specific set of packages, it redefines PACKAGE_ARCH. PACKAGE_EXTRA_ARCHS is then appended with this extra tune name in meta-intel-emgd.inc so when searching for packages, it becomes a 4 level search and packages in this new level will be preferred compared to the standard tune. This means that most software is reused with the common tune but in the cases were needed something specific can be done. - (Note to self) So this is apparently a way to leverage off previously built packages when a smaller subset of differently tuned packages is required. See http://www.yoctoproject.org/docs/1.7/mega-manual/mega-manual.html#var-PACKAGE_EXTRA_ARCHS for a description of the variable. This leads me to believe that during a build, all packages are kept in some common area and then depending on the target, they get installed as needed. In some cases, sstate might not be reused between machines when you'd expect it to. There are some tools which can help debug these kinds of problems, such as sstate-diff-machines.sh and the "-S printdiff" option to bitbake. They are usually due to references to variables like MACHINE in code which is supposed to only be 'tune' specific and these tools can help track down where this is occurring. Patches to fix any issues identified would be most welcome as they occasionaly do occur. - So the generic examples of this case are reference to variables like MACHINE in code that is supposed to be tune-specific. Are these references in code in the recipes? Or, in the software being built? - (Note to self) See http://www.yoctoproject.org/docs/1.7/mega-manual/mega-manual.html#shared-state-cache for some explanation of how shared-state works. - (Note to self) Regarding sstate-diff-machines.sh... You can find this at http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/scripts/sstate-diff-machines.sh in the poky repository. The comments in the script have some instruction on how to use it. How does a user use the tool? - (Note to self) See http://www.yoctoproject.org/docs/1.7/bitbake-user-manual/bitbake-user-manual.html#checksums for some information on the -S printdiff option as described in the BitBake User Manual. (In reply to comment #8) > When faced with several different pieces of hardware (machines) all > using the same software environment, its very tempting to max out the > optimisation flags for each one, optimising each platform exactly to the > processor it contains. This would be done with very specific tunes being > selected for each machine. This is perfectly fine however if build speed > or maintenance of the package feeds are a consideration, there are some > other things that can be considered. > > - what specifically does the user do to “max out the optimization for each > one”? Selecting tunes which are extremely specific to the CPU core used in the system. This may enable some micro optimizations in gcc but usually the more generic tunes don't have much of a performance difference and can be used more widely. > - How do you select specific tunes for a given machine? Including specific tune files and/or selecting a specific DEFAULTTUNE. > Firstly, its advised the builds are performed in the same TMPDIR where > possible. Switching between different MACHINE values in the same TMPDIR > is well supported and regularly used by developers. This is in contrast > to DISTRO settings since changing distro related settings like > DISTRO_FEATURES is not supported without a clean TMPDIR. Using the same > TMPDIR for multiple machines means the existing native and often cross > recipes can be reused, speeding up the build. > > - How does the build system handle it if the user does not purposely change > the TMPDIR but has changed a distro-related setting that apparently is > is not supported? It handles it badly. It may work, it may not. > - Is the DISTRO_FEATURES setting the only distro related setting not > supported without a clean TMPDIR? What other common distro-related > settings could the user set that would force the build system into a > state where it could not use the same TMPDIR? This guideline also applies to variables which change fundamental configuration like filesystem layout. You may get lucky and it may rebuild correctly (it works better today than it ever has done) but to be safe you should clean or change TMPDIR. > - (Note to myself) To use a different TMPDIR, the user sets TMPDIR to a > location by defining the TMPDIR variable in the local.conf file. See > > http://www.yoctoproject.org/docs/1.7/mega-manual/mega-manual.html#var-TMPDIR > for an explanation of the TMPDIR variable. > > By default, there are three levels of package group enabled, 'all' > architecture, 'tune' or 'package' architecture and 'machine' > architecture. Any given recipe will usually select one of these package > types for its output. A recipe just generating scripts can be "all" > archiecture for example since there isn't anything binary in there. > 'machine' specific recipes will ge generated for each machine that is > built and are usualy hardware specific. The middle package type is > 'tune' or 'package' archietcture is usually controlled by the 'tune' > selected. > > - “package group” would be clearer if stated as “package architectures”. yes, it would. > - I don't understand the user role associated with “Any given recipe will > usually select one of these package types for its output.” Does that > mean the user has to select it in the recipe by specifically setting > it? By default PACKAGE_ARCH == TUNE_ARCH which is the architecture of the currently enabled tune. If the user wants machine speific, they'd set: PACKAGE_ARCH = "${MACHINE_ARCH}" and if they want "all" they'd inherit the allarch class. We use a class in this case since there are other variables its useful to set in this case. > And, if so, then the user needs to understand if the recipe is > going to just be generating scripts and would select “all” and > machine-specific recipes when building for specific hardware and would > select “machine”? From the preceding paragraph I don't know when the > user would select “tune” or “package” as the architecture. PACKAGE_ARCH is the variable that gets set and by default its TUNE_ARCH, it call also be MACHINE_ARCH or "all", the latter of which you'd use "inherit allarch" to handle correctly in most cases. > - If the user does not specifically select one of the three package > architectures then how does the build system do it? It defaults to TUNE_ARCH. > Some tunes are more generic and can run on multiple targets, e.g. an > armv5 set of packages could run on armv6 and armv7 processors in most > cases and i486 binaries could work on i586 and higher processors. If the > same tune is selected for several different machines, software will be > built in the first build however in subsequent builds, rather than > rebuild it, it will be reused. A new sysroot for the machine will be > generated however the software itself will not recompile and there is > only one package feed. This means that the build of a second machine > sharing a tune with the first will be extremly fast. > > - Is there any guidance we can give the user regarding tunes that are > generic in nature and can be used across different machines other than > what is stated here? Its very architecture specific so no, general terms should be ok here. > > Sometimes there are cases where injecting another level of package type > into this hierarchy could be useful. An example is the emgd graphics > stack in the meta-intel layer. Here, there is a subset of the software > which are compiled against something different from the rest of the > generic packages. The key code is in classes/emgd-gl.bbclass and for a > specific set of packages, it redefines PACKAGE_ARCH. PACKAGE_EXTRA_ARCHS > is then appended with this extra tune name in meta-intel-emgd.inc so > when searching for packages, it becomes a 4 level search and packages in > this new level will be preferred compared to the standard tune. This > means that most software is reused with the common tune but in the cases > were needed something specific can be done. > > - (Note to self) So this is apparently a way to leverage off previously > built packages when a smaller subset of differently tuned packages is > required. See > http://www.yoctoproject.org/docs/1.7/mega-manual/mega-manual.html#var- > PACKAGE_EXTRA_ARCHS for a description of the variable. > This leads me to believe that during a build, all packages are kept > in some common area and then depending on the target, they get installed > as needed. Near enough, yes. > In some cases, sstate might not be reused between machines when you'd > expect it to. There are some tools which can help debug these kinds of > problems, such as sstate-diff-machines.sh and the "-S printdiff" option > to bitbake. They are usually due to references to variables like MACHINE > in code which is supposed to only be 'tune' specific and these tools can > help track down where this is occurring. Patches to fix any issues > identified would be most welcome as they occasionaly do occur. > > - So the generic examples of this case are reference to variables like > MACHINE in code that is supposed to be tune-specific. Are these references > in code in the recipes? Or, in the software being built? In the recipes. I think that in all your comments you meant TUNE_PKGARCH not TUNE_ARCH. > - So the generic examples of this case are reference to variables like
> MACHINE in code that is supposed to be tune-specific. Are these references
> in code in the recipes? Or, in the software being built?
Other very common but not so easy to see reason for using
PACKAGE_ARCH = "${MACHINE_ARCH}"
is that one of the build or runtime dependencies is already MACHINE_ARCH, which makes your recipe to be also MACHINE_ARCH.
Hi, Here is a link to this new section. I need it reviewed and I suspect we will need to iterate some on the content. Thanks, Scott http://www.yoctoproject.org/docs/latest/dev-manual/dev-manual.html#building-images-for-more-than-one-machine Hi, Richard Purdie provided a review of this new section and I implemented his suggestions. You can see the updated section at http://www.yoctoproject.org/docs/1.8/dev-manual/dev-manual.html#building-images-for-more-than-one-machine. I am leaving the state of the bug in IN PROGRESS REVIEW should anyone want to look at the changes or give feedback. Scott Are you going to implement my comments about TUNE_PKGARCH and MACHINE_ARCH, TUNE_ARCH is still incorrect on the page. From the e-mail, just in case someone has better idea how to explain it.
> > also please mention that not only MACHINE variable, but also (runtime)
> > dependency
> > on something which is already MACHINE_ARCH forces the recipe to be
> > also MACHINE_ARCH
> >
>
> Martin - I am lost on this "please mention" item. Can you give me some
> more specifics here. I am having trouble figuring out just where this
> applies in that "*Enable the Appropriate Package Architecture:" *area of
> the manual. The third paragraph talks some about runtime dependencies. I
> don't understand the coding parts well enough to translate it in my head.
> Sorry to have to bother you with this.
I think best place is in the section you've added. e.g.
These types of situations are usually due to references to variables
such as MACHINE in code that is supposed to only be "tune" specific.
->
These types of situations are usually due to references to machine
specific variables such as MACHINE, SERIAL_CONSOLE, XSERVER, MACHINE_FEATURES, ..
in code that is supposed to only be "tune" specific or when the recipe
depends (DEPENDS, RDEPENDS, RRECOMMENDS, RSUGGESTS variables) on some
other recipe which already has PACKAGE_ARCH = "${MACHINE_ARCH}".
Hi, After some email exchanges with Martin and some consultation with Paul Eggleton, I put some more changes into this section. I had omitted some stuff of Martin's. Please check it over and let me know if there are any further changes required. http://www.yoctoproject.org/docs/1.8/dev-manual/dev-manual.html#building-images-for-more-than-one-machine Thanks, Scott Looks good, thanks. Hi, Looks like this is good now. I have set to RESOLVED and put the doc flag to "done." Scott |
Show few examples how to build images for different MACHINEs. Explain trade-off between using specific DEFAULTTUNEs for each MACHINE or something more common and reusing TUNE_PKGARCH sstate with multiple MACHINE with same architecture. Explain why it's needed to use oe-core/scripts/sstate-diff-machines.sh to be sure that some TUNE_PKGARCH (or even allarch/native) recipes aren't tainted by MACHINE variable. Basically it should show that using tmp-${TCLIBC}-${MACHINE} is even worse idea, because of duplication of native sysroots, feed etc. Richard: This is the problem we talked about on ELCE.