Bug 9510 - gcc-cross-initial parallel make problem / build break (found on aarch64)
Summary: gcc-cross-initial parallel make problem / build break (found on aarch64)
Status: RESOLVED WORKSFORME
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: devtools / tool chain (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium normal
Target Milestone: 2.2 M3
Assignee: Tristan Van Berkom
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2016-04-22 18:14 UTC by Tristan Van Berkom
Modified: 2016-10-17 17:19 UTC (History)
7 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
Split all-gcc and configure-target-libgcc (941 bytes, patch)
2016-04-22 18:14 UTC, Tristan Van Berkom
no flags Details | Diff
gcc-cross-initial build log from poky master & gcc 6.1 (46.30 KB, application/octet-stream)
2016-07-05 06:37 UTC, Tristan Van Berkom
no flags Details
m400 firmware revisions (2.12 KB, text/plain)
2016-07-05 07:32 UTC, Tristan Van Berkom
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Tristan Van Berkom 2016-04-22 18:14:02 UTC
Created attachment 3150 [details]
Split all-gcc and configure-target-libgcc

When building gcc-cross-initial on aarch64 (Ubuntu 16.04), the configure-target-libgcc (make target) fails when trying to execute the in tree compiler at ${gcc_builddir}/gcc/xgcc.

Serializing the all-gcc and configure-target-libgcc targets with separate invocations of oe_runmake() fixes the build.

I'm not sure how this is aarch64 specific or why the build succeeds when building on x86_64.
Comment 1 Ross Burton 2016-07-02 19:59:09 UTC
I'd say that this is clearly a bug in the upstream makefiles.  Can you replicate with current oe-core master, or can you only replicate in a previous stable release?
Comment 2 Ross Burton 2016-07-04 16:14:43 UTC
CCing Koen and Nicolas, as this is clearly building-on-aarch64 specific in some way.

Can someone verify that current master (or even 2.1) works fine when building the OE toolchain on aarch64?  (or send me a aarch64 board).
Comment 3 Fathi Boudra 2016-07-04 17:52:22 UTC
@Ross, you can request an access on https://register.linaro.cloud/
@Tristan, which board is it?

I can also try to reproduce myself.
Comment 4 Tristan Van Berkom 2016-07-05 04:18:05 UTC
(In reply to comment #3)
[...]
> @Tristan, which board is it?
> 
> I can also try to reproduce myself.

Hi, the host board is the m400 hp cartridge (moonshot server) which has 8 AppliedMicro X-Gene ARMv8 64-bit cores at 2.4Ghz.

I suspect that having 8 cores may be more relevant here than what specific chipset it is... I'll see if i can find time to attempt to build master on this board...
Comment 5 Tristan Van Berkom 2016-07-05 06:37:46 UTC
Created attachment 3267 [details]
gcc-cross-initial build log from poky master & gcc 6.1

This is not fixed with poky master / gcc 6.1 - attaching failing build log using poky master.

Note that we reproduce this error via the freedesktop-sdk-base repo:
    https://cgit.freedesktop.org/xdg-app/freedesktop-sdk-base/

Which basically adds its own bitbake layer and checks out the poky
repository as a git submodule (at a ref close to the tip of jethro stable).

Today I did the following steps:
  o Setup the freedesktop-sdk-base repo to use poky master
  o Removed some bbappend files from freedesktop-sdk-base which
    no longer apply to poky master
  o Removed a reference to python-pip which no longer exists in poky master
  o Removed our .bbappend file for gcc-cross-initial which basically overrides
    the do_compile() so as to oe_runmake() in 2 separate steps
  o Ran the build

Note that the build with master passes file if I do not remove the
gcc-cross-initial .bbappend file which fixes this locally (i.e. the
same patch attached here _also_ fixes the problem in master).

Without our local workaround, we get the same error, which can be observed in the attached build log. Problem is that the in tree 'xgcc' binary is not present while executing configure-target-libgcc, causing the libgcc configure script to bail out claiming that the compiler cannot create executables.
Comment 6 Fathi Boudra 2016-07-05 06:50:49 UTC
Do you mind to share the firmware version used on the m400?
Comment 7 Tristan Van Berkom 2016-07-05 07:32:30 UTC
Created attachment 3268 [details]
m400 firmware revisions

Regarding firmware version on m400, I think the answer you are looking for is:

    Cartridge Mellanox ConnectX3Pro Firmware: 2.32.5340

We are running Ubuntu 16.04 on the m400 cartridges.

Attached is the output of:

   hpiLO-> SHOW FIRMWARE REVISIONS LIST

Run from the iLO interface on the chassis manager.
Comment 8 Ross Burton 2016-07-06 19:55:47 UTC
Data point: current poky master builds fine for me out of the box on Linaro's aarch64 cloud service.
Comment 9 Tristan Van Berkom 2016-07-07 06:30:46 UTC
(In reply to comment #8)
> Data point: current poky master builds fine for me out of the box on
> Linaro's aarch64 cloud service.

Curiously, did you (or can you) try that with 8 cores ?
Comment 10 Ross Burton 2016-07-07 08:13:10 UTC
The instance has four cores, so it's doing a parallel build.

Can you replicate the failure every time, or does it sometimes work?
Comment 11 Tristan Van Berkom 2016-07-07 08:47:52 UTC
(In reply to comment #10)
> The instance has four cores, so it's doing a parallel build.
> 

It's true, I'm only grasping at straws thinking that 8 cores might make a difference over 4. Especially since subsequent invocations of the build scripts also fail (if it were indeed a race, after building a couple of times you would consistently start to pass the build).

> Can you replicate the failure every time, or does it sometimes work?

I replicate the failure every time, I have never seen it work without the workaround.
Comment 12 Fathi Boudra 2016-07-07 12:45:56 UTC
I tried to reproduce on APM X-Gene X-C1 (Mustang). It has 8 cores.
In my setup, I built under docker on Ubuntu 16.04 Xenial (kernel 4.4.0-24-generic).

Build Configuration:
BB_VERSION        = "1.31.0"
BUILD_SYS         = "aarch64-linux"
NATIVELSBSTRING   = "Debian-8.2"
TARGET_SYS        = "aarch64-poky-linux"
MACHINE           = "qemuarm64"
DISTRO            = "poky"
DISTRO_VERSION    = "2.1+snapshot-20160707"
TUNE_FEATURES     = "aarch64"
TARGET_FPU        = ""
meta              
meta-poky         
meta-yocto-bsp    = "master:5c11e365e19357f721c49d076971567e7b64b61b"

I've built successfully core-image-minimal and haven't been able to reproduce the issue.
Comment 13 Ross Burton 2016-07-12 17:00:47 UTC
Myself and Fathi can't replicate this using pristine oe-core master on two different aarch64 hosts.  Tristan, can you try to replicate this on your hardware with just poky and none of the freedesktop-sdk stuff?
Comment 14 Saul Wold 2016-09-01 14:50:17 UTC
From Tristan:

Hi Saul, Ross,

I understand, thanks for being so diligent and persistent in following
up on this issue and sorry I have not taken the time to login to the
build machine in question and trying to reproduce this without the
interference of the additional freedesktop flatpak sdk layer.

I'm sure that whether the isolated oe build passes or fails it would
uncover some interesting detail (i.e. why would it fail just because of
the added layer ? finding this out might uncover a bug in our layer
recipes or a deeper problem in the layering system).

In any case please feel free to close the bug, I'm in a bit of a grind
and I'm not sure when I'll find the time to look into this, certainly
not this week.

Best Regards,
    -Tristan