Bug 12592

Summary: go runtime: abi mismatch with libstd.so
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Liam Kelly <liam>
Component: devtools / tool chainAssignee: Matt Madison <matt>
Status: RESOLVED FIXED QA Contact:
Severity: major    
Priority: Medium+ CC: meta.mr.watcher, meta.watcher, randy.macleod, stephano
Version: 2.4.1   
Target Milestone: 2.4.2   
Hardware: All   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Yes (doc changes required)
Attachments:
Description Flags
Experimental Patch to turn ABI mismatch into a warning, not an error none

Description Liam Kelly 2018-03-08 06:20:20 UTC 
Summary
Go applications are appearing to be linked to the instance of, not version, of the go-runtime package used during their compilation. If a go package is updated on a target but the runtime is not, then the following error appears:

`abi mismatch detected between the executable and libstd.so
fatal error: abi mismatch
runtime: panic before malloc heap initialized` 

There are two probable causes:
1) We are not version tracking all of the standard libraries and generating a new libstd.so without reflecting it in the yocto package version 
2) Go is incorrectly calculating the ABI Hash

Technical Background
In Rocko there has been a big change to how go applications are compiled. For most systems the default buildmode is now shared(-buildmode=shared). In order to implement this, first the runtime package generates `libstd.so` which contains all the standard packages in go and then the go application is linked against it and other libraries. 

Go check compatibility between the shared libraries and application by examining the ABI hash. These mechanism are detailed here.
ABI HashChecking Machanism:
https://go-review.googlesource.com/c/go/+/8773
ABI Hash GEneration:
https://go-review.googlesource.com/c/go/+/40401

I experienced this problem compiling for an ARMv7 and there is a stack overflow post concerning an x86.
Comment 1 Liam Kelly 2018-03-08 09:24:30 UTC 
Created attachment 4227 [details]
Experimental Patch to turn ABI mismatch into a warning, not an error

This patch is meant to help debug, not for production
Comment 2 Liam Kelly 2018-03-08 09:47:51 UTC 
I have turned the error into a warning by patching the go-runtime source with the attached patch.

With the patch i can replace a go application and it will run (just with an the warning).

Also the problem appears to be scoped to either time or type of build:
1. I made a new image last night running `core-image-minimal`
2. I loaded the image onto my target this morning
3. I bitbaked my application. The result was the application's RPM and a go runtime RPM.
4. I installed my application rpm into the target, it ran but the patched warning appeared
5. I installed the new runtime rpm. The application ran now ran without warning, but other existing go application now gave the warning.
6. I re-bitbaked my applciation
7. I installed just the applciation rpm onto the target and there was no warning

This makes me think that
1. Either something different is happening when i compile an image vs just a package, or;
2. In the 12 hours between last night and this afternoon there was a change that was collected by the recipe that caused the ABI hash to change; and, that there was no ABI breaking changes between my two bitbake package builds (hour apart).
Comment 3 Liam Kelly 2018-03-08 11:43:36 UTC 
As a production work around I am defaulting to:

GO_LINKSHARED = ""

To stop linking against `libstd.so`
Comment 4 Matt Madison 2018-03-11 10:57:56 UTC 
The ABI hash of libstd.so is the sum of the ABI hashes of the Go packages contained in the library.  Turns out there was an issue with the hash computation, specifically for cgo-generated objects - full absolute path names were getting encoded in the export data for the package, even if the package resides in GOROOT.  The Go 'plugin' package uses cgo, so the hash of that package would change if you ran builds in different directories.

The fix for this went into Go 1.9.2.

Upstream issue: https://github.com/golang/go/issues/21825
Fix: https://go-review.googlesource.com/c/go/+/70975

My testing with this patch cherry-picked on top of the existing Go 1.9 patches we have in rocko shows that it does indeed fix the problem.  Alternatively, we could back port the go 1.9.4 recipe upgrade from master to rocko, since that would also include the fix.
Comment 5 Matt Madison 2018-03-20 15:10:06 UTC 
The update to go 1.9.4 includes the upstream patch that fixes this issue.