12592 2018-03-08 06:20:20 +0000 go runtime: abi mismatch with libstd.so 2018-03-20 15:10:06 +0000 1 1 7 Build System, Metadata & Runtime OE-Core devtools / tool chain 2.4.1 All Multiple RESOLVED FIXED Medium+ major 2.4.2 1 liam matt meta.mr.watcher meta.watcher randy.macleod stephano --- --- Yes (doc changes required) oldest_to_newest 79751 0 liam 2018-03-08 06:20:20 +0000 Summary Go applications are appearing to be linked to the instance of, not version, of the go-runtime package used during their compilation. If a go package is updated on a target but the runtime is not, then the following error appears: `abi mismatch detected between the executable and libstd.so fatal error: abi mismatch runtime: panic before malloc heap initialized` There are two probable causes: 1) We are not version tracking all of the standard libraries and generating a new libstd.so without reflecting it in the yocto package version 2) Go is incorrectly calculating the ABI Hash Technical Background In Rocko there has been a big change to how go applications are compiled. For most systems the default buildmode is now shared(-buildmode=shared). In order to implement this, first the runtime package generates `libstd.so` which contains all the standard packages in go and then the go application is linked against it and other libraries. Go check compatibility between the shared libraries and application by examining the ABI hash. These mechanism are detailed here. ABI HashChecking Machanism: https://go-review.googlesource.com/c/go/+/8773 ABI Hash GEneration: https://go-review.googlesource.com/c/go/+/40401 I experienced this problem compiling for an ARMv7 and there is a stack overflow post concerning an x86. 79756 1 4227 liam 2018-03-08 09:24:30 +0000 Created attachment 4227 Experimental Patch to turn ABI mismatch into a warning, not an error This patch is meant to help debug, not for production 79757 2 liam 2018-03-08 09:47:51 +0000 I have turned the error into a warning by patching the go-runtime source with the attached patch. With the patch i can replace a go application and it will run (just with an the warning). Also the problem appears to be scoped to either time or type of build: 1. I made a new image last night running `core-image-minimal` 2. I loaded the image onto my target this morning 3. I bitbaked my application. The result was the application's RPM and a go runtime RPM. 4. I installed my application rpm into the target, it ran but the patched warning appeared 5. I installed the new runtime rpm. The application ran now ran without warning, but other existing go application now gave the warning. 6. I re-bitbaked my applciation 7. I installed just the applciation rpm onto the target and there was no warning This makes me think that 1. Either something different is happening when i compile an image vs just a package, or; 2. In the 12 hours between last night and this afternoon there was a change that was collected by the recipe that caused the ABI hash to change; and, that there was no ABI breaking changes between my two bitbake package builds (hour apart). 79762 3 liam 2018-03-08 11:43:36 +0000 As a production work around I am defaulting to: GO_LINKSHARED = "" To stop linking against `libstd.so` 79783 4 matt 2018-03-11 10:57:56 +0000 The ABI hash of libstd.so is the sum of the ABI hashes of the Go packages contained in the library. Turns out there was an issue with the hash computation, specifically for cgo-generated objects - full absolute path names were getting encoded in the export data for the package, even if the package resides in GOROOT. The Go 'plugin' package uses cgo, so the hash of that package would change if you ran builds in different directories. The fix for this went into Go 1.9.2. Upstream issue: https://github.com/golang/go/issues/21825 Fix: https://go-review.googlesource.com/c/go/+/70975 My testing with this patch cherry-picked on top of the existing Go 1.9 patches we have in rocko shows that it does indeed fix the problem. Alternatively, we could back port the go 1.9.4 recipe upgrade from master to rocko, since that would also include the fix. 79885 5 matt 2018-03-20 15:10:06 +0000 The update to go 1.9.4 includes the upstream patch that fixes this issue. 4227 2018-03-08 09:24:30 +0000 2018-03-08 09:24:30 +0000 Experimental Patch to turn ABI mismatch into a warning, not an error 0001-allow-replacing-packages.patch text/plain 837 liam RnJvbSBlMTdkNTBhMGEyM2ZhMTVhZmZjYzc2MmY2NzI3NDRjYTdjZjBjNGZlIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBsaWFtIDxsaWFta2VsbHlAY29tbWRldmljZXMuY29tPgpEYXRl OiBXZWQsIDcgTWFyIDIwMTggMjE6MDA6MDggLTA1MDAKU3ViamVjdDogW1BBVENIXSBhbGxvdyBy ZXBsYWNpbmcgcGFja2FnZXMKCi0tLQogc3JjL3J1bnRpbWUvc3ltdGFiLmdvIHwgMiArLQogMSBm aWxlIGNoYW5nZWQsIDEgaW5zZXJ0aW9uKCspLCAxIGRlbGV0aW9uKC0pCgpkaWZmIC0tZ2l0IGEv c3JjL3J1bnRpbWUvc3ltdGFiLmdvIGIvc3JjL3J1bnRpbWUvc3ltdGFiLmdvCmluZGV4IDdkN2Mz NjMuLjMzYTBhY2UgMTAwNjQ0Ci0tLSBhL3NyYy9ydW50aW1lL3N5bXRhYi5nbworKysgYi9zcmMv cnVudGltZS9zeW10YWIuZ28KQEAgLTU2Niw3ICs1NjYsNyBAQCBmdW5jIG1vZHVsZWRhdGF2ZXJp ZnkxKGRhdGFwICptb2R1bGVkYXRhKSB7CiAJZm9yIF8sIG1vZHVsZWhhc2ggOj0gcmFuZ2UgZGF0 YXAubW9kdWxlaGFzaGVzIHsKIAkJaWYgbW9kdWxlaGFzaC5saW5rdGltZWhhc2ggIT0gKm1vZHVs ZWhhc2gucnVudGltZWhhc2ggewogCQkJcHJpbnRsbigiYWJpIG1pc21hdGNoIGRldGVjdGVkIGJl dHdlZW4iLCBkYXRhcC5tb2R1bGVuYW1lLCAiYW5kIiwgbW9kdWxlaGFzaC5tb2R1bGVuYW1lKQot CQkJdGhyb3coImFiaSBtaXNtYXRjaCIpCisJCQlwcmludCgiXHRMaW5rIEhhc2g6ICIsIG1vZHVs ZWhhc2gubGlua3RpbWVoYXNoLCAiXG5cdFJ1bnRpbWUgSGFzaDogIiwgKm1vZHVsZWhhc2gucnVu dGltZWhhc2gsICJcbiIpCiAJCX0KIAl9CiB9Ci0tIAoyLjcuNAoK