14303 2021-03-16 15:58:11 +0000 Result of build is not stored in testresult.json with resulttool 2026-10-29 15:21:08 +0000 1 1 10 QA/Testing Tooling tooling 5.0 x86 Multiple NEW Medium+ enhancement 6.2 M1 1 richard.purdie unassigned jay.shen.teoh mhalstead paul randy.macleod tgamblin --- --- No (bug/feature does not impact docs) oldest_to_newest 89630 0 richard.purdie 2021-03-16 15:58:11 +0000 Whilst the outcome of tests like testimage, testsdk or oe-selftest are stored in testresult.json, there is no storage of "build" results like core-image-sato being built for qemuarm. We'd like to use testresults.json as the way to document testing for Yocto Project Compatible status of a layer. To do that we need to document the build tests completed as well as the runtime and selftests tests. 91258 1 jay.shen.teoh 2021-08-25 05:27:22 +0000 Hi Richard, I would like to propose my approach for this bug and seek for your opinions towards it. I created a patch which added a function to enable the system to write the build outputs into a testresults.json file and I would like to add those changes into the buildhistory.bbclass. Reason of applying those changes to buildhistory is because I had go through the buildhistory class and I found out that this class actually quite satisfied the goal of this bug, it is just that buildhistory doesn't write the build outputs into json files. So I created this patch to document the build outputs into a testresult.json file. This json file will be stored under a folder called 'buildresult'. 91259 2 4824 jay.shen.teoh 2021-08-25 05:29:32 +0000 Created attachment 4824 Sample output This is how the final testresults.json file will be generated 91346 3 jay.shen.teoh 2021-09-07 00:16:39 +0000 Any comment or feedback? 91815 4 randy.macleod 2021-10-21 19:40:43 +0000 Richard, Can you look at the attachment and reply to Teoh? 91816 5 richard.purdie 2021-10-21 20:00:54 +0000 Sorry, I didn't see the reply until now, Randy pinged me on irc since I get a lot (too much) email from bugzilla. I think the information we need in the testresults file needs to be tweaked a bit from this. For example, I think the image name should be a field in the json output in it's own right, not as part of a string of text. If we need that string, we should be able to generate it from the individual pieces of data. I also think the branch and revision of the layer should be separate and specified for each later rather than the ' " ' used in your example. We also need to think about which variables the autobuilder varies in it's tests such as package format, init system and so on as these need to be recorded too. The exact list of variables to be recoded probably needs to be configurable. So yes, buildhistory does recode some information but I think what we should put into testresult.json should be different. 91970 6 randy.macleod 2021-11-11 16:10:25 +0000 Teoh, can you work on this for M1 please. 91985 7 jay.shen.teoh 2021-11-12 02:50:08 +0000 (In reply to comment #6) > Teoh, can you work on this for M1 please. Sure Randy. I will start working on it. 93581 8 randy.macleod 2022-06-23 14:53:49 +0000 Teoh, if you can get to this, great but it's now unassigned. 96440 9 tgamblin 2023-09-14 15:45:49 +0000 Mistakenly moved this to M4. Since it's an enhancement, it should be 5.0 M1. 97671 10 randy.macleod 2023-12-21 23:28:28 +0000 Bulk move of 52 5.0-M1 bugs to M2. 4824 2021-08-25 05:29:32 +0000 2021-08-25 05:29:32 +0000 Sample output testresults.json application/json 3033 jay.shen.teoh ewogICAgImJ1aWxkX2NvcmUtaW1hZ2Utc2F0by1zZGtfMjAyMTA4MjQxNTUyMDkiOiB7CiAgICAg ICAgImNvbmZpZ3VyYXRpb24iOiB7CiAgICAgICAgICAgICJJVEVNIjogWwogICAgICAgICAgICAg ICAgIkJCX1ZFUlNJT04gICAgICAgICAgID0gXCIxLjQ2LjBcIiIsCiAgICAgICAgICAgICAgICAi QlVJTERfU1lTICAgICAgICAgICAgPSBcIng4Nl82NC1saW51eFwiIiwKICAgICAgICAgICAgICAg ICJOQVRJVkVMU0JTVFJJTkcgICAgICA9IFwidW5pdmVyc2FsXCIiLAogICAgICAgICAgICAgICAg IlRBUkdFVF9TWVMgICAgICAgICAgID0gXCJ4ODZfNjQtcG9reS1saW51eFwiIiwKICAgICAgICAg ICAgICAgICJNQUNISU5FICAgICAgICAgICAgICA9IFwiZ2VuZXJpY3g4Ni02NFwiIiwKICAgICAg ICAgICAgICAgICJESVNUUk8gICAgICAgICAgICAgICA9IFwicG9reVwiIiwKICAgICAgICAgICAg ICAgICJESVNUUk9fVkVSU0lPTiAgICAgICA9IFwiMy4xLjEwXCIiLAogICAgICAgICAgICAgICAg IlRVTkVfRkVBVFVSRVMgICAgICAgID0gXCJtNjQgY29yZTJcIiIsCiAgICAgICAgICAgICAgICAi VEFSR0VUX0ZQVSAgICAgICAgICAgPSBcIlwiIiwKICAgICAgICAgICAgICAgICJtZXRhICAgICAg ICAgICAgICAgICAiLAogICAgICAgICAgICAgICAgIm1ldGEtcG9reSAgICAgICAgICAgICIsCiAg ICAgICAgICAgICAgICAibWV0YS15b2N0by1ic3AgICAgICAgPSBcImR1bmZlbGw6MmE4NDhlOTUw NzQzMThmM2EyNDNkZjdiM2Y0MDUxM2ExMzE3M2E4MlwiIiwKICAgICAgICAgICAgICAgICJtZXRh LWludGVsICAgICAgICAgICA9IFwiZHVuZmVsbDpkOGJmODZhZTYyODhhZTUyMGI4ZGRkNzIwOWEw YjQ0OGI5NjkzZjQ4XCIiLAogICAgICAgICAgICAgICAgIm1ldGEteXAtcWEgICAgICAgICAgID0g XCJkdW5mZWxsOmE3NjFlMDYxMDBjOTljMjI3NDY0MTc1ODI4MWFlNDgyMjRlMTYzY2ZcIiIKICAg ICAgICAgICAgXQogICAgICAgIH0sCiAgICAgICAgInJlc3VsdCI6ICJjb3JlLWltYWdlLXNhdG8t c2RrIGhhcyBiZWluZyBidWlsdCBmb3JnZW5lcmljeDg2LTY0IgogICAgfSwKICAgICJidWlsZF9p bWFnZV9jb3JlLWltYWdlLXNhdG8tc2RrXzIwMjEwODI0MTYxNzQxIjogewogICAgICAgICJjb25m aWd1cmF0aW9uIjogWwogICAgICAgICAgICAiQkJfVkVSU0lPTiAgICAgICAgICAgPSBcIjEuNDYu MFwiIiwKICAgICAgICAgICAgIkJVSUxEX1NZUyAgICAgICAgICAgID0gXCJ4ODZfNjQtbGludXhc IiIsCiAgICAgICAgICAgICJOQVRJVkVMU0JTVFJJTkcgICAgICA9IFwidW5pdmVyc2FsXCIiLAog ICAgICAgICAgICAiVEFSR0VUX1NZUyAgICAgICAgICAgPSBcIng4Nl82NC1wb2t5LWxpbnV4XCIi LAogICAgICAgICAgICAiTUFDSElORSAgICAgICAgICAgICAgPSBcImdlbmVyaWN4ODYtNjRcIiIs CiAgICAgICAgICAgICJESVNUUk8gICAgICAgICAgICAgICA9IFwicG9reVwiIiwKICAgICAgICAg ICAgIkRJU1RST19WRVJTSU9OICAgICAgID0gXCIzLjEuMTBcIiIsCiAgICAgICAgICAgICJUVU5F X0ZFQVRVUkVTICAgICAgICA9IFwibTY0IGNvcmUyXCIiLAogICAgICAgICAgICAiVEFSR0VUX0ZQ VSAgICAgICAgICAgPSBcIlwiIiwKICAgICAgICAgICAgIm1ldGEgICAgICAgICAgICAgICAgICIs CiAgICAgICAgICAgICJtZXRhLXBva3kgICAgICAgICAgICAiLAogICAgICAgICAgICAibWV0YS15 b2N0by1ic3AgICAgICAgPSBcImR1bmZlbGw6MmE4NDhlOTUwNzQzMThmM2EyNDNkZjdiM2Y0MDUx M2ExMzE3M2E4MlwiIiwKICAgICAgICAgICAgIm1ldGEtaW50ZWwgICAgICAgICAgID0gXCJkdW5m ZWxsOmQ4YmY4NmFlNjI4OGFlNTIwYjhkZGQ3MjA5YTBiNDQ4Yjk2OTNmNDhcIiIsCiAgICAgICAg ICAgICJtZXRhLXlwLXFhICAgICAgICAgICA9IFwiZHVuZmVsbDphNzYxZTA2MTAwYzk5YzIyNzQ2 NDE3NTgyODFhZTQ4MjI0ZTE2M2NmXCIiCiAgICAgICAgXSwKICAgICAgICAicmVzdWx0IjogImNv cmUtaW1hZ2Utc2F0by1zZGsgaGFzIGJlaW5nIGJ1aWx0IGZvciBnZW5lcmljeDg2LTY0IgogICAg fSwKICAgICJidWlsZF9pbWFnZV9jb3JlLWltYWdlLXNhdG8tc2RrXzIwMjEwODI0MTY0MzI5Ijog ewogICAgICAgICJjb25maWd1cmF0aW9uIjogWwogICAgICAgICAgICAiQkJfVkVSU0lPTiAgICAg ICAgICAgPSBcIjEuNDYuMFwiIiwKICAgICAgICAgICAgIkJVSUxEX1NZUyAgICAgICAgICAgID0g XCJ4ODZfNjQtbGludXhcIiIsCiAgICAgICAgICAgICJOQVRJVkVMU0JTVFJJTkcgICAgICA9IFwi dW5pdmVyc2FsXCIiLAogICAgICAgICAgICAiVEFSR0VUX1NZUyAgICAgICAgICAgPSBcIng4Nl82 NC1wb2t5LWxpbnV4XCIiLAogICAgICAgICAgICAiTUFDSElORSAgICAgICAgICAgICAgPSBcInFl bXV4ODYtNjRcIiIsCiAgICAgICAgICAgICJESVNUUk8gICAgICAgICAgICAgICA9IFwicG9reVwi IiwKICAgICAgICAgICAgIkRJU1RST19WRVJTSU9OICAgICAgID0gXCIzLjEuMTBcIiIsCiAgICAg ICAgICAgICJUVU5FX0ZFQVRVUkVTICAgICAgICA9IFwibTY0IGNvcmUyXCIiLAogICAgICAgICAg ICAiVEFSR0VUX0ZQVSAgICAgICAgICAgPSBcIlwiIiwKICAgICAgICAgICAgIm1ldGEgICAgICAg ICAgICAgICAgICIsCiAgICAgICAgICAgICJtZXRhLXBva3kgICAgICAgICAgICAiLAogICAgICAg ICAgICAibWV0YS15b2N0by1ic3AgICAgICAgPSBcImR1bmZlbGw6MmE4NDhlOTUwNzQzMThmM2Ey NDNkZjdiM2Y0MDUxM2ExMzE3M2E4MlwiIiwKICAgICAgICAgICAgIm1ldGEtaW50ZWwgICAgICAg ICAgID0gXCJkdW5mZWxsOmQ4YmY4NmFlNjI4OGFlNTIwYjhkZGQ3MjA5YTBiNDQ4Yjk2OTNmNDhc IiIsCiAgICAgICAgICAgICJtZXRhLXlwLXFhICAgICAgICAgICA9IFwiZHVuZmVsbDphNzYxZTA2 MTAwYzk5YzIyNzQ2NDE3NTgyODFhZTQ4MjI0ZTE2M2NmXCIiCiAgICAgICAgXSwKICAgICAgICAi cmVzdWx0IjogImNvcmUtaW1hZ2Utc2F0by1zZGsgaGFzIGJlaW5nIGJ1aWx0IGZvciBxZW11eDg2 LTY0IgogICAgfQp9