9721 2016-06-07 08:34:19 +0000 ptest - flex running results not generated as Ptest format 2016-06-21 11:15:36 +0000 1 1 10 QA/Testing Runtime Testing general unspecified All Multiple RESOLVED FIXED https://bugzilla.yoctoproject.org/show_bug.cgi?id=9752 Medium critical 2.2 1 yix.wang leonardo.sandoval.gonzalez benjamin.esquivel francisco.j.pedraza.gonzalez leonardo.sandoval.gonzalez ross.burton francisco.j.pedraza.gonzalez --- --- No (bug/feature does not impact docs) oldest_to_newest 62694 0 yix.wang 2016-06-07 08:34:19 +0000 steps: 1. cd /usr/lib/flex/ptest 2. ./run-ptest the results: root@intel-quark:/usr/lib/flex/ptest# ./run-ptest make[1]: Entering directory '/usr/lib/flex/ptest' touch alloc_extra.log touch array_nr.log touch array_r.log touch basic_nr.log touch basic_r.log touch bison_nr.log touch bison_yylloc.log touch bison_yylval.log touch c_cxx_nr.log touch c_cxx_r.log touch ccl.log touch cxx_basic.log touch cxx_multiple_scanners.log touch debug_nr.log touch debug_r.log touch extended.log touch header_nr.log touch header_r.log touch mem_nr.log touch mem_r.log touch multiple_scanners_nr.log touch multiple_scanners_r.log touch noansi_nr.log touch noansi_r.log touch posix.log touch posixly_correct.c make[1]: touch: open: posixly_correct.c: Permission denied make[1]: Leaving directory '/usr/lib/flex/ptest' Makefile:2249: recipe for target 'check-TESTS' failed make: *** [check-TESTS] Error 2 make[1]: Entering directory '/usr/lib/flex/ptest' /bin/sh: ../build-aux/test-driver: No such file or directory Makefile:2270: recipe for target 'alloc_extra.log' failed make[1]: *** [alloc_extra.log] Error 127 make[1]: Leaving directory '/usr/lib/flex/ptest' Makefile:2249: recipe for target 'check-TESTS' failed make: *** [check-TESTS] Error 2 62788 1 benjamin.esquivel 2016-06-08 20:16:14 +0000 hello Wangyi, to see it at the ptest format you'd need to run it via ptest. It can be done as: $ ptest-runner flex the output you'll see is mostly what you already pasted here with additional things like: START: ptest-runner BEGIN: /usr/lib/flex/ptest and other tags at the end pertaining ptest. The problem I see though is that no test gets marked as passed or failed. I did some initial investigation and it looks like the flex/ptest file is failing spectacularly. This points to a malfunction in the flex Makefile itself(check-TESTS target), it could be permissions, it could be something else, I did not dig up more. Likely a bug in the flex tests themselves or the way we're setting up the environment to run. 62795 2 benjamin.esquivel 2016-06-08 22:08:02 +0000 One interesting thing to note here is that flex ptest results are not being reported, so we can't call this a regression just yet. See how flex is not reported here: https://wiki.yoctoproject.org/wiki/Ptest_6c1c01392d91f512e2949ad1d57a75a8077478ba then the question is whether flex is enabled, which in code seems like it is, even though a (most likely outdated) ptest wiki says it is in progress: https://wiki.yoctoproject.org/wiki/Ptest#Recipes_in_Progress I have requested Francisco a full run of ptest to check the logs and see if it is actually run or not on the QA cycles. It would also be good to check if there are other components in ptest that are enabled but not reported. 62796 3 benjamin.esquivel 2016-06-08 22:09:33 +0000 This even fails in qemux86-64 so setting Platform to all 62872 4 leonardo.sandoval.gonzalez 2016-06-09 20:10:59 +0000 I am seeing a different problem, not related to 'touch' . . touch options.log mafatal: making test-suite.log: failed to create alloc_extra.trs fatal: making test-suite.log: failed to create array_nr.trs fatal: making test-suite.log: failed to create array_r.trs fatal: making test-suite.log: failed to create basic_nr.trs fatal: making test-suite.log: failed to create basic_r.trs fatal: making test-suite.log: failed to create bison_nr.trs fatal: making test-suite.log: failed to create bison_yylloc.trs fatal: making test-suite.log: failed to create bison_yylval.trs . . fatal: making test-suite.log: failed to create tableopts_ver_r-Cam.trs fatal: making test-suite.log: failed to create tableopts_ver_r-Caem.trs fatal: making test-suite.log: failed to create options.trs make[1]: *** [test-suite.log] Error 1 make: *** [check-TESTS] Error 2 /bin/sh: ../build-aux/test-driver: No such file or directory make[1]: *** [alloc_extra.log] Error 127 make: *** [check-TESTS] Error 2 ke[2]: Leaving directory '/usr/lib/flex/ptest' Makefile:2143: recipe for target 'test-suite.log' failed make[1]: Leaving directory '/usr/lib/flex/ptest' Makefile:2249: recipe for target 'check-TESTS' failed make[1]: Entering directory '/usr/lib/flex/ptest' Makefile:2270: recipe for target 'alloc_extra.log' failed make[1]: Leaving directory '/usr/lib/flex/ptest' Makefile:2249: recipe for target 'check-TESTS' failed END: /usr/lib/flex/ptest 2016-06-09T12:03 STOP: ptest-runner 62877 5 benjamin.esquivel 2016-06-09 21:23:49 +0000 My first suspect is the flex tests themselves, these would need to be tested in the same version that the recipe points to. I believe that you have to build flex before you can try the tests. One other update I have is that running ptest-runner DOES execute flex along with the rest of the components. Problem is that the flex output, as stated in this bug, has no PASS/FAIL outcome. Therefore, my theory is that the script that parses the ptest log is not even identifying flex as a component that should have results reported. This is clearly another bug in the parser given that the log signals that flex testing has started in the form of this message: START: ptest-runner BEGIN: /usr/lib/flex/ptest 62943 6 benjamin.esquivel 2016-06-13 22:16:31 +0000 Leo, please paste the mail archives link to the review. 62958 7 leonardo.sandoval.gonzalez 2016-06-14 13:42:23 +0000 (In reply to comment #6) > Leo, please paste the mail archives link to the review. You are right. http://lists.openembedded.org/pipermail/openembedded-core/2016-June/122786.html Please check the SHA indicated in the commit description so you can see the root reason of this change. 62966 8 benjamin.esquivel 2016-06-14 17:23:10 +0000 testing your change on the tip of master. 62973 9 benjamin.esquivel 2016-06-14 19:22:53 +0000 I see the intended change working. I will ACK it over the mailing list. 63184 10 ross.burton 2016-06-21 11:15:36 +0000 Fixed in oe-core c5da9c7a3c6418386e1f923008e1cd5fbee01fdd.