Created attachment 3342 [details] Failure log with stack trace from bitbake The issue is here http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/controllers/masterimage.py?h=krogoth#n109 . "shutdown -h now" may return a non-zero error code but still be successful: http://paste2.org/Bm4vVc1O In the context of the power_cycle function, conn.ignore_status is set to False from http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/controllers/masterimage.py?h=krogoth#n192 and an AssertionError is raised http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/utils/sshcontrol.py?h=krogoth#n127 . I've attached a log of what the failure looks like on my system. The exception thrown isn't particularly helpful. I think this gets thrown the second time conn.run("shutdown -h now") is called and the error is silently ignored the first time. I had to step through with pdb to see what the issue was.
This is an interesting implementation for a remote shutdown, I wonder if does it ever return zero instead of the 255 as the only guarantee you have is that the target process that is servicing your connection will be abruptly killed. I am not proposing the following as a fix but it would help to understand the issue better, what happens if you change the line that: "shutdown -h now" For: "shutdown -h now; exit" Would you test that Nic?
(In reply to comment #1) > Would you test that Nic? Just tested it, no changes.
Okay Nic, thanks for the testing. Sending this bug to Leo for scheduling the investigation. Seems to me like a case where the ssh command return code is not important and if you wanted to check that the device shut down, you could have a function to check for the ping response and assume that once it stops responding then it has turned off.
(In reply to comment #0) > Created attachment 3342 [details] > Failure log with stack trace from bitbake > Nic, I think you did not attach the strack trace, just the python code. In case you still have it, please attach it. > The issue is here > http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/ > controllers/masterimage.py?h=krogoth#n109 . > > "shutdown -h now" may return a non-zero error code but still be successful: > http://paste2.org/Bm4vVc1O I am not seeing a non-zero, after shutdown: $ ssh -l root 192.168.7.2 'shutdown -h now' $ echo $? 0 In your case, I see that you are echoing $status, can you try echoing $? ? > > In the context of the power_cycle function, conn.ignore_status is set to > False from > http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/ > controllers/masterimage.py?h=krogoth#n192 and an AssertionError is raised > http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/utils/ > sshcontrol.py?h=krogoth#n127 . Right, at some point the ignore_status is set to False, so following ssh calls would complain in case non-zero return values are present. One simple approach to fix this would be to always set this (ignore_status) value to true on the power_cycle method, but I am not 100% confident with this change. One other thing: Please attach any other info that can be useful to reproduce the problem. > > I've attached a log of what the failure looks like on my system. The > exception thrown isn't particularly helpful. I think this gets thrown the > second time conn.run("shutdown -h now") is called and the error is silently > ignored the first time. I had to step through with pdb to see what the issue > was.
Created attachment 3418 [details] Bitbake output including stack trace
Ok I think I have the right one this time :P. We're using a valleyisland chip. I'll attach some more files including the extra layer I'm using to reproduce the issue. This is how it plays out in bash: nic@ndonaldson-ubuntu:~$ ssh -l root 172.22.1.52 'shutdown -h now' Connection to 172.22.1.52 closed by remote host. nic@ndonaldson-ubuntu:~$ echo $? 255
Created attachment 3419 [details] bitbake layer that modifies networking As it is on my system, except for recipes-connectivity/dhcp/dhcp/dhclient.conf where I've removed the MAC address at the end.
Created attachment 3420 [details] bblayers
Created attachment 3421 [details] local.conf
Accepting it again. I will give it a try with latest info from Nic.
What about trying 'shutdown -h now || true', this should force a 0 status return.
(In reply to comment #6) > Ok I think I have the right one this time :P. We're using a valleyisland > chip. I'll attach some more files including the extra layer I'm using to > reproduce the issue. > > This is how it plays out in bash: > nic@ndonaldson-ubuntu:~$ ssh -l root 172.22.1.52 'shutdown -h now' > Connection to 172.22.1.52 closed by remote host. > nic@ndonaldson-ubuntu:~$ echo $? > 255 Nic, the error on this log is something different and new: File: '/media/nic/ext/yocto_stuff/meta/lib/oeqa/controllers/masterimage.py', lineno: 109, function: power_cycle 0105: 0106: def power_cycle(self, conn): 0107: if self.powercontrol_cmd: 0108: # be nice, don't just cut power *** 0109: conn.run("shutdown -h now; exit") 0110: time.sleep(10) 0111: self.power_ctl("cycle") 0112: else: 0113: status, output = conn.run("reboot") Exception: AttributeError: 'NoneType' object has no attribute 'run' So you may have found a new error. In this case, conn is None, and in the other case conn.ignore_status is False, so you were seeing the non-zero return code. For the original case, can you try what Saul suggested (I have include a patch, but NOT TESTED) and provide feedback: diff --git a/meta/lib/oeqa/controllers/masterimage.py b/meta/lib/oeqa/controllers/masterimage.py index 522f9eb..9440a51 100644 --- a/meta/lib/oeqa/controllers/masterimage.py +++ b/meta/lib/oeqa/controllers/masterimage.py @@ -106,7 +106,7 @@ class MasterImageHardwareTarget(oeqa.targetcontrol.BaseTarget): def power_cycle(self, conn): if self.powercontrol_cmd: # be nice, don't just cut power - conn.run("shutdown -h now") + conn.run("shutdown -h now || true") time.sleep(10) self.power_ctl("cycle") else:
No luck with 'shutdown -h now || true', the connection still cuts out first. Originally I was getting the conn is None error, which didn't make sense. When I stepped through with pdb I saw that conn.run("shutdown -h now") was being run twice. The first time it raised an AssertionError. That's silently dropped in testimage.bbclass here http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/classes/testimage.bbclass?h=krogoth#n243 . Then target.stop() (line 259) is called and the second time around conn is None for some reason, giving the error you see in the log. For demonstration, I'm attaching a log of the output when I add the following to testimage.bbclass: nic@ndonaldson-ubuntu ~> diff /media/nic/ext/yocto_stuff/meta/classes/testimage.bbclass /media/nic/ext/yocto_stuff/meta/classes/testimage_except.bbclass 256a257,258 > except AssertionError as e: > bb.plain("AssertionError")
Created attachment 3467 [details] Bitbake output with stack trace and AssertionError
V1 Patch sent to the mailing list: http://lists.openembedded.org/pipermail/openembedded-core/2016-December/130157.html Also moving to M2 because M1 is over.
Sent already a ping to the maintainer so he considers for the next patch set to be tested and merged. Moving to M3.
Nik, Patch is there but you need to test the patch. Please test it and let me know the result
In case the problem is observed in the future, bug can be reopened. The reason is that I do not have the full layer structure as the reporter, so I am not able to reproduce the problem.