Code: while time.time() < endtime and not eof: if select.select([self.process.stdout], [], [], 5)[0] != []: data = os.read(self.process.stdout.fileno(), 1024) if not data: self.process.stdout.close() eof = True else: data = data.decode("utf-8") output += data self.log(data) endtime = time.time() + timeout Errors at ' data = data.decode("utf-8")' UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc4 in position 1023: unexpected end of data Reproducibility : < 1/10 Possible workaround: data = data.decode("utf-8", 'replace')
Can you replicate the problem using one of standard tools (not a sample code)? I want to understand the scenario you were trying to set when you found the problem.
Just do some tests with running command likes "python runtest.py -t 192.168.8.111 -m intel-corei7-64 -f testplan/iottest.manifest" , but the reproducibility is very low , so we just met once. Why I listed the sample code since I saw below info. So I assume the possible cause is the 1024 read cache. -------------------------------------------------------- Traceback (most recent call last): File "/home/tester/iotos-tester/workspace/test_beagleboneblack_2/iottest/oeqa/utils/decorators.py", line 97, in wrapped_f return f(*args, **kwargs) File "/home/tester/iotos-tester/workspace/test_beagleboneblack_2/iottest/oeqa/runtime/smack.py", line 580, in test_smack_labels ' '.join([x.rstrip('/') for x in files.split()]), files, files File "iottest/runtest.py", line 67, in run return self.connection.run(cmd, timeout) File "/home/tester/iotos-tester/workspace/test_beagleboneblack_2/iottest/oeqa/utils/sshcontrol.py", line 144, in run return self._internal_run(command, self.defaulttimeout, self.ignore_status) File "/home/tester/iotos-tester/workspace/test_beagleboneblack_2/iottest/oeqa/utils/sshcontrol.py", line 124, in _internal_run status, output = proc.run(command, timeout, logfile=self.logfile) File "/home/tester/iotos-tester/workspace/test_beagleboneblack_2/iottest/oeqa/utils/sshcontrol.py", line 87, in run self._run(command, timeout, logfile) File "/home/tester/iotos-tester/workspace/test_beagleboneblack_2/iottest/oeqa/utils/sshcontrol.py", line 61, in _run data = data.decode("utf-8") UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc4 in position 1023: unexpected end of data --------------------------------------------------------
(In reply to comment #2) > Just do some tests with running command likes "python runtest.py -t > 192.168.8.111 -m intel-corei7-64 -f testplan/iottest.manifest" , but the > reproducibility is very low , so we just met once. Why I listed the sample > code since I saw below info. So I assume the possible cause is the 1024 read > cache. > -------------------------------------------------------- > Traceback (most recent call last): > File > "/home/tester/iotos-tester/workspace/test_beagleboneblack_2/iottest/oeqa/ > utils/decorators.py", line 97, in wrapped_f > return f(*args, **kwargs) > File > "/home/tester/iotos-tester/workspace/test_beagleboneblack_2/iottest/oeqa/ > runtime/smack.py", line 580, in test_smack_labels > ' '.join([x.rstrip('/') for x in files.split()]), files, files > File "iottest/runtest.py", line 67, in run > return self.connection.run(cmd, timeout) > File > "/home/tester/iotos-tester/workspace/test_beagleboneblack_2/iottest/oeqa/ > utils/sshcontrol.py", line 144, in run > return self._internal_run(command, self.defaulttimeout, > self.ignore_status) > File > "/home/tester/iotos-tester/workspace/test_beagleboneblack_2/iottest/oeqa/ > utils/sshcontrol.py", line 124, in _internal_run > status, output = proc.run(command, timeout, logfile=self.logfile) > File > "/home/tester/iotos-tester/workspace/test_beagleboneblack_2/iottest/oeqa/ > utils/sshcontrol.py", line 87, in run > self._run(command, timeout, logfile) > File > "/home/tester/iotos-tester/workspace/test_beagleboneblack_2/iottest/oeqa/ > utils/sshcontrol.py", line 61, in _run > data = data.decode("utf-8") > UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc4 in position 1023: > unexpected end of data > -------------------------------------------------------- Thanks for the info. It will be tackle before the mentioned milestone.
(In reply to comment #2) > Just do some tests with running command likes "python runtest.py -t > 192.168.8.111 -m intel-corei7-64 -f testplan/iottest.manifest" , but the > reproducibility is very low , so we just met once. Why I listed the sample > code since I saw below info. So I assume the possible cause is the 1024 read > cache. What locale setting are you using in your terminal session? This error means that SSH is not returning UTF-8 encoding. Can you reproduce this error setting your terminal to UTF-8?
I'm still not able to reproduce this. Assigning to submitter to get more information.
Not sure about the terminal codec , considering the reproducibility is very low , I am to OK to close this as 'Won't fix'.