9382 2016-04-04 09:26:35 +0000 git fetcher fails if certain environment variables contain non-ASCII chars 2016-04-06 09:48:37 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake unspecified x86 Multiple RESOLVED FIXED Undecided normal --- 1 daniel.klauer richard.purdie poky.bs.watcher poky.watcher --- --- Don't know oldest_to_newest 60728 0 3089 daniel.klauer 2016-04-04 09:26:35 +0000 Created attachment 3089 Suggested fix: make Python sqlite3 module return byte strings for SQL text queries Hello, bitbake's git fetcher is prepending shell export commands for certain environment variables (HOME, PATH, but also others - see [1]) to every shell command it runs via runfetchcmd(). It fails if these variables contain non-ASCII chars: File: '.../poky/bitbake/lib/bb/fetch2/__init__.py', lineno: 812, function: runfetchcmd 0808: 0809: for var in exportvars: 0810: val = d.getVar(var, True) 0811: if val: *** 0812: cmd = 'export ' + var + '=\"%s\"; %s' % (val, cmd) 0813: 0814: logger.debug(1, "Running %s", cmd) 0815: 0816: success = False Exception: UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 17: ordinal not in range(128) Reproducing the issue seems to be as simple as: 1. $ git clone -b jethro git://git.yoctoproject.org/poky.git 2. $ cd poky 3. $ mkdir meta/recipes-support/test 4. create recipe meta/recipes-support/test/testgit.bb: # just a test recipe LICENSE = "CLOSED" SRC_URI = "git://github.com/schacon/simplegit.git;protocol=https" SRCREV = "${AUTOREV}" 5. $ source oe-init-build-env 6. $ SOCKS5_USER=ü bitbake testgit It looks like this error happens because bitbake's internal SQL database (bb.persist_data.SQLTable) returns Unicode strings instead of byte strings, which sometimes trickle down to runfetchcmd() (due to dynamic typing), causing Unicode conversions when appending to the cmd string. I have only observed this problem with the Git fetcher when using AUTOREV. FetchMethod.latest_revision() caches the HEAD revision as a string in an SQL database (bb.persist_data.SQLTable) using the Python 2 sqlite3 module, which defaults to returning unicode strings when querying text, according to [1]. [1] https://docs.python.org/2/library/sqlite3.html#sqlite-and-python-types 60729 1 daniel.klauer 2016-04-04 09:31:12 +0000 oops, messed up the links, that should be: [1] http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/__init__.py?h=jethro&id=2fb7ee2628e23d7efc9b041bb9daae7c4a8de541#n789 [2] https://docs.python.org/2/library/sqlite3.html#sqlite-and-python-types 60745 2 richard.purdie 2016-04-04 15:21:40 +0000 Could you send that patch to the bitbake mailing list with a signed-off-by line and a condensed explanation as a commit message please? The patch seems reasonable to me. 60796 3 daniel.klauer 2016-04-05 11:18:11 +0000 Ok, done. 60838 4 richard.purdie 2016-04-06 09:48:37 +0000 Thanks for posting it, much appreciated. It went through testing and was merged: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=4f07c2220bedf50743bc384dd221eb14f6b0b300 3089 2016-04-04 09:26:35 +0000 2016-04-04 09:26:35 +0000 Suggested fix: make Python sqlite3 module return byte strings for SQL text queries bitbake-git-fetcher-unicode.patch text/plain 467 daniel.klauer ZGlmZiAtLWdpdCBhL2JpdGJha2UvbGliL2JiL3BlcnNpc3RfZGF0YS5weSBiL2JpdGJha2UvbGli L2JiL3BlcnNpc3RfZGF0YS5weQppbmRleCA1Nzk1YmM4Li5lNDUwNDIzIDEwMDY0NAotLS0gYS9i aXRiYWtlL2xpYi9iYi9wZXJzaXN0X2RhdGEucHkKKysrIGIvYml0YmFrZS9saWIvYmIvcGVyc2lz dF9kYXRhLnB5CkBAIC0yMDEsNiArMjAxLDcgQEAgY2xhc3MgUGVyc2lzdERhdGEob2JqZWN0KToK IGRlZiBjb25uZWN0KGRhdGFiYXNlKToKICAgICBjb25uZWN0aW9uID0gc3FsaXRlMy5jb25uZWN0 KGRhdGFiYXNlLCB0aW1lb3V0PTUsIGlzb2xhdGlvbl9sZXZlbD1Ob25lKQogICAgIGNvbm5lY3Rp b24uZXhlY3V0ZSgicHJhZ21hIHN5bmNocm9ub3VzID0gb2ZmOyIpCisgICAgY29ubmVjdGlvbi50 ZXh0X2ZhY3RvcnkgPSBzdHIKICAgICByZXR1cm4gY29ubmVjdGlvbgogCiBkZWYgcGVyc2lzdChk b21haW4sIGQpOgo=