Created attachment 3089 [details]
Suggested fix: make Python sqlite3 module return byte strings for SQL text queries

Hello,

bitbake's git fetcher is prepending shell export commands for certain environment variables (HOME, PATH, but also others - see [1]) to every shell command it runs via runfetchcmd(). It fails if these variables contain non-ASCII chars:

File: '.../poky/bitbake/lib/bb/fetch2/__init__.py', lineno: 812, function: runfetchcmd
     0808:
    0809:    for var in exportvars:
     0810:        val = d.getVar(var, True)
     0811:        if val:
 *** 0812:            cmd = 'export ' + var + '=\"%s\"; %s' % (val, cmd)
     0813:
     0814:    logger.debug(1, "Running %s", cmd)
     0815:
     0816:    success = False
Exception: UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 
17: ordinal not in range(128)

Reproducing the issue seems to be as simple as:

    1. $ git clone -b jethro git://git.yoctoproject.org/poky.git
    2. $ cd poky
    3. $ mkdir meta/recipes-support/test
    4. create recipe meta/recipes-support/test/testgit.bb:
        # just a test recipe
        LICENSE = "CLOSED"
        SRC_URI = "git://github.com/schacon/simplegit.git;protocol=https"
        SRCREV = "${AUTOREV}"
    5. $ source oe-init-build-env
    6. $ SOCKS5_USER=ü bitbake testgit

It looks like this error happens because bitbake's internal SQL database (bb.persist_data.SQLTable) returns Unicode strings instead of byte strings, which sometimes trickle down to runfetchcmd() (due to dynamic typing), causing Unicode conversions when appending to the cmd string.

I have only observed this problem with the Git fetcher when using AUTOREV. FetchMethod.latest_revision() caches the HEAD revision as a string in an SQL database (bb.persist_data.SQLTable) using the Python 2 sqlite3 module, which defaults to returning unicode strings when querying text, according to [1].

[1] https://docs.python.org/2/library/sqlite3.html#sqlite-and-python-types