14822 2022-06-02 03:26:38 +0000 Inconsistent log when offline package is damaged and BB_NO_NETWORK is true 2022-06-16 15:48:59 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake unspecified x86 Multiple RESOLVED OBSOLETE Medium normal 4.1 1 watsondd pavel poky.bs.watcher poky.watcher randy.macleod --- --- No (bug/feature does not impact docs) oldest_to_newest 93363 0 4871 watsondd 2022-06-02 03:26:38 +0000 Created attachment 4871 Contains log.do_fetch.0 and log.do_fetch.1 ---------------Summary--------------- If some packages on local machine for offline build is damaged, the log.do_fetch will change when running the same command. ---------------Details step to reproduce--------------- 1. According to the instructions on https://docs.yoctoproject.org/dev-manual/common-tasks.html#replicating-a-build-offline, prepare an offline premirrors on the local machine, the path is <offline_downloads>. 2. Create a new build directory by `source oe-init-build-env <my_build_path>`. Make sure <my_build_path> is not under git repository, i.e. its ancestor diretories and itself does not contain .git directory. 3. Edit conf/local.conf in <my_build_path>. Add the following lines to the default one. ############### BB_GENERATE_MIRROR_TARBALLS = "1" INHERIT += "own-mirrors" SOURCE_MIRROR_URL = "file://<offline_downloads>" #The location where nativesdk comes from UNINATIVE_URL = "file://<offline_downloads>/uninative/9bfc4c970495b3716b2f9e52c4df9f968c02463a9a95000f6657fbc3fde1f098/" UNINATIVE_TARBALL = "x86_64-nativesdk-libc-3.6.tar.xz" UNINATIVE_CHECKSUM[x86_64] = "9bfc4c970495b3716b2f9e52c4df9f968c02463a9a95000f6657fbc3fde1f098" PREMIRRORS:prepend = "\ git://.*/.* http://example.com/ \n \ ftp://.*/.* http://example.com/ \n \ http://.*/.* http://example.com/ \n \ https://.*/.* http://example.com/ \n" BB_NO_NETWORK = "1" ############### 4. Change the content of git2_github.com.besser82.libxcrypt.git.tar.gz in <offline_downloads> so that it is not a valid tar.gz format. 5. Run the command `bitbake core-image-minimal --runall=unpack`. The content of log.do_fetch is like log.do_fetch.0. This is some analysis for the log. The download(self, urls=None) method of Fetch class calls m.try_premirror(ud, self.d) to check whether to try premirrors. Here, m is Git class type. Because BB_FETCH_PREMIRRORONLY is false and downloads/git2/github.com.besser82.libxcrypt.git/ does not exist, try_premirror(self, ud, d) of Git class return true. The download(self, urls=None) method of Fetch class calls the method try_mirrors(self, fetch, urldata, d, mirrors, check=False) of FetchMethod class because Git class dose not have try_mirrors(). The static method try_mirrors(fetch, d, origud, mirrors, check = False) is called. The static method try_mirror_url(fetch, origud, ud, ld, check = False) is called. In line 64, try_mirror_url() calls origud.method.download(origud, ld). Here, origud is Git class type. I add some log messages to debug this issue. In line 65, the download(self, ud, d) method of Git class calls runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir) to decompress the offline package. In line 69~71, runfetchcmd() fails because the file is an invalid tar.gz format. In line 73, bb.fetch2.BBFetchException is thrown by origud.method.download(). The method try_mirror_url() in __init__.py calls ud.method.clean(ud, ld). Note that ud is Local class here, and origud is Git class. The empty directory downloads/git2/github.com.besser82.libxcrypt.git/ will remain in <my_build_path>. 6. Run the command `bitbake core-image-minimal --runall=unpack` again. The content of log.do_fetch is like log.do_fetch.1. This is some analysis for the log. The download(self, urls=None) method of Fetch class calls m.try_premirror(ud, self.d) to check whether to try premirrors. Here, m is Git class type. Because BB_FETCH_PREMIRRORONLY is false and downloads/git2/github.com.besser82.libxcrypt.git/ already exist, try_premirror(self, ud, d) of Git class return false. In line 8, the download(self, urls=None) method of Fetch class calls m.download(ud, self.d) of Git class to download from Git server. In line 16~17, the download(self, urls=None) method of Fetch class calls m.clean(ud, self.d) to remove downloads/git2/github.com.besser82.libxcrypt.git/. 7. Repeat the command `bitbake core-image-minimal --runall=unpack`. The content of log.do_fetch for git2_github.com.besser82.libxcrypt.git.tar.gz will switch between log.do_fetch.0 and log.do_fetch.1. ---------------Simple conclusion--------------- Currently, the static method try_mirror_url(fetch, origud, ud, ld, check = False) just calls ud.method.clean() for exceptions. In my case, origud.method.download() is called and leaves an empty directory. And it affects the second command. It may be necessary for try_mirror_url() to call origud.method.clean() if some exceptions is thrown by origud.method.download(). If I make such changes to try_mirror_url(), the log.do_fetch will be always like log.do_fetch.0. Note: - In step 2, it is important that the build directory <my_build_path> is not a descendant of a git repository. Otherwise, the behaviour will be different. - In step 4, I just choose the package for my example. The behaviour is same if another package using Git protocol is chosen. - In step 4, alternatively I could change the permission of the file git2_github.com.besser82.libxcrypt.git.tar.gz to un-readable. - The use case is unusual. Although we should take care of the offline packages and keep them consistent, it is still possible to corrupt them by accident. The log file should help find the root cause. 93536 1 pavel 2022-06-16 15:48:59 +0000 This was eventually fixed in https://git.openembedded.org/bitbake/commit/?id=b47ecab3e3aad5c5c376ec023aa82a51aa0f3b86 In current implementation fetcher will not try git commands if it sees fullmirror (even broken one). The flow fails with [1] in both cases. [1] gzip: stdin: not in gzip format tar: Child returned status 1 tar: Error is not recoverable: exiting now 4871 2022-06-02 03:26:38 +0000 2022-06-02 03:26:38 +0000 Contains log.do_fetch.0 and log.do_fetch.1 log.do_fetch.zip application/x-zip-compressed 5715 watsondd UEsDBBQAAAAIAGJNwlTEAPYUPAUAABNCAAAOAAAAbG9nLmRvX2ZldGNoLjDtnF9v2jAQwN/5FBYv tFOTtKyrtkxoooOt1TZAQLWHqkIhMcRrsCPb4U8//c4hQNuVFlSIYelDBYnPd/e7xBdfXFOpnl99 t1F1jN1IEtpH4UT6jKJeRF1J4AseS0y9DscuCXFHTARnTOZq9XbVRhUCpyXycAgimLoEC+RwjK5v EoFLKqQTBNhDhEqGkt72QqB1S8JQmXWgZ8Cx403AIhFSQI8H8omjjVXcQz1CifCxl3sRz2OdHpau /7Jk1xG481i8zSdKttGs/rpsNuvN1qzhG+Mo4gG6LvSJLBwh9eFHXdNlA3VkdbEQmH8sWgHpjl0+ CaWZCCZ/de5hjr0KceXB9UGhyx3q+qrNw0McsLBweIQOCiFnkrksUA2+lKEoHN4c3iCwEjocPAP7 7lCoVvNdbBc+nrYBvSQD8R4J8EzE8hzpWCF2uOWxEQ2Y4wkrZLeTpTp2D18MaZbxQa0+fDBuW9aC fQn35yleKWH7PMMqxUyIYxlxqnCUd6BwqWPKVLGzsGfO7JkP7JnS4Wb/7pG3q9KvZ2RJrHZrjK6k bxvcmgfnSvq2wQ3yYpBJcr+fSezuHc8kd3iaSWwm3Exyxw/sL5lE78kwk9w0vP8g+5IdcPE+k9d7 zYJCZQTVjMfOIAzwvALaB9Q1h/Q+o/oyW6xrlVv7DJsMV8Ei7uIRvJwzGY+n4MqY1QfFrtK5QgTi kM0dNifMlQwq9T/wBnCuc0A4Z9yaWhP/QYC6hMLbwEAYfa+7E3HagfdJeitX3QHQW8Dqptdcx+rG 11vO6qbXXNXqxtdf3OqOwJoT4v8NX3epq5tfb8Wrmz7twnfTi2fqGBTe82e96fQj/5Yjrqd2tVm6 5gWyFEk1L4mlSLrmgNpvUr1FU4qseuujFEE1l0IpkuqtelIE1VzgpEiqu5Z57WQkxX8KSvGqaK6v UiTVXUqliKq3akoRdM2799UFUqpwGhbIUqZbq8DaL7x9WQTbmZC8LXtpz226iTUkvF1ATjsL6mbe l9S423HSnS9/l5u1y9p0PxJinPQjzxxg2L/kmTMTs5A2I6rKJnXThoxL1GhVryr1TuWyVT7/Wa2U Tj6jpKVyftXqtKqt1mW91lHfy5VKEw5L+YiSsR060i9ZPKJWBNfAOjk+LlrdSOTn/Vuti075qn3R adW//ijlLTkILSF8wy22fwejSu0DtZw+ptI8Ofv0/uzD2aJno9y+gA6L6S9l6vEXa0i2eAkjAgxH kiG2xh/POmenRgAXYQzOcHU57EXvaRUoXE5CKeyntY4Yv7VcxnHRAE2qw1Td4sayTs1Ts/jR4MfW dLOZkXhiTL2YGZ55s9CxeZNzWy5nQqRCJsDcdi1s1cB23X/ifoNzsuvc4mewfCakZCy4N2Qu6r+q pTy0DHAsDi0wqUPG+K6H/lUyzx5rThBha+WL2qwXtc0Syq84SaF4WyTqOSSIYAdob5q4N/xyBh2o 3AbDPlDK7c0skR3OHzaKAPMZQ3JCPQUGDvXQW77c7KB5y5evs7BVA9t1f7/ypUoIagv7COTV5nQJ KcHDqHiEWCTDSNq5XP+OhDYS0gPnEWVSZVh1TuXBgSNzoMhGX30SeMmLa1AHO+NlJNDJtLEaJ1Ei 4t4QNTbE3OlC8owtwoQNGka5xURvMcFzA+zQBzlsE6v9KBKgZp78CvMhhOkQjfpYIkOiIjLa6P0x MozQEQKurAF1MjJcZDSeCz8qvNa7QiFXVdvtbXQ+vW1myXsaRxvVsFT3adl1Qc3B6+0dofyORSA/ f241nvuhgvlPISyT/lfwL1BLAwQUAAAACABiTcJUdVeGNOkPAAAUlAEADgAAAGxvZy5kb19mZXRj aC4x7J3xc9IwFMd/96/o+cvUk5bh0DlvelNQd+rwBuqp53GhTaGuJL0mHcPzjzcphbHJXIkpJGl/ UKFt3ks+tC+vbd7XVvvlpzcHVvsCugkN0NCKpnSEkeUnyKUB+wAvKEReP4ZuEME+mZIYY3rnpNNr H1itgG2mlgcjdghEbgCJBWJoff+RHXCMCAVhCD0rQBRbWeuDywO6Z0EUcbeAtQxjCLwp8xgQSliL K8dnHf2Yp3uWH6CAjKB359bhebjvQ+qObj9yAAjsXz+8F0/5sZ8iQlnnx/PNpwlCfPsOvIhwTK2P 3fanVqffOu4evXzfbh3uPrOyPa2Xn7r9brvbPe6c9Pnno1brlH09vJug4OIgAnR06MQJchICY2e3 Xm84g4TcXbTvdt/2jz713va7nVfvDu86dBw5hIxqbqP3JZy0TprIAUOIqL37+Omjx83Hly0/HvXe sgYeoMCJIIgdhPvDgKYWMo6klrBhABqcQ+di/3H/8V4tDFBywToTO4MAHSy1jvDZ1CFuHESUHKy2 OsHxmePiGDZqzBJvkJkLg8GFG08j6uzZe3ZjvxbXndkvWst6Upv1Yu543ptLG/JdLny5MSZkIyMj zF2xHgp1UGz3V5xvbBsdgDP4j2GNMKEU43Dpknnb+dA+vMv2jGF6ONvDjrZqrsU7aPtkilw8+Mki mx+EkBzW+a6ha4OE4hakwB0d+iAk0BrEALkjq1ZzMaIgQMRq1l2/MXjseXu+59f3HtWf7u01/QaE 7r4Hn9Sbu97T5iMw8FmbMCA8cp7DEEdW47nlsM8OSsLQ+m1NXKsW7vAI+PegPDxBIQYecdi3Bv9r lAxsF4/tASQsRuw37AVTm+2tQtIGLtwqJIl7KNRBsd1XLyTFcIwplBU6vhydnhyfsODxGgQ8iWMp XJr9WJ9O3/MOHjhLRpy5EeeKkWezKHmYhbpnUYwpdnF4OKI0Ig8tQCkcR2nG9eH49LRz2rUC3wLn zCMYhHAevl5zvzC2fLY9iWG2wWJ+xwB5VhXU5J7ZVVD7Pw+FOii2+6oGNcufBaEJizj8HpEyMx60 dhv7Dy2c0CihB3d81rXwwELsFhBwZ6xphElAcTy17uHYAmhqRSBm16mVRDycjXHCPkc4YH879+90 Kc5uR6mVdmpKWHSyBuwoD3Abb457PLq86nxun37tH7067XS7/dfH79vdr91e+0PqmUB6376T3eCe sr6fM4sywvGaFvu3WrQpiO3hr+Wb2cswvIi8jFsSh9b3HdZi56G1c2mWf7sh7PNd2Z9O7MEYeq3A pfe+39uZzQd8XzYl7Nx/aN3bmc8LfEc6Nezc/3H/Bw/w7Afj3fqebua7fRrZHhwEANk4HqadmH1l JyszsNozs0XxkhGCQERGmF63BGJ3xC6DucVGfXe//mi33mvsNnfrzW//dlExY5tr2VfOrlF/1Njv 1Z82njQb33J5rBBeQ7jLKD7p1ff3Gs0nFUIRhPWn9f16o1ev7+02H1UIF0ZE7JQADEhE7JQAjBuK 2CkBmFEsYqcEYPxAxE4JwIzOROyUAUwVfFeDCarp+gYwVMROCcD8FLJTAjAIi9gpAZioSvBWg4mr M2Y1GFIleKvBQCJipwRgSJXHrAZDDbqJJPNBDVGyGA77nANH2nQcxDGOiX0GYwTDXBYUBuF66PpI omSQgwXfPZlM1mmsMAY+EvYLRsPFQNgZ4gxT57fA8BeXyVULWevy0vBZR84gYpFjfmY4s7e8syvI uULMKTOteUjx2HKL2NX3BMqG4wFEoe3SpemCwov5S498SNITKKkNYBwG/BSaW3Fe9Y5OSoZk8XIp QSRgZkhCbeCye+jFdZXZIQ734JQUT9qcIZrY0EvYmxK+TUMWWVikIbkWCJzZ1hw0bg1LmSmN4Fx5 qoZ8XPsVRMtJh8M3sm0OC6BOvhAzv5wCd5yeMVG4MMbX3WUXFXPvpA5vNa4uNH4qzOYY5vg8cOGC HAdAnJBgnz2rBOwpSxJ7CeQ4Ug7pqqmURnqMk+Pck+qv876lG+/0osoClw0itlLscvQeW1e/BsKE rGXAaB4iFtQEwncvFsPZBCexC302HmgjmDp1JoAtH/bwMO/l5rO1qQG8nC5mqyrHAdEuWJFlODZh K60RAqMrd+oxDCGrdsszDS4bq83ikaBNrWghjNYaXE5g65pVj1k2788uuQlbapvrqcUlnn88/bm0 WTIeQ8+mySSAiN+RADp/FFZeIGn6PvGGS489soA8M0ts1nIEKB+enpTc8/Sitx+kfu0HOcFcznlT 7FLMHPHl9QvCswtrDkmniYuco4rHnMfMv0I8pJSCsVoNmsRpjSr/zgze3NurPRUsd8h6L4xnLac3 sCzypVK+ddL/XeFQZniySx0qlvJqHiqW8oofzGQpswrCXEKyyiHMJSSrLsJcQrIKJMwlJKtSwlxC skomDCZURep4M0UUBhOSVE1hLiFZZRXmEpJVX2EuIVmFFuYSklVxYS4hWaUX5hKSVYNhLiFZxRjm EpJVlaEUISXKM9Qjst06DfV4/P+iewkFG6ZiKbhyw0Rssko4lGEjY2W+5FoOo9gUWNRhJKf/re5Q DYqaZR7KUNKy3kM5etIKMXKcjVL9rVn4oRR4SRUPUipATAQjYkFxMqrWhCgFSZ/iEB2wrVvOkZPc umYVhiejGkBmuYhRYKTWjRhFpvgCEmVwqVZJIuagADCqlZSIOSgADDuejCs0K+f6YcVlFZfBr7gC swpMtFdxWcUFE7cCc+PdxIuKzQ2ZXwVmFRgULc/XLyoy/5v9zu8CMPuPreF4AD0PerzX2r4QEUx2 jeMgWDdtIgexHN84EmIpvXEYBDN44ziIJezGYRDMz43jIJ6Oy0FRrBLG35IbN/V6PbvXRpGXylo+ 8v9+0tYD56u9lyaoUQJWRelnlAydTLmMcqGTqo6hNboixDC0ByJb+0J7ILKlLrQHIlvZQnsgsoUs tAciW7dCfyBVUC1WlUJ/IJJFKLQHIltzQnsgsiUmtAciW1FCeyCyBSS0ByJbL0J7ILLlIbQHIlsN QnsgssUftgFEKa2HrQFQQ9pha8P//9p7iUoOmlPYkHCDxpRk6zRsGoWMkvqCZBl0RLEBFQadscgS XdgSA7U1FjYNRWtJhW3BkqZokONck+pPUEFhG5wl6QJIFUzQmIOIBbVAqC6HsA0m+qkfKERpXVWC nKDWNasOKxmF6kVoG+jIoRApAx1BbE65YNN0BEu11lsirgEHwVIt4zgIlygZR0KsRMk4DIIlSsZx ECtRMg6DYImScRzES5QKQiG1lEiB/xV40z+oqjIHm+agrKrBpkEIZsZyKhBz2itg2IKJsO7DVkSi YAvDFkv3dR+4WHav+6gFk3ndhy2Wu+s+asFUXfdhCyZyug9bNG/TfdyLdyME8teDthstrQt59fHo JCeHtN0YUsA/lBHATc21YfAix21I9iIrXU6AAGUvk/O8Z/3XktzUTR5zGtLKe9O2BqnarcbU4ZTd ExAQErC0JD3fvPKvxh+Pem9vaP793g77FSh206XuGdP7PxTEwpsNER7DtZmwf0IwuNb4zUnnQ9sM MnlTj+zwt51uz5zTQnTw7LtkAOsFtwKIKJKJbp2DKqnp1kHIStWk5arGEJGZvKoBJU9+pmQ2qw8+ JdLbYt+vXXuRR5inv7ufdV3IwbXhrM1pLWf5f39pBan5tM2kqA+WDVoRMoRlZihLj7DEDKUJE5rD ULZCoVlkZEoVmkVGpmahWWRkiheaRUamiqFZZGTKGRpGporAGxA4NIyMRKVDs8jIlDw0i4xM7UOz yMgUQTSLjEw1RLPIyJRFNIuMTH1Es8jIFEo0i4xMxcStk1FGOlENEn/aO5vdtIEoCu/7FKNsaKQg V+0uURatlG0bRd1HBkxBaQ3CJlHfvoaEov6gzhzOeO6cskskNPb9hjhzx/Z30jsUbXA4XpBHkikq 4ejBqqiCi6lXTMqEYdGL4FnMnklk4aIMH4Z50QIMuwrGpHSydTGaoEaTJHp8+6jHA6SMyYGTrIQ0 O6MKEGQEo0Qs+xqTw8lL3GgVV6hq0ZNY6LAGoTHMfWyVY/ZA6E7H7In0I3dMigl02ZCenzcIBLTc 6AKBRTC6SDBFjC4PUB6jCwTTyujyAIUzukBMSiL574AltkUmnWLwHe9I82sACPqyty4RcPHNfPs9 af3gWlum/kAxhmL9WGshQwDrJGTKBxsHmfqxPkGmfLAtkKkf7wI4CPpa9HPU8wYnEFzjc2bPQP3o kl4GAMtSRPE2SZBg+ZrSw/ARDZnzNP1z6JjYMtBqWuBj3K9pAZHvf+R4os2cKQgaNzO4tFjiZPwS YwkV+kd2ohHnkvMy8r667fi7CotfKjx8tNGqrMezLdnqsfq6WA7OL9zfT4K1qRk2hQHrutREsE1O WRzgpqcsD2wTVBYHuCkqywPfJJVFAm47yvJAtyFlgSR+sCB1+YmfK0hdfuBSnFt+d/DLotjXfqDu q+fyrl9qu9qVdb2tqb9bSAbmCusWOLOVHgDWHKhUD/YCKuVjS3+V6sGVvkr5+MJehQC4jlcpH122 q9TPumV+9MMDKiAYzw6YYJHdowPZUPPt6QOIhSc7pedl/HaoETyG74IaIeS7dIlz8zNzCKx7nuwt l9/H2/wIjek7Y92vPc4aVbLrFyR1dArcf8qNHQR3wkjJgjthpMTByWFkJsJJwmGFwknCYeXCScJh RcNJwmGlw0nCYQXEacI5XZDjx8RpwiElxUnCYYXFScJh5cVJwmFFxknCYaXGScJhBcdJwmFlx0nC YcXHScJhJchZgWMiRM4UjLQ5cqZQHB8GRoiSEyQSOU1OjBgrUM4CFkY+GDlTTgVLxFg5NUTHJssZ 4mEzXM4CoCzz5SyBo0W+eXwHqccLjJizwpwUqkZJmRNjgoxgF4rVrDkrfPKJmzNOLDQazhNa6LA2 uTESxZihcypMqLlzKlDiR89ZIAV6Kkjv+NhkAsorpJkEvmgRmQnt/YiE8UhGZhVzX0SaVxtUMCGG NBLQkiHNBFNnSCMBfRrSTHDJhjQW0LwhzQTVcUhDATuUuIqSfhGADYkSArD/4CCI2G70a9Xrfc6w 7oIzazYgYM2EEgGwd1BCgLUKSgTAzkAJAd4IKFEA1/1KCNBlvhIDlnAuknkvTxjx7HsJePi45DIw 8Bkl57snEECNZ+Hrl1mWJr4kiLKz8SWh5Lu0iWHki7FTQZHb/TzRqh3P4t9udetmc5Duc9/KetLB XTerYjSvi6p+dE9fqtYNW/fWDT+7d2/ccLgsm6a72g+n7dINb10xKduyWFblqqgXm2PuT9MN4p74 YPDq5u7u092l+zBvR+VD9UysWrmbzaCX7mPVPi1WD+/H3dDN69hnc+HO8kF3dr77lt1+b2eL2k3X 9biddz+Myqa6nyzupxuWbjqv582smhz69J8f/AFQSwECPwAUAAAACABiTcJUxAD2FDwFAAATQgAA DgAkAAAAAAAAACAAAAAAAAAAbG9nLmRvX2ZldGNoLjAKACAAAAAAAAEAGAD3aHoZInbYAd1RMlAu dtgB/Ph1GSJ22AFQSwECPwAUAAAACABiTcJUdVeGNOkPAAAUlAEADgAkAAAAAAAAACAAAABoBQAA bG9nLmRvX2ZldGNoLjEKACAAAAAAAAEAGABE+oUZInbYAQ0Ta0MudtgBB/p7GSJ22AFQSwUGAAAA AAIAAgDAAAAAfRUAAAAA