Bug 11501 - current pyro branch: unfs3 Checksum failure fetching
Summary: current pyro branch: unfs3 Checksum failure fetching
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: devtools / tool chain (show other bugs)
Version: 2.3
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 2.3.1
Assignee: Choong Yin Thong
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2017-05-11 18:13 UTC by Robert Berger
Modified: 2017-07-13 01:24 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Robert Berger 2017-05-11 18:13:19 UTC
ERROR: nativesdk-unfs3-0.9.22.r497-r0 do_fetch: Checksum failure fetching http://downloads.yoctoproject.org/mirror/sources/trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz
ERROR: nativesdk-unfs3-0.9.22.r497-r0 do_fetch: Fetcher failure for URL: 'http://downloads.yoctoproject.org/mirror/sources/trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz'. Checksum mismatch!
File: '/tmp/yocto-autobuilder/downloads/trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz' has md5 checksum db06a7cdb1c0752d12f45be4e5520d5d when 2e43e471c77ade0331901c40b8f8e9a3 was expected
File: '/tmp/yocto-autobuilder/downloads/trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz' has sha256 checksum 0872a96910d1df779390f74375cb50cd2df1c5d9d1ffb016aff596384fdff409 when 21009468a9ba07b72ea93780d025a63ab4e55bf8fc3127803c296f0900fe1bac was expected
If this change is expected (e.g. you have upgraded to a new version without updating the checksums) then you can use these lines within the recipe:
SRC_URI[md5sum] = "db06a7cdb1c0752d12f45be4e5520d5d"
SRC_URI[sha256sum] = "0872a96910d1df779390f74375cb50cd2df1c5d9d1ffb016aff596384fdff409"
Otherwise you should retry the download and/or check with upstream to determine if the file has become corrupted or otherwise unexpectedly modified.

Proposed fix:

https://github.com/RobertBerger/poky/commit/08e3ebf970efbb538e666714bd144c07579f3ab4
Comment 1 Choong Yin Thong 2017-05-18 02:39:59 UTC
Cannot reproduce the issue. Tested with bitbake nufs3 and nativesdk-nufs3. Both successful fetch and build. Try to clean before bitbake
bitbake nufs3 -c cleanall
bitbake nativesdk-nufs3 -c cleanall
Comment 2 Robert Berger 2017-05-18 04:15:08 UTC
A quick test on my end shows, that it builds with my patched version.

Version: 0.9.22.r497-r0

bitbake -c cleanall unfs3
bitbake unfs3

bitbake -c cleanall nativesdk-unfs3
bitbake nativesdk-unfs3

But I just did this:

wget http://downloads.yoctoproject.org/mirror/sources/trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz

md5sum trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz
2e43e471c77ade0331901c40b8f8e9a3  trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz

sha256sum trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz
21009468a9ba07b72ea93780d025a63ab4e55bf8fc3127803c296f0900fe1bac  trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz

So it looks like your original checksums are OK.

No idea where I get the different checksum from.

Will further investigate, but this worries me.

Maybe somewhere in my SSTATE or DL caching server?
Comment 3 Robert Berger 2017-05-18 12:32:51 UTC
I found the version with a different checksum on my source caching server.

md5sum trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz-from-source-cache trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz-from-wget
db06a7cdb1c0752d12f45be4e5520d5d  trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz-from-source-cache
2e43e471c77ade0331901c40b8f8e9a3  trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz-from-wget


sha256sum trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz-from-source-cache trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz-from-wget
0872a96910d1df779390f74375cb50cd2df1c5d9d1ffb016aff596384fdff409  trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz-from-source-cache
21009468a9ba07b72ea93780d025a63ab4e55bf8fc3127803c296f0900fe1bac  trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz-from-wget

I extracted the 2 tarballs and compared the results.

diff -r from-source-cache from-wget

they are identical!

A possible explanation would be, that morty fetched from:

svn://svn.code.sf.net/p/unfs3/code;module=trunk;rev=${MOD_PV};protocol=http

SRC_URI[md5sum] = "3687acc4ee992e536472365dd99712a7"
SRC_URI[sha256sum] = "274b43ada9c6eea1da26eb7010d72889c5278984ba0b50dff4e093057d4d64f8"

see:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-devtools/unfs3/unfs3_0.9.22.r497.bb?h=morty

while pyro fetches from:

http://downloads.yoctoproject.org/mirror/sources/trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz

SRC_URI[md5sum] = "2e43e471c77ade0331901c40b8f8e9a3"
SRC_URI[sha256sum] = "21009468a9ba07b72ea93780d025a63ab4e55bf8fc3127803c296f0900fe1bac"

see:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-devtools/unfs3/unfs3_0.9.22.r497.bb?h=pyro

First conclusion:

It looks like on the same tarball with different checksums exists. I suggest that instead of a tarball a subversion checkout should being used.

---

On further investigation I saw that it's fetched from

http://downloads.yoctoproject.org/mirror/sources/trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz

and not from upstream

so I tried:
svn co svn://svn.code.sf.net/p/unfs3/code;module=trunk;rev=497;protocol=http

And it turns out, that the tarball on the yoctoproject mirror is not the same with what's under subversion:

diff -r from-wget/ code/trunk
Only in code/trunk: aclocal.m4
Only in code/trunk: afsgettimes.c
Only in code/trunk: afssupport.c
Only in code/trunk: afssupport.h
Only in code/trunk: attr.c
Only in code/trunk: attr.h
Only in code/trunk: backend.h
Only in code/trunk: backend_unix.h
Only in code/trunk: backend_win32.h
Only in code/trunk: bootstrap
Only in code/trunk: BRANCHES
Only in code/trunk: Config
Only in code/trunk: configure.ac
Only in code/trunk: contrib
Only in code/trunk: CREDITS
Only in code/trunk: daemon.c
Only in code/trunk: daemon.h
Only in code/trunk: doc
Only in code/trunk: error.c
Only in code/trunk: error.h
Only in code/trunk: Extras
Only in code/trunk: fd_cache.c
Only in code/trunk: fd_cache.h
Only in code/trunk: fh.c
Only in code/trunk: fh_cache.c
Only in code/trunk: fh_cache.h
Only in code/trunk: fh.h
Only in code/trunk: indent-all
Only in code/trunk: install-sh
Only in code/trunk: LICENSE
Only in code/trunk: locate.c
Only in code/trunk: locate.h
Only in code/trunk: Makefile.in
Only in code/trunk: md5.c
Only in code/trunk: md5.h
Only in code/trunk: mount.c
Only in code/trunk: mount.h
Only in code/trunk: NEWS
Only in code/trunk: nfs.c
Only in code/trunk: nfs.h
Only in code/trunk: password.c
Only in code/trunk: password.h
Only in code/trunk: readdir.c
Only in code/trunk: readdir.h
Only in code/trunk: README
Only in code/trunk: README.nfsroot
Only in from-wget/: trunk
Only in code/trunk: unfs3.spec
Only in code/trunk: unfsd.8
Only in code/trunk: unfsd.init
Only in code/trunk: user.c
Only in code/trunk: user.h
Only in code/trunk: winerrno.h
Only in code/trunk: winsupport.c
Only in code/trunk: winsupport.h
Only in code/trunk: xdr.c
Only in code/trunk: xdr.h

I am a bit confused as how this tarball was created.

Can you please clarify?
Comment 4 Paul Eggleton 2017-05-18 22:14:54 UTC
FYI the reason we switched to a tarball snapshot is this was the last remaining subversion-native dependency in a normal build for qemu machines, and for something that changes relatively infrequently it seemed superfluous.

I suspect the issue here is if you'd already generated the tarball locally, the new one (generated on a different machine) wouldn't have the same checksums. However, I can't explain why the content is different to the repository. I would have assumed it was just a mirror tarball that bitbake created after fetching from svn at revision 497.
Comment 5 Choong Yin Thong 2017-05-22 07:01:08 UTC
I tested 2 source both are identical.
wget http://downloads.yoctoproject.org/mirror/sources/trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz

This link will download as trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz
We use tar zxvf trunk_svn.code.sf.net_.p.unfs3.code_497_.tar.gz to untar it and the folder name is trunk/.

Continue download with the svn source:-
svn co svn://svn.code.sf.net/p/unfs3/code;module=trunk;rev=497;protocol=http

The downloaded source will be name code/ folder. Now we got folder trunk/ and folder code/. We only wish to compare folder trunk/ because the main source is only in trunk folder. Therefore, the diff command as below:-
diff -r code/trunk/ trunk/ - (1st is code from svn, 2nd is code from yoctoproject download link)

And result both folder content is identical. Please verify the diff path. Only compare with the trunk/ folder.
Comment 6 Robert Berger 2017-05-22 19:27:41 UTC
I confirm it's the same.

Somehow one of my yocto builds created a tarball with a different checksum, but the same content and my scripts copied this over to my SOURCE_MIRROR server.

After I removed the file with the other checksum from my SOURCE_MIRROR server all is fine.
Comment 7 Paul Eggleton 2017-05-22 20:33:19 UTC
Well, it's not quite fixed - we should still rename the tarball and update the recipe to match so that anyone else with their own source mirror tarball doesn't hit this.
Comment 8 Robert Berger 2017-07-12 19:45:22 UTC
As Paul suggests I reopen it.
Comment 9 Choong Yin Thong 2017-07-13 01:24:59 UTC
(In reply to comment #8)
> As Paul suggests I reopen it.

I already change the SRC_URI with updated source without SVN to avoid confuse as suggested by Paul.

https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=4063637e687f388131914c4f85cadb71e9048bb2