Bug 15335

Summary: AB-INT: sstatetests.SStateMirrors.test_cdn_mirror: Missing objects in the cache
Product: [QA/Testing] Functional (self) Testing Reporter: Alexandre Belloni <alexandre.belloni>
Component: oe-selftestAssignee: Michael Halstead <mhalstead>
Status: RESOLVED DUPLICATE QA Contact:
Severity: normal    
Priority: Medium+ CC: alex.kanavin, alexandre.belloni, joaomarcos.costa, mathieu.dubois-briand, randy.macleod, richard.purdie
Version: unspecified   
Target Milestone: 5.2 M4   
Hardware: x86   
OS: Multiple   
Whiteboard: AB-INT
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Alexandre Belloni 2024-01-05 15:14:27 UTC 
As discussed with RP, a new issue for this:

https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2350/steps/23/logs/stdio

AssertionError: 1 != 0 : Missing objects in the cache:
file://4a/14/sstate:go-runtime:core2-64-poky-linux:1.20.12:r0:core2-64:11:4a14706d4b273a0ce5eb61432a8f390e2e74db6d7016a7d0075d005a49e2d392_populate_sysroot.tar.zst


https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2349/steps/23/logs/stdio

AssertionError: 3 != 0 : Missing objects in the cache:
file://2e/2a/sstate:systemtap:core2-64-poky-linux:5.0:r0:core2-64:11:2e2aab245c75a994a5357dd062a43ba3c3069cbac89aab29149ee0763da29fef_package.tar.zst
file://0b/1c/sstate:python3:core2-64-poky-linux:3.11.5:r0:core2-64:11:0b1c7aed5aa7e0dbfad926102840981ff9aa98ec71a0175ab22907edecf69cf7_package.tar.zst
file://1f/88/sstate:libstd-rs:core2-64-poky-linux:1.74.1:r0:core2-64:11:1f88bc9a41a8f4089744b0f4c906d68e3c16c1bc1b586cd7656f9838d0b75cb3_package.tar.zst

AssertionError: 5 != 0 : Missing objects in the cache:
file://32/99/sstate:rust-llvm:cortexa57-poky-linux:1.74.1:r0:cortexa57:11:329909679fad51ece872cf3f4d40cebeb261f809261c98e5bb29bd1b7d099cec_package_write_rpm.tar.zst
file://8d/65/sstate:linux-yocto:qemuarm64-poky-linux:6.5.13+git:r0:qemuarm64:11:8d656e6b84b1d0899b56641da17ce668d6d9935185330d8373c9959b5cd42416_deploy.tar.zst
file://47/f4/sstate:go-runtime:cortexa57-poky-linux:1.20.12:r0:cortexa57:11:47f4fa34c45a09762988b351300781c1bf7860b58614f572bf496a66aa8bb0c0_populate_sysroot.tar.zst
file://universal/ae/20/sstate:go-cross-cortexa57:x86_64-poky-linux:1.20.12:r0:x86_64:11:ae20bed56c11a95cc0acca69ced8c9a50547a57008de8b344f28fabbd874f608_populate_sysroot.tar.zst
file://83/e1/sstate:libstd-rs:cortexa57-poky-linux:1.74.1:r0:cortexa57:11:83e1a6824cb85839f369dd5fb1777a702a2099425a7634527d7657de1eb8fa0e_package.tar.zst
Comment 1 Alexander Kanavin 2024-01-05 20:26:36 UTC 
A couple observations:

1. tests for the same objects in the local (AB NFS) cache all pass.

2. the missing objects are all large in size, typically 50-60Mb in size

The suspicion is it might be the same '504 Gateway Timeout' reported here:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15303#c11

However this time I couldn't reproduce it. It succeeds all the time. So it might be  a different error.

I suppose we need better diagnostics on failure still, particularly the full wget output for the objects that couldn't be obtained.
Comment 2 Alexander Kanavin 2024-01-08 14:48:59 UTC 
Patches to improve the diagnostics:
https://patchwork.yoctoproject.org/project/bitbake/list/?series=21056
https://patchwork.yoctoproject.org/project/oe-core/list/?series=21057

If this happens again, we should have a better idea what the problem is.
Comment 3 Alexandre Belloni 2024-01-09 21:46:37 UTC 
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6433/steps/23/logs/stdio

This one has the debug you added.
Comment 4 Alexander Kanavin 2024-01-09 22:08:45 UTC 
AssertionError: 2 != 0 : Missing objects in the cache:
file://2b/e5/sstate:perl:cortexa57-poky-linux:5.38.2:r0:cortexa57:11:2be55a74f3b1b5787a1ce659d298cf398634ce7e55421f89f8d13cc55d6d77e2_package_write_rpm.tar.zst
file://universal/50/17/sstate:go-binary-native:x86_64-linux:1.20.12:r0:x86_64:11:50175dd8f117402be7f0c0319520da633dc936bea838e6260972276cd252e056_populate_sysroot.tar.zst

DEBUG: checkstatus() urlopen failed for http://cdn.jsdelivr.net/yocto/sstate/all/2b/e5/sstate:perl:cortexa57-poky-linux:5.38.2:r0:cortexa57:11:2be55a74f3b1b5787a1ce659d298cf398634ce7e55421f89f8d13cc55d6d77e2_package_write_rpm.tar.zst: HTTP Error 504: Gateway Timeout
DEBUG: checkstatus() urlopen failed for http://cdn.jsdelivr.net/yocto/sstate/all/universal/50/17/sstate:go-binary-native:x86_64-linux:1.20.12:r0:x86_64:11:50175dd8f117402be7f0c0319520da633dc936bea838e6260972276cd252e056_populate_sysroot.tar.zst: HTTP Error 504: Gateway Timeout

'HTTP Error 504: Gateway Timeout' for both missing objects, so reassigning to Michael.
Comment 16 Randy MacLeod 2024-03-28 15:03:26 UTC 
Alex, 
Can we work-around the CDN limitation by running the test twice.
On the first time, ignore the error.
The second test should always pass since the first test is slow enough to allow the CDN to get caught up.

The new autobuilder should have faster connections so the problem may not persist.
Comment 17 Alexander Kanavin 2024-03-28 17:04:56 UTC 
Patch:
https://patchwork.yoctoproject.org/project/oe-core/patch/20240328170200.981712-1-alex@linutronix.de/

I made it quickly without testing, as I'm about to leave on holiday until 9th April. But it should work :)
Comment 18 Richard Purdie 2024-03-30 22:31:12 UTC 
Worked around for now, thanks Alex:
https://git.yoctoproject.org/poky/commit/?id=d1622b87f8cd9244e503a0a8d9cd056638499525
Hence deferring to 5.1 when we can try and resolve the infrastructure issues with the data centre/autobuilder changes
Comment 25 Alexander Kanavin 2024-06-17 13:19:22 UTC 
I checked the links in the last three messages: the errors are a mix of
- '404 not found' on CDN (where the item is present on local cache, so didn't propagate to CDN properly?)
- 'read operation timed out' errors
Comment 26 Alexander Kanavin 2024-06-17 13:22:13 UTC 
Went slightly further, there's a sporadic 'HTTP Error 504: Gateway Timeout' too, which means the fix of running the test twice didn't quite do the trick.
Comment 27 Randy MacLeod 2024-06-27 14:57:47 UTC 
The retry seemed to be effective for a month or two but now it's not working.
We're not sure why but suspect that since there are multiple nodes in the CDN, you might get a different node on retry. The CDN folks may have added more servers to deal with load. Wild speculation but might be useful...
-- YP bug review.
Comment 44 Mathieu Dubois-Briand 2024-09-25 14:48:20 UTC 
a-quick debian12-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2372/steps/23/logs/stdio
a-quick ubuntu2204-ty-3
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2387/steps/23/logs/stdio
a-quick debian11-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2431/steps/23/logs/stdio
a-quick debian12-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2437/steps/23/logs/stdio
a-quick fedora38-ty-3
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2438/steps/23/logs/stdio
a-quick debian11-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2445/steps/23/logs/stdio
a-quick opensuse154-ty-2
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2456/steps/25/logs/stdio
a-quick opensuse154-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2457/steps/25/logs/stdio
a-quick fedora39-ty-2
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2467/steps/23/logs/stdio
a-quick opensuse154-ty-2
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2469/steps/25/logs/stdio
a-quick stream8-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2470/steps/25/logs/stdio
a-quick debian11-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2471/steps/23/logs/stdio
a-quick ubuntu2004-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2474/steps/23/logs/stdio
a-quick alma9-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2477/steps/23/logs/stdio
a-quick opensuse154-ty-3
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2478/steps/25/logs/stdio
a-quick alma8-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2480/steps/25/logs/stdio
a-quick debian11-ty-3
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2481/steps/23/logs/stdio
a-quick debian11-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2492/steps/23/logs/stdio
a-quick alma9-ty-2
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2513/steps/23/logs/stdio
a-quick alma8-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2521/steps/25/logs/stdio
a-quick debian11-ty-3
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2576/steps/23/logs/stdio
a-quick opensuse154-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2577/steps/25/logs/stdio
Comment 46 Mathieu Dubois-Briand 2024-10-11 07:21:51 UTC 
a-quick debian12-vk-6
https://valkyrie.yoctoproject.org/#/builders/62/builds/13/steps/23/logs/stdio
Comment 48 Mathieu Dubois-Briand 2024-10-16 11:53:29 UTC 
a-quick fedora40-vk-2
https://valkyrie.yoctoproject.org/#/builders/62/builds/17/steps/23/logs/stdio
Comment 49 Mathieu Dubois-Briand 2024-10-17 14:23:23 UTC 
a-quick fedora41-vk-1
https://valkyrie.yoctoproject.org/#/builders/62/builds/18/steps/23/logs/stdio
Comment 51 Mathieu Dubois-Briand 2024-10-21 07:51:25 UTC 
a-full debian12-vk-8
https://valkyrie.yoctoproject.org/#/builders/29/builds/218/steps/23/logs/stdio

a-full opensuse155-vk-1
https://valkyrie.yoctoproject.org/#/builders/29/builds/219/steps/25/logs/stdio

a-full opensuse156-vk-1
https://valkyrie.yoctoproject.org/#/builders/29/builds/221/steps/25/logs/stdio

a-full debian11-vk-1
https://valkyrie.yoctoproject.org/#/builders/29/builds/222/steps/23/logs/stdio

a-full debian12-vk-7
https://valkyrie.yoctoproject.org/#/builders/29/builds/226/steps/23/logs/stdio

a-full fedora39-vk-2
https://valkyrie.yoctoproject.org/#/builders/29/builds/228/steps/23/logs/stdio

a-full debian12-vk-9
https://valkyrie.yoctoproject.org/#/builders/29/builds/239/steps/23/logs/stdio

a-quick fedora40-vk-3
https://valkyrie.yoctoproject.org/#/builders/62/builds/12/steps/23/logs/stdio

a-full ubuntu2410-vk-1
https://valkyrie.yoctoproject.org/#/builders/29/builds/244/steps/23/logs/stdio

a-full ubuntu2004-vk-1
https://valkyrie.yoctoproject.org/#/builders/29/builds/249/steps/23/logs/stdio

a-full debian12-vk-8
https://valkyrie.yoctoproject.org/#/builders/29/builds/252/steps/23/logs/stdio

a-full fedora39-vk-1
https://valkyrie.yoctoproject.org/#/builders/29/builds/273/steps/23/logs/stdio

a-full ubuntu2410-vk-1
https://valkyrie.yoctoproject.org/#/builders/29/builds/275/steps/23/logs/stdio

a-full debian12-vk-1
https://valkyrie.yoctoproject.org/#/builders/29/builds/278/steps/23/logs/stdio

a-full ubuntu2204-vk-4
https://valkyrie.yoctoproject.org/#/builders/29/builds/281/steps/23/logs/stdio

a-full debian12-vk-1
https://valkyrie.yoctoproject.org/#/builders/29/builds/284/steps/23/logs/stdio

a-full debian12-vk-1
https://valkyrie.yoctoproject.org/#/builders/29/builds/287/steps/23/logs/stdio

a-quick debian11-vk-2
https://valkyrie.yoctoproject.org/#/builders/62/builds/19/steps/23/logs/stdio

a-full fedora38-ty-6
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7321/steps/23/logs/stdio

a-full debian12-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7329/steps/23/logs/stdio
Comment 52 Mathieu Dubois-Briand 2024-10-21 07:51:43 UTC 
a-full debian12-vk-4
https://valkyrie.yoctoproject.org/#/builders/29/builds/243/steps/23/logs/stdio

a-full debian12-vk-1
https://valkyrie.yoctoproject.org/#/builders/29/builds/247/steps/23/logs/stdio

a-full debian12-vk-3
https://valkyrie.yoctoproject.org/#/builders/29/builds/248/steps/23/logs/stdio

a-full alma9-vk-2
https://valkyrie.yoctoproject.org/#/builders/29/builds/256/steps/23/logs/stdio

a-full rocky9-vk-3
https://valkyrie.yoctoproject.org/#/builders/29/builds/259/steps/23/logs/stdio

a-full debian11-vk-3
https://valkyrie.yoctoproject.org/#/builders/29/builds/262/steps/23/logs/stdio

a-full debian12-vk-8
https://valkyrie.yoctoproject.org/#/builders/29/builds/263/steps/23/logs/stdio

a-full ubuntu2004-vk-2
https://valkyrie.yoctoproject.org/#/builders/29/builds/264/steps/23/logs/stdio

a-full debian12-vk-7
https://valkyrie.yoctoproject.org/#/builders/29/builds/266/steps/23/logs/stdio

a-full rocky9-vk-3
https://valkyrie.yoctoproject.org/#/builders/29/builds/267/steps/23/logs/stdio

a-full alma9-vk-2
https://valkyrie.yoctoproject.org/#/builders/29/builds/269/steps/23/logs/stdio

a-full debian11-vk-3
https://valkyrie.yoctoproject.org/#/builders/29/builds/270/steps/23/logs/stdio

a-full rocky9-vk-3
https://valkyrie.yoctoproject.org/#/builders/29/builds/272/steps/23/logs/stdio

oe-selftest-ubuntu ubuntu2004-vk-3
https://valkyrie.yoctoproject.org/#/builders/54/builds/255/steps/11/logs/stdio

oe-selftest-debian debian12-vk-8
https://valkyrie.yoctoproject.org/#/builders/35/builds/252/steps/11/logs/stdio

a-full debian12-vk-3
https://valkyrie.yoctoproject.org/#/builders/29/builds/285/steps/23/logs/stdio

oe-selftest-armhost ubuntu2204-vk-arm2
https://valkyrie.yoctoproject.org/#/builders/23/builds/281/steps/11/logs/stdio

oe-selftest-fedora fedora40-vk-3
https://valkyrie.yoctoproject.org/#/builders/48/builds/211/steps/14/logs/stdio

a-full debian12-vk-8
https://valkyrie.yoctoproject.org/#/builders/29/builds/288/steps/23/logs/stdio

oe-selftest-armhost ubuntu2404-vk-arm1
https://valkyrie.yoctoproject.org/#/builders/23/builds/283/steps/11/logs/stdio

oe-selftest-fedora fedora40-vk-3
https://valkyrie.yoctoproject.org/#/builders/48/builds/213/steps/11/logs/stdio

a-full ubuntu2404-vk-1
https://valkyrie.yoctoproject.org/#/builders/29/builds/289/steps/23/logs/stdio

oe-selftest-fedora fedora40-vk-3
https://valkyrie.yoctoproject.org/#/builders/48/builds/214/steps/11/logs/stdio

oe-selftest-ubuntu ubuntu2404-vk-1
https://valkyrie.yoctoproject.org/#/builders/54/builds/260/steps/11/logs/stdio

oe-selftest-debian debian12-vk-2
https://valkyrie.yoctoproject.org/#/builders/35/builds/257/steps/11/logs/stdio

qemuarm64 debian12-vk-9
https://valkyrie.yoctoproject.org/#/builders/36/builds/263/steps/13/logs/stdio

oe-selftest-centos rocky8-vk-1
https://valkyrie.yoctoproject.org/#/builders/76/builds/265/steps/12/logs/stdio

oe-selftest-armhost ubuntu2004-vk-arm1
https://valkyrie.yoctoproject.org/#/builders/23/builds/287/steps/11/logs/stdio

oe-selftest-centos rocky9-vk-1
https://valkyrie.yoctoproject.org/#/builders/76/builds/267/steps/14/logs/stdio

a-full rocky9-vk-1
https://valkyrie.yoctoproject.org/#/builders/29/builds/297/steps/23/logs/stdio

oe-selftest-armhost ubuntu2410-vk-arm1
https://valkyrie.yoctoproject.org/#/builders/23/builds/290/steps/14/logs/stdio

oe-selftest-centos stream9-vk-1
https://valkyrie.yoctoproject.org/#/builders/76/builds/270/steps/11/logs/stdio

oe-selftest-ubuntu ubuntu2204-vk-3
https://valkyrie.yoctoproject.org/#/builders/54/builds/265/steps/11/logs/stdio

oe-selftest-debian debian11-vk-3
https://valkyrie.yoctoproject.org/#/builders/35/builds/266/steps/14/logs/stdio

a-quick ubuntu2004-vk-2
https://valkyrie.yoctoproject.org/#/builders/62/builds/20/steps/23/logs/stdio

oe-selftest-armhost ubuntu2404-vk-arm1
https://valkyrie.yoctoproject.org/#/builders/23/builds/291/steps/11/logs/stdio

oe-selftest ubuntu2004-vk-3
https://valkyrie.yoctoproject.org/#/builders/69/builds/22/steps/11/logs/stdio

oe-selftest-debian debian11-vk-3
https://valkyrie.yoctoproject.org/#/builders/35/builds/270/steps/11/logs/stdio

a-full debian12-vk-2
https://valkyrie.yoctoproject.org/#/builders/29/builds/298/steps/23/logs/stdio
Comment 55 Richard Purdie 2024-10-28 23:26:28 UTC 
We're starting to see green CDN tests now!
Comment 56 Mathieu Dubois-Briand 2024-11-05 11:06:41 UTC 
a-full rocky9-vk-3
https://valkyrie.yoctoproject.org/#/builders/29/builds/363/steps/23/logs/stdio
Comment 57 Mathieu Dubois-Briand 2024-11-13 07:04:59 UTC 
a-full stream9-vk-1 mathieu/master-next completed at 2024-11-13T01:11:26Z
https://valkyrie.yoctoproject.org/#/builders/29/builds/425/steps/23/logs/stdio
Comment 58 Randy MacLeod 2025-04-10 14:53:53 UTC 
Dup

*** This bug has been marked as a duplicate of bug 15658 ***