Bug 11265

Summary: buildgalculator.GalculatorTest tries to download galculator-2.1.4.tar.bz2 despite it being present in DL_DIR
Product: [QA/Testing] Runtime Testing Reporter: Joshua Lock <joshuagloe>
Component: generalAssignee: Joshua Lock <joshuagloe>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium CC: anibal.limon, benjamin.esquivel, mariano.lopez
Version: 2.3   
Target Milestone: 2.4   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
Copy of the logs from AB's stdout monitor none

Description Joshua Lock 2017-03-29 12:30:40 UTC 
Created attachment 3678 [details]
Copy of the logs from AB's stdout monitor

On the new Autobuilder cluster we saw a failure (log snippet below and copy of log attached) when running testimage for nightly-x86-lsb.  

The buildgalculator.GalculatorTest test is trying to fetch the galculator-2.1.4.tar.bz2 tarball to /tmp despite it being present in DL_DIR.

[joshuagl@fedora24 ~]$ ls /srv/autobuilder/autobuilder.yoctoproject.org/current_sources/galculator-2.1.4.tar.bz2 -la
-rw-r--r--. 1 pokybuild users 472989 Sep  5  2015 /srv/autobuilder/autobuilder.yoctoproject.org/current_sources/galculator-2.1.4.tar.bz2

We can see from the stdout of the builder (attached) that:

DL_DIR=/srv/autobuilder/autobuilder.yoctoproject.org/current_sources

therefore we'd expect oeqa.utils.buildproject._download_archive() to copy the file from DL_DIR, but clearly this isn't happening.

Note: This is related to but not a duplicate of #11115

Log on AB (transient link): https://autobuilder.yocto.io/builders/nightly-x86-lsb/builds/222/steps/Running%20Sanity%20Tests/logs/stdio

8<------------ Log snippet --------------------------------------------------

NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: 
NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: ======================================================================
NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: ERROR: setUpClass (buildgalculator.GalculatorTest)
NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: ----------------------------------------------------------------------
NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: Traceback (most recent call last):
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-lsb/build/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass
    clss.setUpClassMethod()
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-lsb/build/meta/lib/oeqa/runtime/cases/buildgalculator.py", line 16, in setUpClass
    cls.project.download_archive()
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-lsb/build/meta/lib/oeqa/runtime/utils/targetbuildproject.py", line 17, in download_archive
    status, output = self.target.copyTo(self.localarchive, self.targetdir)
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-lsb/build/meta/lib/oeqa/core/target/ssh.py", line 103, in copyTo
    return self._run(scpCmd, ignore_status=False)
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-lsb/build/meta/lib/oeqa/core/target/ssh.py", line 61, in _run
    "status %d:\n%s" % (command, status, output))
AssertionError: Command '['scp', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-o', 'LogLevel=ERROR', '/tmp/galculator-2.1.4.tar.bz2', 'root@192.168.7.10:~/']' returned non-zero exit status 1:
/tmp/galculator-2.1.4.tar.bz2: No such file or directory
NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: ----------------------------------------------------------------------
NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: Ran 47 tests in 97.868s
NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: 
NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: FAILED (errors=1, skipped=1)
NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: Stopping logging thread
NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: Stop event received
NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: Tearing down logging thread
NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: Sending SIGTERM to runqemu
core-image-lsb-sdk-1.0-r0 do_testimage: SUMMARY:
core-image-lsb-sdk-1.0-r0 do_testimage: core-image-lsb-sdk () - Ran 47 tests in 97.870s
core-image-lsb-sdk-1.0-r0 do_testimage: core-image-lsb-sdk - FAIL - Required tests failed (skipped=1)
core-image-lsb-sdk-1.0-r0 do_testimage: RESULTS:
core-image-lsb-sdk-1.0-r0 do_testimage: RESULTS - buildcpio.BuildCpioTest.test_cpio - Testcase 205: PASSED
core-image-lsb-sdk-1.0-r0 do_testimage: RESULTS - buildgalculator.GalculatorTest.test_galculator - Testcase 1526: ERROR
core-image-lsb-sdk-1.0-r0 do_testimage: Traceback (most recent call last):
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-lsb/build/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass
    clss.setUpClassMethod()
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-lsb/build/meta/lib/oeqa/runtime/cases/buildgalculator.py", line 16, in setUpClass
    cls.project.download_archive()
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-lsb/build/meta/lib/oeqa/runtime/utils/targetbuildproject.py", line 17, in download_archive
    status, output = self.target.copyTo(self.localarchive, self.targetdir)
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-lsb/build/meta/lib/oeqa/core/target/ssh.py", line 103, in copyTo
    return self._run(scpCmd, ignore_status=False)
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-lsb/build/meta/lib/oeqa/core/target/ssh.py", line 61, in _run
    "status %d:\n%s" % (command, status, output))
AssertionError: Command '['scp', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-o', 'LogLevel=ERROR', '/tmp/galculator-2.1.4.tar.bz2', 'root@192.168.7.10:~/']' returned non-zero exit status 1:
/tmp/galculator-2.1.4.tar.bz2: No such file or directory
Comment 1 Richard Purdie 2017-03-29 15:19:15 UTC 
I just tried this on the autobuilder with:
diff --git a/meta/lib/oeqa/utils/buildproject.py b/meta/lib/oeqa/utils/buildproject.py
index fc8879c..b1b78ae 100644
--- a/meta/lib/oeqa/utils/buildproject.py
+++ b/meta/lib/oeqa/utils/buildproject.py
@@ -25,9 +25,11 @@ class BuildProject(metaclass=ABCMeta):
     # Download self.archive to self.localarchive
     def _download_archive(self):
         if self.dl_dir and os.path.exists(os.path.join(self.dl_dir, self.archive)):
+            print("here 1")
             shutil.copyfile(os.path.join(self.dl_dir, self.archive), self.localarchive)
             return
 
+        print("here 2")
         cmd = "wget -O %s %s" % (self.localarchive, self.uri)
         subprocess.check_output(cmd, shell=True)

and it clearly prints "here 1" so the code path is executing. 

The problem is the /tmp/ prefix which it copies the file to, we need to use a path within workdir or a proper temp directory.
Comment 2 Joshua Lock 2017-03-29 20:42:28 UTC 
GalculatorTest creates a project object which is an instance of SDKBuildProject with the first, testpath, parameter set to self.tc.sdk_dir + "/galculator/".

When the super oeqa.utils.BuildProject constructor is called it's passed testpath to its named tmpdir parameter, and it's here that the _download_archive() method tries to copy or fetch the tarball to.
Comment 3 Joshua Lock 2017-04-05 12:12:20 UTC 
(In reply to comment #2)
> GalculatorTest creates a project object which is an instance of
> SDKBuildProject with the first, testpath, parameter set to self.tc.sdk_dir +
> "/galculator/".
> 
> When the super oeqa.utils.BuildProject constructor is called it's passed
> testpath to its named tmpdir parameter, and it's here that the
> _download_archive() method tries to copy or fetch the tarball to.

I am stupid and was looking at meta/lib/oeqa/sdk/buildgalculator.py not meta/lib/oeqa/runtime/cases/buildgalculator.py.

I've submitted some patches to remove the hard-coding of /tmp:

http://lists.openembedded.org/pipermail/openembedded-core/2017-April/135194.html
https://patchwork.openembedded.org/series/6163/#