Bug 7850

Hi Cathy,

For deploying files to the DUT we use two methods currently. 

1. target.copy_to()/target.copy_from() methods (lib/oeqa/utils/sshcontrol.py)
With these methods you can easily copy files to and from the DUT.

2. For bulk files you can create an http server with a file repository. (oeqa.utils.httpserver)

Hi Musat, the problem is not about how to push binaries to target, but how test cases know where the binaries are. For every image build, the test binaries will be built as well, and located at some place like build/tmp-glibc/work/i586-oe-linux/... for x86, we need a way to get the binaries for both testimage and exported test.

I think there are two parts to this problem:

a) We need to markup the tests so we know which binaries they need to run the tests. The export code can then use this markup to ensure the binaries needed are included

b) For native tools, we have two options, either we build a tools tarball (like buildtools-tarball) which contains the needed tooling, or we ship the sstate for the relevant pieces and install that. The buildtools approach is probably going to be easier/faster.

In testing, some PyUnit test cases reply on binary files which are also built by yocto. More complicated is, some binary are native for host and some are for target platform. The  problem is how to locate the path of binary file from PyUnit TC. Currently, we have a temporary solution as below
In meta-iotqa, case developer need provide a path relative to output dir(e.g. "mraa-test/1.0-r0/hello_mraa"), then testrunner copy all these files to a specific folder, case need call a specific function to get this folder.
For native files, need add a prefix "native:", and test cases call another function to get native folder.
for example:
# lib/files.manifest:
target-test/1.0-r0/target-binary
native:native-test/1.0-r0/native-binary

#case.py
import os
from oeqa.oetest import oeRuntimeTest
from oeqa.utils.helper import get_files_dir, get_native_dir
class TestCase(oeRuntimeTest):
    def test_case(self):
		...
        (status,output) = self.target.copy_to(os.path.join(get_files_dir(),
                          ' target-binary'), "/opt/ ")
        ...
		os.system("%s/native-binary" % get_native_dir())
		...

In this solution, need case developer provide their files path to a file(meta-iotqa/lib/files.manifest). It's not friendly way for a case developer. 
Is here a better way for test runner to get their information automatically?

We had a better solution to handle binary files:
Runner provide a task to copy these files, and add this task after do_compile,
When a case developer need this feature, he need inherit this bbclass and append his binary file path to some variables,
then copy task will copy chem to the specific folder when he build his package.
In this solution, we don't need files.manifest any more.
for example:
#target-test.bb:
...
inherit copybin
TARGET_FILES += "${WORKDIR}/target-binary"
#native-test.bb:
...
inherit copybin
NATIVE_FILES += "${WORKDIR}/native-binary"
#case.py don't need change

Hi,

From what i understood the idea is to follow these steps:

1) build an image and deploy it to an embedded board
2) build an app. for the board that will serve as test app. (it may be used to test an API provided by a library existent on the image already, eg. make an app. for libmraa)

*** step 1 and two may be corroborated into one step
3) write a test case that will be handled by testimage. This test case will deploy the app. to the board and will launch via ssh the app. with different options. testimage will provide means to selective launch test cases.

This is a simplified version of the workflow. If this is the scheme you think of, a good example of such an approach is rpm.py under lib/oeqa/runtime.

Please note that for this to run, before doing the bitbake -c testimage we build rpm (alongside psplash and package-index:  bitbake rpm psplash && bitbake package-index). I'm saying this because a task like do_compile cannot be executed from inside testimage task, so binaries must be compiled before launching testimage

Hi Costin, pls. try the code and steps Junhui shared to you. Let's start with current solution and fix the unreasonable parts if there is any.

But let's align on below requirements:
*) Some TC reply on test binary files which should be built before execution.
*) There are 2 kinds of test binary files, one is for DUT, one is for Host. For example, Iotivity test client and server (client is on host side and server is on DUT side).
*) Test Binary should not be part of image when build it as QA image, because test harness should support both QA image from local build and Released image from build system.
*) Test Case which reply on test binary file should be workable on both build testing (testimage) and export testing (runexported).
*) Handle different HWs with diff arch, like i586, x86_64, arm, etc.
*) PLs. keep in minder: rpm/ipk are not always supported on target devices. That's why tar ball is much appreciated in this case.

Hi,

This is a summary of the last days chat and tests done on the Wang Jing and Junhui Zhang code, proposed by them to solve this feature.

The path proposed by them (which i think is a good approach) is:

1) for testing purposes a new meta-qa layer was created
2) create recipes for binaries that will be tested against the resulting BSP image, both for the target machine and for host machine (native pkgs.) that can be built during image creation by extending via a .bbappend the initial .bb image recipe(in our case it was a core-image-sato for qemux86) or built separately without need to extend the image .bb file 
3) using a new .bbclass file that inherits testimage (i believe this can be added directly to testimage itself for a more generic approach) and prepares test runs that use built binaries both for a live runusing poky env.  and for exporting tests
4) using a new copybin.bbclass file to facilitate copy of resulting binary files to a common destination dir where test cases can access them. I think here would have been better to use the tmp/deploy/*/*.rpm files and obtain the binaries from there (since the target machine is not sure to have rpm or other similar tools): rpm2cpio myrpmfile.rpm | cpio -idmv
5) run bitbake image-type -c export_tests results in two .tar.gz files that can be extracted on an other host machine. One of these two .tar.gz contains a runtests.py executable that will launch and execute tests on tearget machine. I got stuck at this step, but will further sync with Junhui.

Please if anything was skipped feel free to add.

(In reply to comment #9)
Hi Costin,
Thanks for the proposal! Let's have a PoC for the proposal and find out what else need to be considered. Usually with more experiments more efforts are needed. 
So far, we have two proposals, one is listed in comment #8 based on lots of discussion, another one is Junhui's PoC shared to you in last week. After we get above one PoCed, we are be able to compare these two solutions and seek more feedbacks from experts through mailing list to finalize the solution.

Hello,

For a POC on obtaining the necessary binaries both native and DUT specific here are the steps to follow:

1) for native binaries, the best approach is as on comment#3 by using buildtools-tarball where one prepares
the necesary tools needed on a host machine that lacks of them (http://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#building-your-own-buildtools-tarball
). In my case i set up SDKMACHINE ?= "x86_64" in local.conf
Before building buildtools-tarball i performed few steps:
- made a hello world recipe ():
meta/recipes-test/hello$ cat hello_0.1.bb 

SUMMARY = "Hello world recipe"
DESCRIPTION = "test application"
LICENSE = "CLOSED"
SRC_URI = "file://hello.c \
"
S = "${WORKDIR}"
do_compile() {
    ${CC} hello.c -o hello
}

do_install() {
    install -d ${D}/${bindir}
    install -m 0755 hello ${D}/${bindir}
}

BBCLASSEXTEND = "nativesdk"



where hello.c is located in:

/meta/recipes-test/hello/files$ cat hello.c hello.c
#include <stdio.h>

int main(int argc, char** argv)
{
  printf("Hello world for native binary\n");
  return 0;

}

The key here is BBCLASSEXTEND = "nativesdk" (http://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#ref-classes-nativesdk)

- added nativesdk-hello to buildtools-tarball recipe meta/recipes-core/meta/buildtools-tarball.bb:


TOOLCHAIN_HOST_TASK ?= "\
    ...
    nativesdk-hello \
    "

more on usage of TOOLCHAIN_HOST_TASK: http://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#var-TOOLCHAIN_HOST_TASK
With the help of this variable we can add native packages for the remote host.

- bitbake buildtools-tarball
After this step, copy the resulting build/tmp/deploy/sdk/*buildtools-tarball*.sh on the needed machine and run it. 

- on the remote host: 
/opt/poky/1.8+snapshot/sysroots$ ./x86_64-pokysdk-linux/usr/bin/hello
Hello world for native binary

2) for DUT speciffic binaries:
- prepare the required recipe, very similar to the native version
- bitbake the recipe
- extract the needed binaries:  rpm2cpio build/tmp/deploy/*/name_of_package*.rpm | cpio -idm

Hi Cathy, Junhui, Costin

Are there any updates on this?

Hi Cathy, Junhui,

Does the solution proposed by Costin address all your requirements.

Costin, how the solution proposed by you addresses the requirement that Richard formulated:
"a) We need to markup the tests so we know which binaries they need to run the tests. The export code can then use this markup to ensure the binaries needed are included" ?

Also, does it also satisfy Comment 4 from Junhui? Cathy, Junhui, can you please let us know if you agree with the PoC that Costin provided?

Beside TCs knowing where binaries are located, is there a need to change that on case by case scenario? Is there a need to run the same TC with a different set of binaries?

Waiting for your input.

Regards,
Cristian Iorga.

Costin,
  We are still under evaluating this solution. Actually, for implementation, the progress is not fast given we had some debate about how to track meta information, e.g. rpm name, binary name, target/native path, and how to deploy test binary to host and target. Meanwhile, Patrick propose a sstate safe copybin solution, it is simple and we  have prove it workable.
  I'll suggest we have a meeting to review 2 solutions

Hi Cristi,

Related to your questions, the idea was first to have a way to generate binaries both for DUT and remote PC. This was the first step and based on comment #3 this is what i did. The main thing here, was to have a way to obtain binaries and ship them somewhere else in a packed manner. I addressed more comment #7.

What i proposed is not treating how to markup tests and what binaries they will use, as this is a matter of test case logic and it's something separate.

Thanks,

Costin

Hi Jing,

Thanks for the update. We'll schedule a meeting.

Costin

Created attachment 2692 [details]
deploy-files.bbclass

 This file provide a sstate safe approach to deploy files which are generated by recipe. With it, it is very convenient for recipe to choose specific files to deploy but not break origin deploy logic. By default, the location of deploy-files is under ${DEPLOYDIR}/files/. For target machine file, the destination is ${DEPLOYDIR}/files/target/${MACHINE}. For native file, it is ${DEPLOYDIR}/files/target/${BUILD_ARCH}. Thanks ohly Patrick's help to complete this file.

Hello,

Got to an end with this feature. The link on contrib:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=cc-tin/YOCTO7850

Solution is based on comment's #3 recommendation by using as base buildtols-tarball.

How to use it:
1) write your own recipe for binaries(apps.) that need to be tested in the end.
I created two new recipes, one that generates a binary for the host system and 
one for the DUT (hello_0.1.bb and hello-DUT_0.1.bb  both in meta/recipes-test). 
Please note that horst related recipes contain BBCLASSEXTEND = "nativesdk"

2) build the DUT related recipes:

bitbake hello-DUT

3) add to local.conf the required binaries needed on the host system + necessary info regarding DUT's IP and tests that are required to run on the remote host:

TOOLCHAIN_HOST_TASK_append = " nativesdk-hello"
TEST_SUITES = "scp ssh date mytest"
TEST_TARGET_IP = "192.168.7.2"
TEST_SERVER_IP = "192.168.7.1"

4) build the recipe that will generate everything needed on remote host. I named it btools-with-tc (if this is not suggestive enough, it can be changed). No need to build anything else before (like core-image-*):

bitbake btools-with-tc

5) copy/send via scp to an other Linux host the resulting script:
scp tmp/deploy/sdk/poky*-glibc*btools-with-tc-i586-buildtools-nativesdk-standalone-1.8+snapshot-20150929.sh user@IP:/desired_path

6)run the script on the remote machine. 

7)go to the installation directory and source the environment:
/opt/poky-lsb/1.8+snapshot# . ./environment*

8) run the runexported.py script:

/opt/poky-lsb/1.8+snapshot# ./runexported.py testdata.json (in the near future only 
./runexported.py will be needed without the testdata.json)
SSH log file: /opt/poky-lsb/1.8+snapshot/ssh_target_log.20150930085326
test_scp_file (oeqa.runtime.scp.ScpTest) ... ok
test_ssh (oeqa.runtime.ssh.SshTest) ... ok
test_date (oeqa.runtime.date.DateTest) ... ok
test_hello_on_DUT (oeqa.runtime.mytest.mytest) ... 
Hello world on DUT

ok
test_hello_on_host_machine (oeqa.runtime.mytest.mytest) ... Hello world on host machine 
ok

----------------------------------------------------------------------
Ran 5 tests in 7.399s

OK


Important notes:
a) If one would like to write new tests (for the eg. I used the mytest.py in 
meta/lib/oeqa/runtime/mytest.py) there is no need to redo steps 1) to 6). New tests can be written and than just add in testdata.json file in this section info. related to your test.
eg.:
    "testslist": [
        "oeqa.runtime.scp", 
        "oeqa.runtime.ssh", 
        "oeqa.runtime.date", 
        "oeqa.runtime.mytest",
       "oeqa.runtime.newtest"
    ], 

I added here a "newtest" that coresponds to a newtest.py file added on the host machine (just for demo reasons it's a copy of mytest.py)

b) for testing new binaries, again there is no need to redo steps 1) to 6). Only step 1) to 2) are nedded. The idea here is to extract the resulting .rpm packages with rpm2cpio tmp/deploy/rpm/path_to_resulting_rpm | cpio -idm and port it to the remote machine in extracted_binaries directory creating a directory structure similar to what is already there.

It would make a good title for a new feature to just copy the .rpm file on the remote machine and let the testing environment care for the rest. 

Marking as resolved

Hello,

Small add to previous comment. When writing test cases the structure should be like this:

    @TestNeedsBin("hello-DUT")
    def test_hello_on_DUT(self):
         (status, output) = self.target.run("hello")
         print "\n" + str(output) + "\n"
         self.assertEqual(status, 0, "bad stuff happened on DUT")

where the first parameter of TestNeedsBin() decorator is the name of the recipe from which results the needed binaries. It also accepts a second parameter, the binary version. 
When running the resulting binary on the remote machine, the self.target.run() takes as argument the name of the binary found in the resulting rpm file.

Costin, a typical test binary dependent TC involve both Native binary (host) and Target binary (device), e.g client binary on host talk with sever binary on device. could you provide a example how to handle such case? thanks.

Hi Junhui,

Please see the mytest.py example on git-contrib link.

Costin

just for the full path:
meta/lib/oeqa/runtime/mytest.py

Per our discussion, a new requirement #8481 is filed for test tar ball generation. We can track test binary requirement in this one only.

Hi Cathy,

I agree with you for tracking test tarball generation. 
This feature was intended to be like a "milestone" of including test related binaries in test cases and make them available both on poky environment (e.g. bitbake -c testimage core-image-sato) and on a non-poky environment with ./runexported on a different machine.
The big advantage of this approach is including necessary SDK tools for the remote machine, which for the 8481 is not necessary. We'll approach separately that one.

I have tried the demo with http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=cc-tin/YOCTO7850 . It works on my environment just like the demo Costin did, but we still need time to see if there is any gap to meet overall export test requirement, e.g. if all the information to generate test tar balls are available and retrievable. Of course, the test tar ball generation is tracked as another feature: #8481. Once we get the gaps tracked as bugs/requirements in bugzilla, we are OK to close this one.

Thanks Costin for the great effort! We tried this and think it basically works. Some minor issues and requirements are filed as #8530 ~ #8536. Together with #8481, hopefully we can switch to yocto upstream solution for export test.

Implementation is done, but it's not integrated into master at this point.

waiting for the patches to be sent to master

+set the flag to docs needed.

What's progress for this feature's integration?

Hello,

Because of new features: 8478~8483 and 8530~8536 the implementation for this one changed in the way that a lot of functionality was shifted from btools-with-tc.bb to testimage.bbclass. Like so, part of functionality can be used for the above features, meaning that integration to YP master will be correlated with completing part of those other features too.

Patches sent to mailing list

The current code [1] needs some work based on feedback of Ross,

1) discovery of the decorator examines the source files instead of
introspecting
2) deploy of binaries requires RPM packaging, the solution will be isolating for work in any package manager


[1] http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=ross/export

Applying additional resources to try and get this accomplished in time for 2.1M4. Current estimate is ~1 week.

After additional discussion, it makes more sense to move this to 2.2M1. See email below.

From:	"Esquivel, Benjamin" <benjamin.esquivel@intel.com>
Date:	Tue, 5 Apr 2016 16:56:20 +0000 (04/05/2016 09:56:20 AM)

I brought this topic to the SXT meeting and the consensus was that 2.1 M4 seems difficult for the following reasons:

1. The change is based on a configuration that is no longer used, that amount of work is not going to be ready for M4
2. This change, even if it could be finished on time, jeopardizes the sanity of the release

The course of action that seems natural today is to defer this to 2.2 M1 with the following actions:

A. Assign GDC resources for taking this over and plan for 2.2 M1 as the support we have from the RO team is practically over.
B. Communicate with Jing taking into account that they will not be happy with this. We also need communication with them to better understand their current needs.

Daniel Istrate made a rebase to master from the ross/export branch:

https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=distrate/export_rebased

Needs review for the upcoming work.

Sent patches to the mailing list:

http://lists.openembedded.org/pipermail/openembedded-core/2016-June/122508.html

All the implementation of the features requested are in master branch. Jing just requested a way to define the sdk packages to be defined outside local.conf, I proposed to use the environment var BBPOSTCONF in order to do that. Keep the bug opened to have an accepted solution to define sdk packages.

Jing accepted the solution, the documentation has been updated and the functionality is integrated in master branch. Setting to RESOLVED