Bug 10833

I'm not sure I understand what this bug is about. Where ip tap should be generated? It can't be generated when wic image is built, right? As soon as it's built it's done, it's ready to boot. I don't see at which point ip should be generated.

Can you elaborate on this?

(In reply to comment #1)
> I'm not sure I understand what this bug is about. Where ip tap should be
> generated? It can't be generated when wic image is built, right? As soon as
> it's built it's done, it's ready to boot. I don't see at which point ip
> should be generated.
> 
> Can you elaborate on this?

IP should be generated when runqemu is called and passed setto the wic image booted on qemu instance

This requires modification of the image as tap ip is provided in kernel command line, which is located in bootloader configuration file inside the image. To modify the image it has to be mounted, which generally requires root privileges. It can be done by running qemu with as special instrumental image which would mount wic image and modify it, but I doubt we want that.

It's easier to use serial interface to access the image inside qemu. You can look at test_qemu test case in oe-selftest wic module for details.

looking at the amount of work that a proper solution (see bug 10847) would take, fixing this for 2.3-M3 would mean modifying the existing automation flow to account for this WIC particularity.

The first step would be to have a scripted way of setting up the IP, assuming the serial connection is known.

moving to M4. I don't time to do it in M3 time frame, sorry.

Jose, can you give me an example of wic test case(s) that fails because of this?

As I can't make this working due to the reasons discussed above I need real test case that I can fix by setting ip using serial connection.

Another question is: do we really need that ip at all? Can we just run tests using serial connection like it's done in oe-selftest wic test suite?

Hi Edd

The main idea is to execute testimage (runtime suite) for WIC on qemu, this suite is designed to send test by SSH hence we need that qemu booted has properly an IP set to be able to execute the the entire test suite.

Simple runtime tests are also performed by oe-selftest wic suite? Why do we need both, especially if one doesn't work at all?

(In reply to comment #8)
> Simple runtime tests are also performed by oe-selftest wic suite? Why do we
> need both, especially if one doesn't work at all?

As part of the QA cycle and it is required that testimage suite is executed in all artifacts mentioned here [1], so if its not requiered/possible to execute testimage on WIC images for qemu we need to have the go ahead from CCB as well as Sthepen and BSP Team.

Can you point me out on testimage docs/sources? I'm still not convinced on feasibility of doing duplicate work I'm willing to at least look at what's tested there.

(In reply to comment #10)
> Can you point me out on testimage docs/sources? I'm still not convinced on
> feasibility of doing duplicate work I'm willing to at least look at what's
> tested there.

- The Steps to execute test cases are here :

 https://wiki.yoctoproject.org/wiki/Quality_Assurance_yocto_project#Steps_for_BSP_Automated_Testing

- Test cases are locate at:
  
   Testopia: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6564
   Source: poky/meta/lib/oeqa/runtime/cases

OK, thanks.

Double-checking just to ensure we're on the same page here:
Would it be enough to fix this bug if I change testimage code in a way that it sets qemu instance IP using serial connection to the instance?

(In reply to comment #12)
> OK, thanks.
> 
> Double-checking just to ensure we're on the same page here:
> Would it be enough to fix this bug if I change testimage code in a way that
> it sets qemu instance IP using serial connection to the instance?
This could be enough for WIC on qemu but the code should be still working for all other devices/images as is working right know (with SSH). 

Mariano, Anibal do you have any comment on this regards?

(In reply to comment #12)
> OK, thanks.
> 
> Double-checking just to ensure we're on the same page here:
> Would it be enough to fix this bug if I change testimage code in a way that
> it sets qemu instance IP using serial connection to the instance?

Ed,

Remember that testimage can use several targets, right now OE-Core uses only two: simpleremote (a booted device that connects using SSH) and qemu.

Also testexport is based in testimage to run runtime tests in a host without OE or bitbake, so the tests would be able to work without the build environment.

Just keep in mind this during the implementation.

Thanks for the info!

Is it possible to run wic image in qemu and then use simpleremote target to run tests on it?

(In reply to comment #15)
> Thanks for the info!
> 
> Is it possible to run wic image in qemu and then use simpleremote target to
> run tests on it?

Currently not; and this is why this bug was oppend; to do this we need that image booted has an IP set to be able to do the connection.

> Currently not; and this is why this bug was oppend; 

I thought this bug was opened because it's not possible to run testimage for wic images in qemu mode. I'm asking about simpleremote mode now.

> to do this we need that
> image booted has an IP set to be able to do the connection.

How do you assign ip on real devices to run testimage in simpleremote mode?

(In reply to comment #17)
> > Currently not; and this is why this bug was oppend; 
> 
> I thought this bug was opened because it's not possible to run testimage for
> wic images in qemu mode. I'm asking about simpleremote mode now.
> 
> > to do this we need that
> > image booted has an IP set to be able to do the connection.
> 
> How do you assign ip on real devices to run testimage in simpleremote mode?

The idea of testimage is to test an image, this could be run in qemu or in hardware (simpleremote). testimage will initilize the target (in qemu it will start qemu, in simpleremote it will do nothing) and then run the test in the target, once it finished it will stop the target (in qemu it will halt the target, in simpleremote do nothing).

So asking your question, in qemu it is assigned automatically depending in the tap interface used by runqemu, in simpleremote it needs to be assigned manually by the person that setup the target.

So, what it can be done is to in the initialization of qemu check for an IP address and if not is what you need, assign one.

> So asking your question, in qemu it is assigned automatically depending in
> the tap interface used by runqemu, in simpleremote it needs to be assigned
> manually by the person that setup the target.
> 

Why can't this person setup ip manually in qemu instance? What's the differece? I think it's even easier as this person don't need to go to the lab where real device is located :)

(In reply to comment #19)
> > So asking your question, in qemu it is assigned automatically depending in
> > the tap interface used by runqemu, in simpleremote it needs to be assigned
> > manually by the person that setup the target.
> > 
> 
> Why can't this person setup ip manually in qemu instance? What's the
> differece? I think it's even easier as this person don't need to go to the
> lab where real device is located :)

Real hardware can be a qemu image too, I think the problem is that QA wants to run testimage in the autobuilders with wic images, so right now, there is no way of doing this.

Ok, thanks. I think we're slowly getting somewhere :)

when run on autobuilder images are built there and then run testimage, right?
if ip we want to assign is known *before* image is built then we can assign it when image is built. It's all about adding some parameters to the kernel command line, right? if it's so we can just put them into .wks and build wic image with that wks. The result image will run kernel with thise parameters.

Does this make sense?

(In reply to comment #21)
> Ok, thanks. I think we're slowly getting somewhere :)
> 
> when run on autobuilder images are built there and then run testimage, right?
> if ip we want to assign is known *before* image is built then we can assign
> it when image is built. It's all about adding some parameters to the kernel
> command line, right? if it's so we can just put them into .wks and build wic
> image with that wks. The result image will run kernel with thise parameters.
> 
> Does this make sense?

This seems to be a correct approach.

(In reply to comment #22)
> (In reply to comment #21)
> > Ok, thanks. I think we're slowly getting somewhere :)
> > 
> > when run on autobuilder images are built there and then run testimage, right?
> > if ip we want to assign is known *before* image is built then we can assign
> > it when image is built. It's all about adding some parameters to the kernel
> > command line, right? if it's so we can just put them into .wks and build wic
> > image with that wks. The result image will run kernel with thise parameters.
> > 
> > Does this make sense?
> 
> This seems to be a correct approach.

I think we need another approach, if we generate a wks file for every image build we could hit a race condition, because in the autobuilder more than one image is build and tested at the same time.

(In reply to comment #23)
> (In reply to comment #22)
> > (In reply to comment #21)
> > > Ok, thanks. I think we're slowly getting somewhere :)
> > > 
> > > when run on autobuilder images are built there and then run testimage, right?
> > > if ip we want to assign is known *before* image is built then we can assign
> > > it when image is built. It's all about adding some parameters to the kernel
> > > command line, right? if it's so we can just put them into .wks and build wic
> > > image with that wks. The result image will run kernel with thise parameters.
> > > 
> > > Does this make sense?
> > 
> > This seems to be a correct approach.
> 
> I think we need another approach, if we generate a wks file for every image
> build we could hit a race condition, because in the autobuilder more than
> one image is build and tested at the same time.

I think this apply to very specific target (WIC on qemu) so wks file wont be created for every image on the AB, afaiu.

> I think we need another approach, if we generate a wks file for every image
> build we could hit a race condition, because in the autobuilder more than
> one image is build and tested at the same time.

wks file is just a small text file. We can create tons of them if needed and each of them can have different name(we can add ip address to the name, for example). they're needed only when wic starts, so  we can remove them as soon as image is built. I don't see where we can get race here.

Can you explain which parameters are required to be put to kernel command line to set ip? Can you point out to the code where it's done?

(In reply to comment #25)
> > I think we need another approach, if we generate a wks file for every image
> > build we could hit a race condition, because in the autobuilder more than
> > one image is build and tested at the same time.
> 
> wks file is just a small text file. We can create tons of them if needed and
> each of them can have different name(we can add ip address to the name, for
> example). they're needed only when wic starts, so  we can remove them as
> soon as image is built. I don't see where we can get race here.

I think you will create these wks files with the image creation using FSTYPES, the problem that I see is the autobuilders create several images at the same time. Is there a way to specify which IP address should be assigned in wks file for such scenario?

Moreover, after the image creation the autobuilder will test all the images with testimage. At these point the wic images are already created with the kernel parameters set, isn't it? So the race condition that I'm saying, is how do you know what tap interface would be used by minimal, sato, sato-sdk?

> I think you will create these wks files with the image creation using
> FSTYPES, the problem that I see is the autobuilders create several images at
> the same time. Is there a way to specify which IP address should be assigned
> in wks file for such scenario?

That's why asked this question before:
> when run on autobuilder images are built there and then run
> test image, right?
> if ip we want to assign is known *before* image is built
> then we can assign it when image is built.

So, is ip known *before* image is built or not?

If it's known then we can generate wks and put ip into it. If it's not then we need to come up with something else.

> Moreover, after the image creation the autobuilder will test all the images
> with testimage. At these point the wic images are already created with the
> kernel parameters set, isn't it?

yes, it is. When we create them we know which image we create and which ip we put into its .wks file, right?

> So the race condition that I'm saying, is
> how do you know what tap interface would be used by minimal, sato, satsdk?

if we know about it when we create an image how we manage to 'forget' it when we test it?

(In reply to comment #28)
> > I think you will create these wks files with the image creation using
> > FSTYPES, the problem that I see is the autobuilders create several images at
> > the same time. Is there a way to specify which IP address should be assigned
> > in wks file for such scenario?
> 
> That's why asked this question before:
> > when run on autobuilder images are built there and then run
> > test image, right?
> > if ip we want to assign is known *before* image is built
> > then we can assign it when image is built.
> 
> So, is ip known *before* image is built or not?

Because of how runqemu works, we don't know.

ok, then we probably need to modify runqemu to set ip through serial console for wic images. Would that work?

(In reply to comment #30)
> ok, then we probably need to modify runqemu to set ip through serial console
> for wic images. Would that work?

I would think so, but wouldn't be easier setup a DHCP server? I just did the set up in my machine and took me about 2 hours.

If you think it will be easier then go ahead and take this bug. Just let me know if you want to do it. I'll switch to something else.

(In reply to comment #32)
> If you think it will be easier then go ahead and take this bug. Just let me
> know if you want to do it. I'll switch to something else.

I don't think it will be accepted such solution for M4, perhaps the change can be made on 2.4

Let's move it to 2.4 then. What's the problem?

I'd propose to use slirp http://wiki.qemu-project.org/Documentation/Networking#User_Networking_.28SLIRP.29 to setup networking.

Just adding 'slirp' parameter to runqemu and running dchp client makes the guest accessible by ssh.

Unlike tap slirp doesn't require root privileges to setup it, which enables running tests without root privileges. 

Here is an example:
1. Add ssh server and dhcp client to conf/local.conf
IMAGE_FEATURES += "ssh-server-openssh"
CORE_IMAGE_EXTRA_INSTALL += "busybox-udhcpc"

2. build an image:
bitbake core-image-minimal

3. Run it in slirp networking mode, log into it and run dchp client to get ip address from qemu internal dhcp server:

runqemu ./tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic nographic slirp

runqemu - INFO - CONFFILE: /home/ed/git/yocto/poky/build/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.qemuboot.conf
runqemu - INFO - Continuing with the following parameters:

MACHINE: [qemux86-64]
FSTYPE: [wic]
ROOTFS: [/home/ed/git/yocto/poky/build/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic]
CONFFILE: [/home/ed/git/yocto/poky/build/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.qemuboot.conf]

runqemu - INFO - Port forward: hostfwd=tcp::2222-:22 hostfwd=tcp::2323-:23
runqemu - INFO - Using scsi drive
runqemu - INFO - Running ldd /home/ed/git/yocto/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot-native/usr/bin/qemu-system-x86_64...
runqemu - INFO - Running /home/ed/git/yocto/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot-native/usr/bin/qemu-system-x86_64 -device virtio-net-pci,netdev=net0,mac=52:54:00:12:35:02 -netdev user,id=net0,hostfwd=tcp::2222-:22,hostfwd=tcp::2323-:23 -drive if=none,id=hd,file=/home/ed/git/yocto/poky/build/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic,format=raw -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=hd -no-reboot -vga vmware -show-cursor -usb -usbdevice tablet -device virtio-rng-pci  -nographic  -cpu core2duo -m 256 -serial mon:stdio -serial null 

...

qemux86-64 login: root
root@qemux86-64:~# udhcpc eth0
udhcpc (v1.24.1) started
Sending discover...
Sending select for 10.0.2.15...
Lease of 10.0.2.15 obtained, lease time 86400
/etc/udhcpc.d/50default: Adding DNS 10.0.2.3


4. Ssh to the guest machine from the host:

$ ssh root@localhost -p 2222
Warning: Permanently added '[localhost]:2222' (ECDSA) to the list of known hosts.
Last login: Mon Mar 13 16:32:07 2017 from 10.0.2.2
root@qemux86-64:~# uname -a
Linux qemux86-64 4.10.0-yocto-standard #2 SMP PREEMPT Mon Mar 13 12:35:26 EET 2017 x86_64 GNU/Linux

------------------------

Note, that runqemu is clever enough to use different ports in hostfwd option, so there is no conflicts even if multiple images are running on the same host.

Opinions?

Adding ip=dhcp to the kernel commandline in .wks file makes kernel to configure interface on boot, so we even don't need to run dhcp client.

So, my proposal is to use slirp instead of tap to configure networking in test images. This method doesn't require root privileges to configure networking, uses internal qemu dhcp server and  works the same way for any type of image including wic.

One disadvantage of slirp is that it's slower than tap. However, that shouldn't be a problem unless we're transferring a lot of data through network interfaces. Even in this case we should measure performance impact before making decision which method to use.

Thoughts?

As implementation of slirp networking seems to be quite complex change and I haven't got any feedback here I decided to postpone it. Please, create separate bug if you want this to be implemented.

Instead I implemented setting up guest networking through serial console as I proposed earlier.

Please review: http://lists.openembedded.org/pipermail/openembedded-core/2017-March/134279.html

The fix has been merged to master.