Bug 10604 (AB-device-testing)

The easiest alternative is to use the AB and then exec aft in every worker with certain parameters.

The more elaborated solution will be modify the AFT to make it a services that could handle RPC requests from multiple clientes such as the Autobuilder or a CLI cmdline interface.

Simo is working to make a Distributed AFT - (DAFT :-) where AFT runs mostly on a Beagle Bone Black (one for each DUT).

This configuration simplifies a lot the cabling, by using one USB cable for:
- replacing the USB stick / SD card for the support image with the BBB emulating a a mass storage
- providing connectivity to the DUT by doing ethernet over USB
- acting as input device (keyboard/mouse emulator) with the BBB emulating a HID device


The part I described is already implemented and tested.

What is still missing is to move the queuing (for multiple devices) on to the central worker which acts as DHCP server to the fleet of BBB.

We should have a call next week so that we can explain this evolution (the code has been already committed to the repo, though).

(In reply to comment #2)
> Simo is working to make a Distributed AFT - (DAFT :-) where AFT runs mostly
> on a Beagle Bone Black (one for each DUT).
> 
> This configuration simplifies a lot the cabling, by using one USB cable for:
> - replacing the USB stick / SD card for the support image with the BBB
> emulating a a mass storage
> - providing connectivity to the DUT by doing ethernet over USB
> - acting as input device (keyboard/mouse emulator) with the BBB emulating a
> HID device
> 
> 
> The part I described is already implemented and tested.
> 
> What is still missing is to move the queuing (for multiple devices) on to
> the central worker which acts as DHCP server to the fleet of BBB.
> 
> We should have a call next week so that we can explain this evolution (the
> code has been already committed to the repo, though).

Hi Igor,

Thanks for the update, sounds a great progress and seems the direction that we want to do.

One difference is about we don't plan to use the support image for prepare (write image and boot) the DUT for testing instead we have an USB multiplexer and the idea is that the AFT central instance will do.

Other question that came into my mind if you are planning to provide an RPC interface for the AFT.

Cheers,
alimon

(In reply to comment #3)

> One difference is about we don't plan to use the support image for prepare
> (write image and boot) the DUT for testing instead we have an USB
> multiplexer and the idea is that the AFT central instance will do.

It depends on what you want to do and what sort of DUT you have.

If you are happy to boot the test image from an external storage (like the USB key that you would multiplex), then you do not need the support image.

Similarly, you do not need the support image if your DUT can be flashed with some tool like DFU, or otherwise exposes its internal storage - ex as USB mass storage.

But.
If your device has internal storage and doesn't let you write it by using some special mode, then the support image is needed (IMHO).

Why? Because a device typically boots from its internal storage, can behave differently depending from where it booted and the kernel driver that controls the internal boot device is not necessarily the same used for external boot.

IOW not using the internal storage leaves a significant gap in the test coverage.
Ideally both types of boot (internal and external) would be tested.
Having to pick one, the safest choice is definitely to go with the internal one.

If the support image causes issues, please list them, so that they can be addressed, but so far it doesn't seem to be possible to avoid it, when deploying to internal storage of a device that acts like a PC (ex: NUC).

Even on a minnowboard, that doesn't have a real internal storage, I would expect that a productized version would either use a semi-permanent SD card or would replace the SD with an eMMC.

Notice also that in the evolution of AFT, the "central instance" is anyway fading away because of the more complex cabling it implies.

The decentralized version allows to rely solely on ethernet as transportation media.

But it's easier to discuss this live ...

> Other question that came into my mind if you are planning to provide an RPC
> interface for the AFT.

Why not? Given a set of requirements, it can be planned, if the existing interface is not sufficient. IOW, what use cases would need it?

https://bugzilla.yoctoproject.org/show_bug.cgi?id=10604(In reply to comment #4)
> (In reply to comment #3)
> 
> > One difference is about we don't plan to use the support image for prepare
> > (write image and boot) the DUT for testing instead we have an USB
> > multiplexer and the idea is that the AFT central instance will do.
> 
> It depends on what you want to do and what sort of DUT you have.
> 
> If you are happy to boot the test image from an external storage (like the
> USB key that you would multiplex), then you do not need the support image.
> 
> Similarly, you do not need the support image if your DUT can be flashed with
> some tool like DFU, or otherwise exposes its internal storage - ex as USB
> mass storage.
> 
> But.
> If your device has internal storage and doesn't let you write it by using
> some special mode, then the support image is needed (IMHO).
> 
> Why? Because a device typically boots from its internal storage, can behave
> differently depending from where it booted and the kernel driver that
> controls the internal boot device is not necessarily the same used for
> external boot.
> 
> IOW not using the internal storage leaves a significant gap in the test
> coverage.
> Ideally both types of boot (internal and external) would be tested.
> Having to pick one, the safest choice is definitely to go with the internal
> one.
> 
> If the support image causes issues, please list them, so that they can be
> addressed, but so far it doesn't seem to be possible to avoid it, when
> deploying to internal storage of a device that acts like a PC (ex: NUC).
> 
> Even on a minnowboard, that doesn't have a real internal storage, I would
> expect that a productized version would either use a semi-permanent SD card
> or would replace the SD with an eMMC.
> 
> Notice also that in the evolution of AFT, the "central instance" is anyway
> fading away because of the more complex cabling it implies.
> 
> The decentralized version allows to rely solely on ethernet as
> transportation media.

Yes i understand the reasons for use the supported image but in our case the AFT
could do it instead of go to the supported media, also i think the abstraction in the AFT exists to support the both modes.

> 
> But it's easier to discuss this live ...
> 
> > Other question that came into my mind if you are planning to provide an RPC
> > interface for the AFT.
> 
> Why not? Given a set of requirements, it can be planned, if the existing
> interface is not sufficient. IOW, what use cases would need it?

There is no reason to not implement it, i'm only asking if you have some plans to do that.

The idea is to be able to request a test cycle by the RPC (JSON, XML) and the AFT be able to schedule that request, do the testing and the publish back the results. We
need to define the interface needed. In this way the AB/Jenkins/CLI or whatever could request an AFT test cycle.

(In reply to comment #5)
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=10604(In reply to comment
> #4)

> Yes i understand the reasons for use the supported image but in our case the
> AFT could do it instead of go to the supported media

What does it mean?

1) you do not care to validate that the device boots properly from internal storage

2) you have devices where you can deploy directly to the internal storage

3) ???? something else?


> also i think the abstraction in the AFT exists to support the both modes.

Sure, and I do not object with adding support for it, but I would like to understand better what "in our case"  means.


[...]

> There is no reason to not implement it, i'm only asking if you have some
> plans to do that.
> 
> The idea is to be able to request a test cycle by the RPC (JSON, XML) and
> the AFT be able to schedule that request, do the testing and the publish
> back the results. We
> need to define the interface needed. In this way the AB/Jenkins/CLI or
> whatever could request an AFT test cycle.

The idea is to follow the KISS principle.

AFT is currently a command line tool, it is meant to be run even by individual developer on own machine.

As I described in a previous comment, we are now discussing to split it, so that one part will run on the server and to the scheduling, while the other part will run on the BBB and take care of handling its DUT.

The server part _might_ get the RPC you describe, however I'm not sure it would be really needed: if we use jenkins in CI, a specific slave can just act as frontend toward the master. That's what the slave is for and I would not want to replicate this functionality inside AFT.

I have the feeling that every and each CI incarnation will have its needs and these would seem to be better served by creating a wrapper specific to the CI setup that will use AFT, rather than adding the functionality straight into AFT.

(In reply to comment #6)
> (In reply to comment #5)
> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=10604(In reply to comment
> > #4)
> 
> > Yes i understand the reasons for use the supported image but in our case the
> > AFT could do it instead of go to the supported media
> 
> What does it mean?
> 
> 1) you do not care to validate that the device boots properly from internal
> storage

Yes we care the AFT central instance will use the USB multiplexer for write the image into usb stick then switch to the DUT and finally installs the image into internal storage of the DUT.

> 
> 2) you have devices where you can deploy directly to the internal storage

No.

> 
> 3) ???? something else?
> 
> 
> > also i think the abstraction in the AFT exists to support the both modes.
> 
> Sure, and I do not object with adding support for it, but I would like to
> understand better what "in our case"  means.
> 
> 
> [...]
> 
> > There is no reason to not implement it, i'm only asking if you have some
> > plans to do that.
> > 
> > The idea is to be able to request a test cycle by the RPC (JSON, XML) and
> > the AFT be able to schedule that request, do the testing and the publish
> > back the results. We
> > need to define the interface needed. In this way the AB/Jenkins/CLI or
> > whatever could request an AFT test cycle.
> 
> The idea is to follow the KISS principle.
> 
> AFT is currently a command line tool, it is meant to be run even by
> individual developer on own machine.
> 
> As I described in a previous comment, we are now discussing to split it, so
> that one part will run on the server and to the scheduling, while the other
> part will run on the BBB and take care of handling its DUT.
> 
> The server part _might_ get the RPC you describe, however I'm not sure it
> would be really needed: if we use jenkins in CI, a specific slave can just
> act as frontend toward the master. That's what the slave is for and I would
> not want to replicate this functionality inside AFT.
> 
> I have the feeling that every and each CI incarnation will have its needs
> and these would seem to be better served by creating a wrapper specific to
> the CI setup that will use AFT, rather than adding the functionality
> straight into AFT.

Yes i agree, the service part could/couldn't go into the AFT, if you are more comfortable to write separate wrappers/programs to use the AFT cmdline tool is ok for me, but we want to merge the resulted program into the AFT upstream repo.

(In reply to comment #7)

> Yes we care the AFT central instance will use the USB multiplexer for write
> the image into usb stick then switch to the DUT and finally installs the
> image into internal storage of the DUT.

I see. This covers the installation scenario.
Even in this case, though, the approach of using the BBB would avoid the need for the mux. It would also avoid the delay of writing the installer to the USB key.

Furthermore, I'm still not convinced that there is no need to test the case where - for example in an assembly line - an image would be written directly to the internal storage and then booted. The installer doesn't address it.


[...]

> but we want to merge the resulted program into the AFT upstream repo.

Yes, there could be a directory with the various wrappers.

(In reply to comment #3)
...
> One difference is about we don't plan to use the support image for prepare
> (write image and boot) the DUT for testing instead we have an USB
> multiplexer and the idea is that the AFT central instance will do.
One small correction here, we plan to use both methods, the service image and the image media multiplexing.

(In reply to comment #8)
> (In reply to comment #7)
> 
> > Yes we care the AFT central instance will use the USB multiplexer for write
> > the image into usb stick then switch to the DUT and finally installs the
> > image into internal storage of the DUT.
> 
> I see. This covers the installation scenario.
> Even in this case, though, the approach of using the BBB would avoid the
> need for the mux. It would also avoid the delay of writing the installer to
> the USB key.
The USBmux helps to cover the testing of the instructions from the manual that tell the user to burn the image to a removable media and then insert it in the board for booting for installing it. While its agreeable to make this installing/booting process easier for the rest of the testing, it is still required to verify as the manual says and the usb muxer helps QA to automate that process. 
> 
> Furthermore, I'm still not convinced that there is no need to test the case
> where - for example in an assembly line - an image would be written directly
> to the internal storage and then booted. The installer doesn't address it.
QA needs to test both usage types.

With the distributed version of AFT, bug 9689 is not a requirement for the automation.

The first version is out review at,

https://lists.yoctoproject.org/pipermail/yocto/2017-June/thread.html

It has several areas of improvement like,

- Don't have fixed DUT's into the parameters.
- Don't use sudo to execute DAFT? (why use root?)
- Be able to select which image wants to build.

Patches on review move to M2.

Refactored patches sent v3,

https://lists.yoctoproject.org/pipermail/yocto/2017-July/037153.html

First version merged at,

http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/commit/?id=2eab5294eff6008d0ce6cd1026c498961ecabd81