At the moment there's no way to trigger asynchronous operations from the web interface. This would be useful for example to trigger one-off manual updates, run the OE-Classic update script, etc. This bug is for the basic ability to trigger such operations from within the code, and then to use it for sending the email to maintainers when a new layer is submitted (since that is currently synchronous, and if an SMTP error occurs it blocks the submission which is really not desirable). FWIW I did look into using Celery to implement this but struggled to figure out how to use it. That doesn't mean Celery isn't the way to do it, but it did seem pretty heavyweight. Then again, we do not want to be implementing such an asynchonous execution framework ourselves. It may also be worth talking to Michael Halstead to ensure a proposed solution will work well for the layers.openembedded.org infrastructure.
I did a play project using celery+rabbitmq and it was relatively straightforward. https://tests4geeks.com/python-celery-rabbitmq-tutorial/
(In reply to comment #1) > I did a play project using celery+rabbitmq and it was relatively > straightforward. > https://tests4geeks.com/python-celery-rabbitmq-tutorial/ (In reply to comment #0) > At the moment there's no way to trigger asynchronous operations from the web > interface. This would be useful for example to trigger one-off manual > updates, run the OE-Classic update script, etc. > > This bug is for the basic ability to trigger such operations from within the > code, and then to use it for sending the email to maintainers when a new > layer is submitted (since that is currently synchronous, and if an SMTP > error occurs it blocks the submission which is really not desirable). > > FWIW I did look into using Celery to implement this but struggled to figure > out how to use it. That doesn't mean Celery isn't the way to do it, but it > did seem pretty heavyweight. Then again, we do not want to be implementing > such an asynchonous execution framework ourselves. It may also be worth > talking to Michael Halstead to ensure a proposed solution will work well for > the layers.openembedded.org infrastructure. I've got RabbitMQ and Celery up locally, running the update script on command. I'll tie it into the layer submission view and submit the WIP for review.
OK, so I got Celery to run the update command asynchronously, but the update.py main() function parses options from sys.argv, meaning the Celery task can't configure it on the fly. As part of my patch, I will split out the heavy lifting of main() into do_update() which will accept function parameters. then, main() can call do_update() with appropriate options from sys.argv, and the Celery task can call do_update() with appropriate options from the update view. Does that sound reasonable?
Yep, that sounds reasonable, although if there's anything we can do to mitigate adding a separate interface that would be great (e.g. perhaps with the function we still just pass options as a structure rather than exploding it into separate function parameters - of course that means all callers need to ensure they keep up with any changes to the command line options, but I think we can live with that for now.)
Diana and Paul have work in progress at https://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/celery
Actually the base celery changes and the ability to send layer submission emails in the background using it were already merged. I suspect we should mark this bug resolved and open a new one for additional capabilities.
As per my comment above I have opened bug 12484 to cover triggering updates from the web interface, and am now marking this as resolved.