11197 2017-03-21 04:15:31 +0000 Add ability to trigger asynchronous operations 2018-01-16 02:02:28 +0000 1 1 6 Yocto Project Subprojects Layer Index Layer Index unspecified x86 Multiple RESOLVED FIXED Medium+ enhancement Future 1 bluelightning bluelightning brian.avery libertad.gonzalez.de.la.cruz --- --- No (bug/feature does not impact docs) oldest_to_newest 71503 0 bluelightning 2017-03-21 04:15:31 +0000 At the moment there's no way to trigger asynchronous operations from the web interface. This would be useful for example to trigger one-off manual updates, run the OE-Classic update script, etc. This bug is for the basic ability to trigger such operations from within the code, and then to use it for sending the email to maintainers when a new layer is submitted (since that is currently synchronous, and if an SMTP error occurs it blocks the submission which is really not desirable). FWIW I did look into using Celery to implement this but struggled to figure out how to use it. That doesn't mean Celery isn't the way to do it, but it did seem pretty heavyweight. Then again, we do not want to be implementing such an asynchonous execution framework ourselves. It may also be worth talking to Michael Halstead to ensure a proposed solution will work well for the layers.openembedded.org infrastructure. 71642 1 brian.avery 2017-03-23 15:16:05 +0000 I did a play project using celery+rabbitmq and it was relatively straightforward. https://tests4geeks.com/python-celery-rabbitmq-tutorial/ 73962 2 garbados 2017-06-09 19:43:27 +0000 (In reply to comment #1) > I did a play project using celery+rabbitmq and it was relatively > straightforward. > https://tests4geeks.com/python-celery-rabbitmq-tutorial/ (In reply to comment #0) > At the moment there's no way to trigger asynchronous operations from the web > interface. This would be useful for example to trigger one-off manual > updates, run the OE-Classic update script, etc. > > This bug is for the basic ability to trigger such operations from within the > code, and then to use it for sending the email to maintainers when a new > layer is submitted (since that is currently synchronous, and if an SMTP > error occurs it blocks the submission which is really not desirable). > > FWIW I did look into using Celery to implement this but struggled to figure > out how to use it. That doesn't mean Celery isn't the way to do it, but it > did seem pretty heavyweight. Then again, we do not want to be implementing > such an asynchonous execution framework ourselves. It may also be worth > talking to Michael Halstead to ensure a proposed solution will work well for > the layers.openembedded.org infrastructure. I've got RabbitMQ and Celery up locally, running the update script on command. I'll tie it into the layer submission view and submit the WIP for review. 73966 3 garbados 2017-06-09 21:51:59 +0000 OK, so I got Celery to run the update command asynchronously, but the update.py main() function parses options from sys.argv, meaning the Celery task can't configure it on the fly. As part of my patch, I will split out the heavy lifting of main() into do_update() which will accept function parameters. then, main() can call do_update() with appropriate options from sys.argv, and the Celery task can call do_update() with appropriate options from the update view. Does that sound reasonable? 73975 4 bluelightning 2017-06-12 07:54:17 +0000 Yep, that sounds reasonable, although if there's anything we can do to mitigate adding a separate interface that would be great (e.g. perhaps with the function we still just pass options as a structure rather than exploding it into separate function parameters - of course that means all callers need to ensure they keep up with any changes to the command line options, but I think we can live with that for now.) 78658 5 rebecca.swee.fun.chang 2017-12-11 09:54:05 +0000 Diana and Paul have work in progress at https://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/celery 78659 6 bluelightning 2017-12-11 10:09:52 +0000 Actually the base celery changes and the ability to send layer submission emails in the background using it were already merged. I suspect we should mark this bug resolved and open a new one for additional capabilities. 79081 7 bluelightning 2018-01-16 02:02:28 +0000 As per my comment above I have opened bug 12484 to cover triggering updates from the web interface, and am now marking this as resolved.