Asyncio

nested.py

import asyncio

async def stuff(val):
    print("Going to sleep for", val)
    await asyncio.sleep(val)
    print("slept")
    return val * 2

def do_stuff():
    print("do stuff")
    task = asyncio.ensure_future(stuff(1))
    loop = asyncio.get_event_loop()
    loop.run_until_complete(task)
    print("Task result is", task.result())

async def main():
    print("Start main")
    do_stuff()
    print("Main complete")

output

>>> do_stuff()
do stuff
Going to sleep for 1
slept
Task result is 2

>>> loop = asyncio.get_event_loop()
>>> loop.run_until_complete(main())
Traceback
...
RuntimeError: This event loop is already running

Summarising the related Twitter discussion:

1. You've run into the fundamental "What colour is my function?" problem that afflicts all explicitly async programming models: http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/
2. The first path out is to concede defeat, make evaluate() a coroutine, and require user handlers that want to call it also be coroutines
3. The second path out is to run synchronous user handlers in a separate thread, such that evaluate() can be a synchronous API that uses https://docs.python.org/3/library/asyncio-task.html#asyncio.run_coroutine_threadsafe to delegate the call back to the original event loop
4. Offer both, such that event handlers end up with 3 modes of execution:
   • synchronous in the main thread (can access other widgets without locks, but can't wait for events)
   • coroutine in the main thread (can access other widgets without locks, and can wait for events with await)
   • synchronous in a separate thread (can block waiting for events, but can't directly access or manipulate widget state)

I'm going to suggest that you eventually aim for option 4, and offer two evaluate() APIs:

• WebView.evaluate_async(): the coroutine version that actually does the work in the main thread

• WebView.evaluate(): a synchronous version that does roughly:

   def evaluate(self, *args, **kwds):
       if this_is_the_main_thread():
           raise RuntimeError("Handler must be decorated with '@toga.blocking' to use WebView.evaluate()")
       eval_cr = self.evaluate_async(*args, **kwds)
       eval_future = asyncio.run_coroutine_threadsafe(eval_cr, main_loop)
       return eval_future.result() # This blocks waiting for the main thread
  

To declare blocking synchronous handlers, toga would need to gain a new @toga.blocking decorator that worked something like the following:

def blocking(handler):
    @functools.wraps(handler):
    async def async_wrapper(widget):
        widget_proxy = WidgetProxy(widget)
        sync_call = functools.partial(handler, widget_proxy)
        return await asyncio.get_event_loop().run_in_executor(None, sync_call)
   return async_wrapper

The WidgetProxy helper would be needed to deal with the fact that, while synchronous handlers marked as blocking APIs would gain the freedom to block without blocking the main thread, they'd also gain the restriction of not being allowed to access API widgets directly, since doing so would be a good way to create race conditions.

I'm not familiar enough with toga internals to be able to fully design WidgetProxy here, but the conceptual heart of it would be similar to the suggested synchronous evaluate API above:

async def _method_coroutine(widget, method_name, args, kwds):
    return getattr(widget, method_name)(*args, **kwds)

def _call_widget_method_via_proxy(proxy, method_name, *args, **kwds):
    if this_is_the_main_thread():
        raise RuntimeError("Widget proxies are only needed for handlers decorated with '@toga.blocking'")
    method_cr = _method_coroutine(proxy._widget, args, kwds)
    method_future = asyncio.run_coroutine_threadsafe(method_cr, main_loop)
    return method_future.result() # This blocks waiting for the main thread

@ncoghlan Thanks for that. I was pointed at some vaguely similar code from @dabaez - there's definitely some room here for a decorator that lets us offer an async api, but provide a naïve synchronous interface for ease of use.

I know it's not 100% the same, but crochet for twisted does something like this doesn't it? Just leave the event loop running in another thread and call in/out from it as required?

Conceptually, I don't see a reason why this can't be done. There is an existing event loop. I want to defer to it, and get back control when the work is done. As far as I can make out, this is exactly what asyncio is meant to achieve.

I know Nick has covered all the meaty ground above, but I find this comment interesting. This implies something will block until you get a result back, but blocking a thread that also has the event loop in it means the event loop won't be able to run. I guess it could be possible if you use signals and timers from the OS, but that does mean the event loop won't be running in those blocking times. In fact I'm pretty sure twisted does something like this to get async behaviour on windows.. it does remind me of some stuff @glyph has written about in the past too: https://glyph.twistedmatrix.com/2014/02/unyielding.html

Anyway... interesting conversation :)

@dpnova But I think that WKWebView must be registered with the event loop on the main(UI) thread.

Also, even if this task delegated to another thread, using any synchronous way to get the result will still block the main thread and freeze the GUI.

The intention of the original issue(beeware/toga#68) is that I want to effectively avoid cases like this to freeze the GUI.