From what I've read in this comment (issue #264), it seems like the idea of:
- allowing yield inside a trio nursery
- AND allowing it to cancel the current task regardless of its current position in the code
is out of the picture. The main argument against that is presented in this comment (issue 638):
And the issues with nurseries are actually worse: if you can yield out of a nursery block, you can actually recreate the equivalent of a go statement by mistake.
In my opinion, yielding doesn't have to be about "getting out": with careful flow control, it can be seen as stepping inside a controlled scope. First we will see how the current implementation of asynccontextmanager deal with that, and then I will try to show that the same reasoning can be applied to async generators in general, although there are a few limitations.
Consider this generic use case of an async context manager that contains a nursery:
@asynccontextmanager
async def manager():
with open_nursery() as nursery:
[...] # 1
yield
[...] # 3
async def main():
async with manager() as m:
[...] # 2The execution flow looks roughly like this:
|
| Enter an async context manager
|
|_____ anext ______
|
| Enter nursery (1)
______ yield ______|
|
|
| Might raise an exception (2)
|
| Might get cancelled by nursery
|
| Exit the async context manager
|
|__ anext/athrow __
|
| Exit nursery (3)
__ return/raise ___|
|
|
V
The two horizontal lines represent the main scope and the manager scope. Here it might look like the nursery is leaking out of the generator, but with a bit of unfolding, 3 distinct nested scopes appear:
|
| Enter an async context manager
|
|_____ anext ______
|
| Enter nursery (1)
|
|______ yield ______
|
| Might raise an exception (2)
|
| Might get cancelled by nursery
|
| Exit the async context manager
__ anext/athrow ___|
|
| Exit nursery
_ return/raise ___|
|
|
V
The first scope is the main coroutine, the second scope is the async generator and the third scope is the code within the context manager. Since they're all properly nested it is fine for the nursery to cancel the task, even though it looks like it's running code "outside" of the generator. The reason why this unfolding is conceptually correct is because the asynccontextmanager re-throw any exception it gets within the underlying generator using athrow (see the implementation).
Now let's try to apply the same unfolding to asynchronous generators. In order to so, we need a context manager to control the execution of the async generator using a technique similar to asynccontextmanager. Consider the following implementation of an aitercontext context manager:
@asynccontextmanager
async def aitercontext(resource):
aiterator = resource.__aiter__()
try:
yield aiterator
except BaseException as exc:
if hasattr(aiterator, 'athrow'):
await aiterator.athrow(exc)
raise
finally:
if hasattr(aiterator, 'aclose'):
await aiterator.aclose()Now let's try to apply to some generic piece of code:
async def agen():
with open_nursery() as nursery:
[...] 1
yield item
[...] 2
yield item
[...] 3
async def main():
async with aitercontext(agen()) as safe_agen:
async for item in sage_agen:
[...] (X)The execution flow should look something like this:
|
| Enter an async iterator context
|
| Get first item
|
|_____ anext ______
|
| Enter nursery (1)
|
| Produce item
______ yield ______|
|
|
| Process item (X)
|
| Get next item
|
|_____ anext ______
|
|
| Produce item (2)
|
______ yield ______|
|
| Process item (X)
|
| [...]
|
| Might raise an exception
|
| Might get cancelled by nursery
|
|__ anext/athrow __
|
| No more item to be produced (3)
|
| Exit nursery
|
__ return/raise ___|
|
|
V
And applying the same unfolding used previously for the async context manager:
|
| Enter an async iterator context
|
| Get first item
|
|_____ anext ______
|
| Enter nursery
|
| Produce item (1)
|
|______ yield ______
|
|
| Process item (X)
|
| Get next item
|
_____ anext _______|
|
|
| Produce item (2)
|
|______ yield ______
|
| Process item (X)
|
| [...]
|
| Might raise an exception
|
| Might get cancelled by nursery
__ anext/athrow ___|
|
| No more item to be produced (3)
|
| Exit nursery
_ return/raise ___|
|
|
V
Again the first scope is the main coroutine, the second scope is the async generator and the third scope is the code within the async for block. And again the nursery is expected to be able to cancel the task especially if it is running some code within the third scope.
However, I purposely ignored a pretty important issue: the 3 scopes are not so clearly defined this time. In particular, those two code sections are problematic:
async def main():
async with aitercontext(agen()) as safe_agen:
[...] # It looks like this section belongs to scope 3 but it's actually scope 1
async for item in sage_agen:
[...] # Definitely scope 3
[...] # It's undefined whether this section belongs to scope 1 or 3I have to admit I'm not sure what to do about those, maybe someone can come up with a smart way to deal with this issue (PEP 533 maybe?). In any case the execution is still protected by the context manager so there should be no way for the nursery to leak outside of it.