munet: add Commander.async_spawn()

[liambrady]

spawn(), which conducts a send/expect process is
blocking due to the abscence of async_=True to
pexpect.expect() within the method's main
send/expect loop. Some async methods call
spawn(), however, which leads to such methods
being blocked.

This is particularly problematic in cases where
multiple VMs requiring custom console expect/send
prompts are required (e.g. routers). Since each
QEMU VM sets up both console logging and completes
an expect/send loop without yielding control to
the coroutine of the other QEMU VMs, only one VM
can ever have logging configured (and thus advance in
an expect/send loop) at any given time. Not
only is this inefficient given that all QEMU VMs
are running and waiting for input, but this can
be fatal if console logging is not set up on each
VM in time. If some mandatory console output is
missed, then there is no way to recover and the
setup of the QEMU VM will time out.

However, setting async_=True in pexpect.expect()
leads to errors when mixed with PopenSpawn.
To bypass this issue, we do not use pexpect.expect()
with async_=True, but repeatedly call
pexpect.expect() with a timeout set at 0.1 (Until
a timeout that we track expires).

The end result of this change is that both logging is set
up on all QEMU nodes immediately and that
independent progress can be made in each VM's
expect/send loop simultaneously.

[codecov]

Codecov Report

❌ Patch coverage is 89.85507% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.12%. Comparing base (b310137) to head (ed8bb5d).
⚠️ Report is 9 commits behind head on main.

Files with missing lines	Patch %	Lines
munet/base.py	89.70%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #58      +/-   ##
==========================================
+ Coverage   59.66%   60.12%   +0.46%     
==========================================
  Files          19       19              
  Lines        5781     5831      +50     
==========================================
+ Hits         3449     3506      +57     
+ Misses       2332     2325       -7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[choppsv1]

As discussed elsewhere, we need to use a new async_spawn() variant for this to preserve the existing non-asyncio API. The use case you're interested in is coming from an async function so should be fine to just switch to the new variant inside console/shell_spawn.

Also, since async/await only fails with non-pty pexpect.popen_spawn (i.e., pipes) you should restrict the busy loop fix to only that case. For use_pty == True use the supported async from pexpect.

[liambrady]

As discussed elsewhere, we need to use a new async_spawn() variant for this to preserve the existing non-asyncio API. The use case you're interested in is coming from an async function so should be fine to just switch to the new variant inside console/shell_spawn.

I modified both console/shell_spawn and launch/monitor to use the new async variant (both methods are already async and so should use the variant), and now spawn() appears to have fallen out of the code coverage. It appears that commander.spawn() is not called anywhere else within the codebase.

Does this affect the decision to continue to maintain commander.spawn()? Do we need it for future additions to the munet API? It doesn't feel correct to leave dead code within the project (or write new tests to maintain a section of code that isn't being meaningfully used.)

[choppsv1]

Need to test both async and non-async variants, and the vty and non-vty case.

[liambrady]

linted new tests