Prevent Paramiko deadlock when test sends more than 2MB to stdout

[jburgess777]

The paramiko library has a known problem where checking the exit status can cause a deadlock if the command has written a lot of output to the stdout (or stderr) channel. We hit this problem when using testinfra to return data from log files during a test.

https://docs.paramiko.org/en/stable/api/channel.html#paramiko.channel.Channel.recv_exit_status

We can work around this, for the stdout case, by reading the data before checking the exit status.

I do not attempt to fix the stderr case. Fixing this would likely require a more complex fix, which reads from both stdout and stderr until they are closed, like https://stackoverflow.com/a/78765054

Here is an example of running the test case on the original code. I enabled pytest-timeouts with a 10 second timeout so you can see it stuck in recv_exit_status

$ tox -- -v -k test_command_deadlock test ========================================================================================= test session starts =========================================================================================== platform linux -- Python 3.10.12, pytest-8.3.3, pluggy-1.5.0 -- /home/jburgess/git/pytest-testinfra/.tox/py/bin/python cachedir: .tox/py/.pytest_cache rootdir: /home/jburgess/git/pytest-testinfra configfile: setup.cfg plugins: testinfra-0.1.dev789, cov-5.0.0, xdist-3.6.1, timeouts-1.2.1 setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s collected 177 items / 171 deselected / 6 selected tes=t/test_backends.py::test_command_deadlock[ssh://debian_bookworm] PASSED [ 16%] test/test_backends.py::test_command_deadlock[safe-ssh://debian_bookworm] PASSED [ 33%] test/test_backends.py::test_command_deadlock[docker://debian_bookworm] PASSED [ 50%] test/test_backends.py::test_command_deadlock[paramiko://debian_bookworm] FAILED [ 66%] test/test_backends.py::test_command_deadlock[ansible://debian_bookworm] PASSED [ 83%] test/test_backends.py::test_command_deadlock[ansible://debian_bookworm?force_ansible=True] PASSED [100%] ================================================================================================ FAILURES ================================================================================================ ___________________________________________________________________________ test_command_deadlock[paramiko://debian_bookworm] ____________________________________________________________________________ host = <testinfra.host.Host paramiko://debian_bookworm> @pytest.mark.testinfra_hosts(*HOSTS) @pytest.mark.execution_timeout(10) def test_command_deadlock(host): # Test for deadlock when exceeding Paramiko transport buffer (2MB) # https://docs.paramiko.org/en/latest/api/channel.html#paramiko.channel.Channel.recv_exit_status size = 3 * 1024 * 1024 > output = host.check_output(f"python3 -c 'print(\"a\" *{size})'") test/test_backends.py:651: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ testinfra/host.py:98: in run return self.backend.run(command, *args, **kwargs) testinfra/backend/paramiko.py:150: in run rc, stdout, stderr = self._exec_command(cmd) testinfra/backend/paramiko.py:141: in _exec_command rc = chan.recv_exit_status() .tox/py/lib/python3.10/site-packages/paramiko/channel.py:400: in recv_exit_status self.status_event.wait() /usr/lib/python3.10/threading.py:607: in wait signaled = self._cond.wait(timeout) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

With the fix, the test passes

========================================================================================== test session starts =========================================================================================== platform linux -- Python 3.10.12, pytest-8.3.3, pluggy-1.5.0 -- /home/jburgess/git/pytest-testinfra/.tox/py/bin/python cachedir: .tox/py/.pytest_cache rootdir: /home/jburgess/git/pytest-testinfra configfile: setup.cfg plugins: testinfra-0.1.dev789, cov-5.0.0, xdist-3.6.1, timeouts-1.2.1 setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s collected 177 items / 171 deselected / 6 selected test/test_backends.py::test_command_deadlock[ssh://debian_bookworm] PASSED [ 16%] test/test_backends.py::test_command_deadlock[safe-ssh://debian_bookworm] PASSED [ 33%] test/test_backends.py::test_command_deadlock[docker://debian_bookworm] PASSED [ 50%] test/test_backends.py::test_command_deadlock[paramiko://debian_bookworm] PASSED [ 66%] test/test_backends.py::test_command_deadlock[ansible://debian_bookworm] PASSED [ 83%] test/test_backends.py::test_command_deadlock[ansible://debian_bookworm?force_ansible=True] PASSED [100%] =================================================================================== 6 passed, 171 deselected in 4.24s ==================================================================================== py: OK ✔ in 5.43 seconds 

I didn't see any regressions in the rest of the test suite.

[philpep]

Merged, thanks for digging into this!