Defunct ssh processes and CPU load are kept after executing powershell through ssh on linux

[jwboon-t4e]

Steps to reproduce

Setup powershell ssh on linux system using the following guide: https://docs.microsoft.com/en-us/powershell/scripting/learn/remoting/ssh-remoting-in-powershell-core?view=powershell-7

On the host machine (debian 10) execute the following command.

Invoke-Command -ScriptBlock{return "kaas"} -SSHTransport -HostName ps@localhost -KeyFilePath /home/ps/.ssh/id_rsa

Expected behavior

No defunct processes from SSH on the machine executing PowerShell.

Actual behavior

Each execution of the PowerShell script leaves a defunct ssh process. See example of 'ps aux' below.

On the parent process there remains a CPU load which is unexpected for a server application. This becomes a major issue with a large amount defunct processes which seems to stack the CPU load on the parent process.

root@<machine>:/app# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 3880 1504 ? Ss 09:34 0:00 /bin/bash /entrypoint.sh
root 21 0.1 1.1 3794740 70496 ? Sl 09:34 0:02 dotnet <dll>
root 81 0.0 0.0 4012 3116 pts/0 Ss 09:35 0:00 bash
root 338 0.0 0.0 4012 3196 pts/1 Ss 09:39 0:00 bash
root 1201 0.0 0.4 2968504 27624 pts/0 Sl+ 09:53 0:00 pwsh
root 1209 1.9 2.0 4157756 125756 pts/0 Sl+ 09:53 0:04 dotnet /root/.dotnet/tools/.store/powershell/7.0.3/powershe
root 1240 0.0 0.0 0 0 pts/0 Z+ 09:53 0:00 [ssh] <defunct>
root 1247 0.0 0.0 0 0 pts/0 Z+ 09:53 0:00 [ssh] <defunct>
root 1253 0.0 0.0 0 0 pts/0 Z+ 09:53 0:00 [ssh] <defunct>
root 1267 0.0 0.0 0 0 pts/0 Z+ 09:53 0:00 [ssh] <defunct>
root 1274 0.0 0.0 0 0 pts/0 Z+ 09:53 0:00 [ssh] <defunct>
root 1373 0.3 0.0 0 0 pts/0 Z+ 09:57 0:00 [ssh] <defunct>
root 1377 0.0 0.0 7652 2684 pts/1 R+ 09:57 0:00 ps aux 

Environment data

Host machine:

Name Value
---- -----
PSVersion 7.0.3
PSEdition Core
GitCommitId 7.0.3
OS Linux 4.19.104-microsoft-standard #1 SMP Wed Feb 19 06:37:35 UTC 2020
Platform Unix
PSCompatibleVersions{1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0

Target machine:

Name Value
---- -----
PSVersion 7.0.3
PSEdition Core
GitCommitId 7.0.3
OS Linux 4.19.104-microsoft-standard #1 SMP Wed Feb 19 06:37:35 UTC 2020
Platform Unix
PSCompatibleVersions{1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0

The configuration is that this is a Docker container (host) running a sidecar container (target).

The used image for the host is: 'mcr.microsoft.com/dotnet/core/aspnet:3.1'
The used image for the target is 'mcr.microsoft.com/powershell'

[daxian-dbw]

Quoted from https://askubuntu.com/questions/201303/what-is-a-defunct-process-and-why-doesnt-it-get-killed

"defunct" means the process has either completed its task or has been corrupted or killed, but its child processes are still running or these parent process is monitoring its child process.

Processes marked are dead processes (so-called "zombies") that remain because their parent has not destroyed them properly.

/cc @PaulHigin for insights.

[PaulHigin]

I suspect this is due to the way we create the SSH client process to ignore the termination signal (SIG_IGN):

https://github.com/PowerShell/PowerShell/blob/master/src/System.Management.Automation/engine/remoting/common/RunspaceConnectionInfo.cs#L2220

This was done so that a 'Ctrl+C' keyboard command during a remote session does not terminate the SSH client process (and thus the remote connection), but instead is passed on through the PSRP protocol to stop a running remote command, as expected.

These defunct SSH processes are terminated when the parent pwsh process is terminated. So that is a (not very satisfactory) workaround.

But I am not enough of a Linux expert to know the right way to have PowerShell ensure the process is terminated, on remote connection close. Perhaps someone in the community would know?

GitHub

PowerShell/PowerShell

PowerShell for every system! Contribute to PowerShell/PowerShell development by creating an account on GitHub.

[daxian-dbw]

I found the following from the wiki Zombie process

defunct process is a process that has completed execution (via the exit system call) but still has an entry in the process table: it is a process in the "Terminated state". This occurs for the child processes, where the entry is still needed to allow the parent process to read its child's exit status: once the exit status is read via the wait system call, the zombie's entry is removed from the process table and it is said to be "reaped".

I think we probably just need to call waitpid after the ssh process is terminated. An example from the wiki:

int main(void)

{
 pid_t pids[10];
 int i;

for (i = 9; i >= 0; --i){
 pids[i] = fork();
 if (pids[i] == 0){
 printf("Child%d\n",i);
 sleep(i+1);
 _exit(0);
 }
}

for (i = 9; i >= 0; --i){
 printf("parent%d\n",i);
 waitpid(pids[i], NULL, 0);
 }
return 0;
}

[PaulHigin]

We can experiment with this. I am concerned about calling a blocking function. Since the child process is created without SIG_IGN, I don't know if it will respond to the call, and I hope we don't end up with a hang. I just don't know enough about Linux processes, but we can see what works.

[jwboon-t4e]

Hi guys, any update on this ticket?

[lightcore-x]

I'm experiencing the same issue, any update?

[awakecoding]

We are facing the same issue, PowerShell leaves defunct ssh processes after the connection is closed. We are working on implementing PowerShell remoting over SSH, without SSH (https://blog.devolutions.net/2020/07/powershell-remoting-over-ssh-without-ssh) with a drop-in replacement for the 'ssh' executable. We originally thought this was limited to our custom way of doing it, until we figured out that the real PowerShell remoting over SSH with the OpenSSH client exhibited the same symptoms.

The Devolutions Blog

PowerShell Remoting Over SSH, Without SSH!

A few months ago, PowerShell 7 came out, finally bridging the gap between Windows PowerShell and PowerShell Core. It is now possible to use the same PowerShell everywhere (Windows, macOS, Linux!) whil...

[awakecoding]

I found more information about the Ctrl+C / SIGINT behaviour in a regular SSH session here: https://goteleport.com/blog/ssh-client-signal-handling/

It appears to me that in a regular OpenSSH session, the client does not ignore but forwards the signals to the remote host. This effectively means the OpenSSH client does not process the signal for itself, I tried doing Ctrl+C multiple times locally and it wouldn't kill the OpenSSH client. This appears to be close to the approach taken by PowerShell: it spawns the OpenSSH client process but makes it ignore the signals so the PowerShell client can forward them over PSRP to the remote host.

The main issue I see with the current code is that the defunct ssh processes remain until the parent process (pwsh) is killed. This can be very problematic when spawning multiple PowerShell sessions within the same pwsh process, because you would get a defunct ssh process for each.

Other people have suggested calling waitpid, which is normally a blocking call. There is a WNOHANG flag that can be passed to waitpid, but we'd have to look into how portable this flag is (is it only with a Linux glibc?).

In all cases, I think attempts to fix this would need to be added in the part of code called when the PowerShell remoting session is freed. Can anybody point to where that would be? The code creating the ssh process is easy to find, but it is not so clear to me where the code freeing the PowerShell remoting session for SSH is being called. The goal would be to call some cleanup code at that point in time where we are sure that the ssh process should either be cleaned up if defunct, or force killed if still there.

This procedure would attempt the following, in order:

• waitpid + WNOHANG for a defunct process
• force kill process (SIGKILL) if process is "alive"

Force killing the process is obvious the last resort, but it shouldn't happen under normal circumstances, it's just there to ensure we clean up the process in all possible cases. Most of the time, we would just find the defunct process and clean it up by calling waitpid.

Chasing missing SIGINT signals - SSH

This post is the first of an ongoing series about interesting issues and bugs that the Teleport team has worked on. This post, about missing SIGINTs and SSH, should be interesting for developers who leverage signal handling in terminal-based applications written in Go.

[PaulHigin]

The code already tries to terminate the SSH client process on remote session termination (and other error conditions). We need to investigate why that isn't happening, but I know that .NET 'System.Diagnostics.Process' does not always work the same over different platforms.
https://github.com/PowerShell/PowerShell/blob/master/src/System.Management.Automation/engine/remoting/fanin/OutOfProcTransportManager.cs#L1654

The fact that terminating the parent pwsh process terminates the SSH client process mitigates this somewhat.

GitHub

PowerShell/PowerShell

PowerShell for every system! Contribute to PowerShell/PowerShell development by creating an account on GitHub.

[PaulHigin]

I verified that the clean up code does run. However, the issue is not terminating the SSH process, it is terminated, but the fact that it still exists as a 'zombie' process to the parent process as @daxian-dbw describes above. We can try using waitpid during clean up as Dongbo suggests above, but again my concern with that is hanging PowerShell during the wait.

[awakecoding]

@PaulHigin waitpid + the WNOHANG flag should make the call non-blocking. I think this flag may be Linux-specific, but it can't hurt to try it out. Here is an example: https://stackoverflow.com/a/7255053

Stack Overflow

Example of waitpid, WNOHANG, and SIGCHLD

I need an example of waitpid, WNOHANG and SIGCHLD combined in C, and how I can use them all with fore\background?

signal( SIGCHLD, SIG_IGN );

waitpid(child, status, 0);

[dinhngtu]

After looking a bit into this, it seems like OutOfProcTransportManager (and therefore RunspaceConnectionInfo) uses its own implementation of fork/exec in libpsl for spawning SSH instead of using System.Diagnostics.Process. I'm not sure why it's done this way so could someone chime in?

Regardless, the easiest fix for this would be to fix System.Diagnostics.Process.WaitForExit() to call waitpid() for unmanaged processes, and then calling WaitForExit() after killing SSH.