A kubectl plugin to display the pods and containers which have recently been OOMKilled.
Via krew
kubectl krew install oomd kubectl oomd Or manually from the releases page, for example with linux_amd64
# Download the release for your OS wget https://github.com/jdockerty/kubectl-oomd/releases/download/v0.0.7/oomd_linux_amd64.tar.gz # Extract files from the archive tar -xvf oomd_linux_amd64.tar.gz # Move the binary into your PATH, renaming the application to 'kubectl-oomd' to# satisfy the plugin convention mv oomd "${HOME}"/.local/bin/kubectl-oomd # Run the plugin kubectl oomdRunning the command will display the pods that have recently been OOMKilled in your current namespace. This also shows the specific container which was killed too, helpful in the case of multi-container pods.
kubectl oomd POD CONTAINER REQUEST LIMIT TERMINATION TIME my-app-5bcbcdf97-722jp infoapp 1G 8G 2022-11-07 13:03:49 +0000 GMT my-app-5bcbcdf97-7j5rd infoapp 1G 8G 2022-11-07 14:35:34 +0000 GMT my-app-5bcbcdf97-k8g8g infoapp 1G 8G 2022-11-07 14:35:02 +0000 GMT my-app-5bcbcdf97-mf65j infoapp 1G 8G 2022-11-07 14:34:57 +0000 GMT You can specify another namespace, as you would with other kubectl commands or use --all-namespaces/-A to check against them all.
kubectl oomd -n oomkilled POD CONTAINER REQUEST LIMIT TERMINATION TIME my-app-5bcbcdf97-722jp infoapp 1G 8G 2022-11-07 13:03:49 +0000 GMT my-app-5bcbcdf97-7j5rd infoapp 1G 8G 2022-11-07 14:35:34 +0000 GMT my-app-5bcbcdf97-k8g8g infoapp 1G 8G 2022-11-07 14:35:02 +0000 GMT my-app-5bcbcdf97-mf65j infoapp 1G 8G 2022-11-07 14:34:57 +0000 GMT kubectl oomd --no-headers my-app-5bcbcdf97-722jp infoapp 1G 8G 2022-11-07 13:03:49 +0000 GMT my-app-5bcbcdf97-7j5rd infoapp 1G 8G 2022-11-07 14:35:34 +0000 GMT my-app-5bcbcdf97-k8g8g infoapp 1G 8G 2022-11-07 14:35:02 +0000 GMT my-app-5bcbcdf97-mf65j infoapp 1G 8G 2022-11-07 14:34:57 +0000 GMT Experimental sorting is enabled through the --sort-field flag. By default, this is none. At the moment, only time is supported which sorts by termination time of containers, this is mainly useful in larger outputs across all namespaces (-A), used in conjunction with a pipe to tail. A simple example of this in action is shown below.
# The default with no sorting. kubectl oomd -n tracing POD CONTAINER REQUEST LIMIT TERMINATION TIME jaeger-agent-4k845 jaeger-agent 100Mi 100Mi 2022-11-11 21:06:31 +0000 GMT jaeger-agent-j5vb8 jaeger-agent 100Mi 100Mi 2022-11-09 23:20:38 +0000 GMT # Most recently OOMKilled pods are shown first kubectl oomd -n tracing --sort-field time POD CONTAINER REQUEST LIMIT TERMINATION TIME jaeger-agent-j5vb8 jaeger-agent 100Mi 100Mi 2022-11-09 23:20:38 +0000 GMT jaeger-agent-4k845 jaeger-agent 100Mi 100Mi 2022-11-11 21:06:31 +0000 GMT If you wish to force some OOMKilled pods for testing purposes, you can use oomer or simply run
kubectl apply -f https://raw.githubusercontent.com/jdockerty/oomer/main/oomer.yaml This will create the oomkilled namespace and a Deployment with pods that continually exit with code 137, in order to be picked up by oomd.