Data Plane API is killed by SIGTERM in openshift

[git001]

Introduction

I try to run haproxy in front of craftcms and want to use the Data Plane API for management.

Data Plane API infos

This is the dataplane api version.

curl -vLo dataplaneapi-haproxy-v2.9.1.tar.gz \
 https://github.com/haproxytech/dataplaneapi/archive/refs/tags/v2.9.1.tar.gz

Due to the fact that the trace output of the dataplaneapi binary was not very helpful have I added this lines to the code which creates the output below.

original code

dataplaneapi/client-native/cn.go

Lines 95 to 103 in 14ab8aa

	} else{
	// if nbproc is not set, use master socket with 1 process
	ms := runtime_options.MasterSocket(masterSocket, 1)
	runtimeClient, err = runtime_api.New(ctx, mapsDir, ms)
	if err == nil{
	return runtimeClient
	}
	log.Warningf("Error setting up runtime client with master socket: %s : %s", masterSocket, err.Error())
	}

my "patch"

# client-native/cn.go:96
			} else{
				// if nbproc is not set, use master socket with 1 process
				out, myerr := exec.Command("ls", "-la", "/data/haproxy/run/master-socket").Output()

				if myerr != nil{
					fmt.Printf("%s", myerr)
				}

				fmt.Println("Command Successfully Executed")
				output := string(out[:])
				fmt.Println(output)

				ms := runtime_options.MasterSocket(masterSocket, 1)
				runtimeClient, err = runtime_api.New(ctx, mapsDir, ms)
				if err == nil{
					return runtimeClient
				}
				log.Warningf("Error setting up runtime client with master socket (1): %s : %s", masterSocket, err.Error())
			}

That's the output when I run the HAProxy with the dataplane api

alex@CPC-aleks-RW2GP on 02/04/2024 at 22:49:17_UTC /mnt/c/local_data/git-repos/craftcms_k8s$ oc -n craftcms logs craftcms-hap-b87b89874-ppssv
[NOTICE] (1) : New program 'api' (8) forked
[NOTICE] (1) : New worker (9) forked
[NOTICE] (1) : Loading success.
[WARNING] (9) : fcgi-servers/craftcms1 changed its IP from (none) to 10.129.2.11 by kube-dns/dns1.
fcgi-servers/craftcms1 changed its IP from (none) to 10.129.2.11 by kube-dns/dns1.
[WARNING] (9) : Server fcgi-servers/craftcms1 ('craftcms-php.craftcms.svc.cluster.local') is UP/READY (resolves again).
Server fcgi-servers/craftcms1 ('craftcms-php.craftcms.svc.cluster.local') is UP/READY (resolves again).
[WARNING] (9) : Server fcgi-servers/craftcms1 administratively READY thanks to valid DNS answer.
Server fcgi-servers/craftcms1 administratively READY thanks to valid DNS answer.
configuration file /data/haproxy/etc/dataplaneapi.yaml does not exists, creating one

time="2024-04-02T22:49:16Z" level=info msg="Build from: "
time="2024-04-02T22:49:16Z" level=info msg="HAProxy Data Plane API .dev.dirty"
time="2024-04-02T22:49:16Z" level=info msg="Build date: 2024-04-02T22:45:03Z"
time="2024-04-02T22:49:16Z" level=info msg="Reload strategy: custom"
Command Successfully Executed
srwxr-xr-x. 1 1000940000 root 0 Apr 2 22:49 /data/haproxy/run/master-socket

time="2024-04-02T22:49:16Z" level=warning msg="Error setting up runtime client with master socket (1): /data/haproxy/run/master-socket;sockpair@7 : dial unix /data/haproxy/run/master-socket;sockpair@7: connect: no such file or directory"

[NOTICE] (1) : haproxy version is 2.9.6-9eafce5
[NOTICE] (1) : path to executable is /usr/local/sbin/haproxy
[ALERT] (1) : Current program 'api' (8) exited with code 1 (Exit)
[ALERT] (1) : exit-on-failure: killing every processes with SIGTERM
[ALERT] (1) : Current worker (9) exited with code 143 (Terminated)
[WARNING] (1) : All workers exited. Exiting... (1)

My observations

This line confuses me

time="2024-04-02T22:49:16Z" level=warning \
msg="Error setting up runtime client with master socket (1): \
 /data/haproxy/run/master-socket;sockpair@7 : \
 dial unix /data/haproxy/run/master-socket;sockpair@7: \
 connect: no such file or directory"

because the ls before the command runtime_api.New(...) shows that the socket is there.

srwxr-xr-x. 1 1000940000 root 0 Apr 2 22:49 /data/haproxy/run/master-socket

and I can execute the help command on the master socket

alex@CPC-aleks-RW2GP on 02/04/2024 at 23:11:02_UTC /mnt/c/local_data/git-repos/craftcms_k8s$ oc -n craftcms rsh --shell /bin/bash craftcms-hap-b87b89874-ddxs8
groups: cannot find name for group ID 1000940000
1000940000@craftcms-hap-b87b89874-ddxs8:/$ echo "help"|socat /data/haproxy/run/master-socket -
The following commands are valid at this level:
 @!<pid> : send a command to the <pid> process
 @<relative pid> : send a command to the <relative pid> process
 @master : send a command to the master process
 hard-reload : achieve a hard-reload (-st) of haproxy
 operator : lower the level of the current CLI session to operator
 reload : achieve a soft-reload (-sf) of haproxy
 show cli level : display the level of the current CLI session
 show cli sockets : dump list of cli sockets
 show proc : show processes status
 show startup-logs : report logs emitted during HAProxy startup
 show version : show version of the current process
 user : lower the level of the current CLI session to user
 help [<command>] : list matching or all commands
 prompt [timed] : toggle interactive mode with prompt
 quit : disconnect

My assumption is that dataplaneapi tries to connect to /data/haproxy/run/master-socket;sockpair@7 which of course does not exist.

Important

When is the ;sockpair@7 added to the master-socket?

haproxy infos

haproxy run

This is how the haproxy is started.

oc -n craftcms exec craftcms-hap-b87b89874-ddxs8 -- ps axf
 PID TTY STAT TIME COMMAND
 24 ? Rs 0:00 ps axf
 1 ? Ss 0:00 haproxy -f /data/haproxy/etc/haproxy.cfg -db -W -S /data/haproxy/run/master-socket
 8 ? Sl 0:01 haproxy -f /data/haproxy/etc/haproxy.cfg -db -W -S /data/haproxy/run/master-socket

haproxy config

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
 log stdout format raw daemon debug
 pidfile /data/haproxy/run/haproxy.pid
 master-worker
 stats socket /data/haproxy/run/stats mode 660 level admin expose-fd listeners

resolvers kube-dns
 nameserver dns1 dns-default.openshift-dns.svc.cluster.local:53
 accepted_payload_size 4096
 resolve_retries 3
 timeout resolve 1s
 timeout retry 1s
 hold other 30s
 hold refused 30s
 hold nx 30s
 hold timeout 30s
 hold valid 10s
 hold obsolete 30s

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
 mode http
 balance leastconn
 log global
 option httplog
 option dontlognull
 option log-health-checks
 option redispatch
 retries 3
 timeout http-request 10s
 timeout queue 1m
 timeout connect 10s
 timeout client 1m
 timeout server 1m
 timeout http-keep-alive 10s
 timeout check 10s

userlist haproxy-dataplaneapi
 user admin insecure-password mypassword
#
program api
 command /usr/bin/dataplaneapi -f=/data/haproxy/etc/dataplaneapi.yaml --log-to=stdout --log-level=trace --spoe-dir=/data/haproxy/spoe --maps-dir=/data/haproxy/maps --ssl-certs-dir=/data/haproxy/ssl --general-storage-dir=/data/haproxy/general --host 0.0.0.0 --port 5555 --haproxy-bin /usr/sbin/haproxy --config-file /data/haproxy/etc/haproxy.cfg --reload-cmd "kill -SIGUSR2 1" --restart-cmd "kill -SIGUSR2 1" --reload-delay 5 --userlist haproxy-dataplaneapi --socket-path=/data/haproxy/run/data-plane.sock
 no option start-on-reload

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend craft-cms
 bind *:8080

 tcp-request inspect-delay 5s
 tcp-request content accept if HTTP

 monitor-uri /health
 http-request deny if{path_sub -i %0a %0d }
 http-request deny if{hdr_len(content-length) 0 }
 http-request del-header Proxy
 http-request set-header Host %[req.hdr(Host),lower]

 acl exist-php-ext path_sub -i .php
 http-request set-path /index.php%[path] if !exist-php-ext !{path_end .php }

 http-response set-header Strict-Transport-Security "max-age=16000000; includeSubDomains; preload;"

 default_backend fcgi-servers

listen stats
 bind *:1936
 monitor-uri /healthz
 http-request use-service prometheus-exporter if{path /metrics }
 stats enable
 stats uri /

backend fcgi-servers

 option httpchk
 http-check connect proto fcgi
 http-check send meth GET uri /fpm-ping
 
 use-fcgi-app php-fpm

 # https://www.haproxy.com/blog/circuit-breaking-haproxy
 server-template craftcms 5 craftcms-php.craftcms.svc.cluster.local:9000 proto fcgi check resolvers kube-dns init-addr none observe layer7 error-limit 5 on-error mark-down inter 10s rise 30 slowstart 40s

fcgi-app php-fpm
 log-stderr global
 option keep-conn
 option mpxs-conns
 option max-reqs 10

 docroot /app/web
 index index.php
 path-info ^(/.+\.php)(/.*)?$

Output of haproxy -vv

haproxy -vv
HAProxy version 2.9.6-9eafce5 2024/02/26 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2025.
Known bugs: http://www.haproxy.org/bugs/bugs-2.9.6.html
Running on: Linux 5.14.0-284.52.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jan 30 08:35:38 EST 2024 x86_64
Build options :
 TARGET = linux-glibc
 CPU = generic
 CC = cc
 CFLAGS = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
 OPTIONS = USE_PTHREAD_EMULATION=1 USE_LINUX_TPROXY=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_TFO=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_QUIC_OPENSSL_COMPAT=1
 DEBUG = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX +PTHREAD_EMULATION +QUIC +QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
 bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=8).
Built with OpenSSL version : OpenSSL 3.0.2 15 Mar 2022
Running on OpenSSL version : OpenSSL 3.0.2 15 Mar 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with Lua version : Lua 5.4.4
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.39 2021-10-29
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 11.4.0

Available polling systems :
 epoll : pref=300, test result OK
 poll : pref=200, test result OK
 select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
 quic : mode=HTTP side=FE mux=QUIC flags=HTX|NO_UPG|FRAMED
 h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
 fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
 <default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
 h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
 <default> : mode=TCP side=FE|BE mux=PASS flags=
 none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG

Available services : prometheus-exporter
Available filters :
 [BWLIM] bwlim-in
 [BWLIM] bwlim-out
 [CACHE] cache
 [COMP] compression
 [FCGI] fcgi-app
 [SPOE] spoe
 [TRACE] trace

Dockerfile

That's the Dockerfile of the image.

FROM haproxytech/haproxy-ubuntu:2.9

COPY container-files/ /

RUN set -x \
 && cp /usr/bin/dataplaneapi /usr/bin/dataplaneapi.orig \
 && cp /data/haproxy/bin/dataplaneapi /usr/bin/dataplaneapi \
 && mkdir -p /data/haproxy/etc \
 /data/haproxy/run \
 /data/haproxy/maps \
 /data/haproxy/ssl \
 /data/haproxy/general \
 /data/haproxy/spoe \
 && chown -R 1001:0 /data \
 && chmod -R g=u /data \
 && /usr/bin/dataplaneapi -v

USER 1001

[mjuraga]

Hi, can you paste the output of: /data/haproxy/etc/dataplaneapi.yaml?

[git001]

that's it

$ cat /data/haproxy/etc/dataplaneapi.yaml
config_version: 2
name: craftcms-hap-debug
mode: single
status: ""
dataplaneapi:
 advertised:
 api_address: ""
 api_port: 0
haproxy:
 reload:
 reload_strategy: custom

[git001]

Maybe there is another issue and the message above hides the original problem.
As OpenShift restricts the Pod run environment could the socket call fail.

https://github.com/haproxytech/client-native/blob/e914b0d0f77265cc83cb61eea069dea75c38a706/runtime/runtime_single_client.go#L209-L223

Which, I think, is called from here.

https://github.com/haproxytech/client-native/blob/e914b0d0f77265cc83cb61eea069dea75c38a706/runtime/runtime_single_client.go#L62-L80

[mjuraga]

Hi @git001, thanks for your input, but the bug is in the dataplaneapi code, it picks up the master socket location from a env variable set by the HAProxy: https://github.com/haproxytech/dataplaneapi/blob/master/configure_data_plane.go#L121 and it doesn't properly sanitize the socket location (doesn't remove sockpair@ suffix added by the latest versions of HAProxy). We can fix this, or you could give it a shot if that is interesting for you.

[git001]

@mjuraga thanks for the tip. I will try to fix it with a PR

[git001]

I have now fixed the sockpair@ bug with this code.

...
	// Override options with env variables
	if os.Getenv("HAPROXY_MWORKER") == "1"{
		mWorker = true
		masterRuntime := os.Getenv("HAPROXY_MASTER_CLI")
		if misc.IsUnixSocketAddr(masterRuntime){

			fmt.Printf("before Replace masterRuntime :%v:\n", masterRuntime)

			if strings.HasPrefix(masterRuntime, "unix@"){
				haproxyOptions.MasterRuntime = strings.Replace(masterRuntime, "unix@", "", 1)
				if strings.Contains(haproxyOptions.MasterRuntime, "sockpair@"){
					semikolon := strings.Index(haproxyOptions.MasterRuntime, ";")
					haproxyOptions.MasterRuntime = haproxyOptions.MasterRuntime[:semikolon]
				}
			}

			fmt.Printf("after Replace masterRuntime :%v:\n", haproxyOptions.MasterRuntime)
		}
	}
...

From the output below is shown that the socketpair is gone.

before Replace masterRuntime :unix@/data/haproxy/run/master-socket;sockpair@7:
after Replace masterRuntime :/data/haproxy/run/master-socket:

The problem with the SIGTERM is still there.

 haproxy -f /data/haproxy/etc/haproxy.cfg -db -W -S /data/haproxy/run/master-socket
[NOTICE] (9) : New program 'api' (11) forked
[NOTICE] (9) : New worker (12) forked
[NOTICE] (9) : Loading success.
[WARNING] (12) : fcgi-servers/craftcms1 changed its IP from (none) to 10.129.2.11 by kube-dns/dns1.
fcgi-servers/craftcms1 changed its IP from (none) to 10.129.2.11 by kube-dns/dns1.
[WARNING] (12) : Server fcgi-servers/craftcms1 ('craftcms-php.craftcms.svc.cluster.local') is UP/READY (resolves again).
Server fcgi-servers/craftcms1 ('craftcms-php.craftcms.svc.cluster.local') is UP/READY (resolves again).
[WARNING] (12) : Server fcgi-servers/craftcms1 administratively READY thanks to valid DNS answer.
Server fcgi-servers/craftcms1 administratively READY thanks to valid DNS answer.
configuration file /data/haproxy/etc/dataplaneapi.yaml does not exists, creating one
time="2024-04-03T11:26:23Z" level=info msg="HAProxy Data Plane API .dev.dirty"
time="2024-04-03T11:26:23Z" level=info msg="Reload strategy: custom"
time="2024-04-03T11:26:23Z" level=info msg="Build from: "
time="2024-04-03T11:26:23Z" level=info msg="Build date: 2024-04-03T11:23:50Z"

before Replace masterRuntime :unix@/data/haproxy/run/master-socket;sockpair@7:
after Replace masterRuntime :/data/haproxy/run/master-socket:

Command Successfully Executed
srwxr-xr-x. 1 1000950000 root 0 Apr 3 11:26 /data/haproxy/run/master-socket

ms :{/data/haproxy/run/master-socket 1}: masterSocket :/data/haproxy/run/master-socket: mapsDir :{/data/haproxy/maps}:
[NOTICE] (9) : haproxy version is 2.9.6-9eafce5
[NOTICE] (9) : path to executable is /usr/local/sbin/haproxy
[ALERT] (9) : Current program 'api' (11) exited with code 1 (Exit)
[ALERT] (9) : exit-on-failure: killing every processes with SIGTERM
[ALERT] (9) : Current worker (12) exited with code 143 (Terminated)
[WARNING] (9) : All workers exited. Exiting... (1)

As mentioned in #329 (comment) could it be that the socket call in client-native be another issue?

[mjuraga]

For test could you try running dpapi standalone, without running from the program section.

[git001]

Any chance to merge the PR #330 ?