HTTPSever.handle_request not respecting timeout as documented between first and second request

[eXhumer]

Bug report

The HTTPServer times out as expected before receiving the first request and after receiving the second request. However, HTTPServer doesn't timeout as expected between the first and second request, blocking until the second request is received by HTTPServer.

Your environment

• CPython versions tested on: 3.11.1
• Operating system and architecture: Windows 11 22H2 x64

Example code demonstrating the issue

from http.server import BaseHTTPRequestHandler, HTTPServer


class HelloWorldHTTPRequestHandler(BaseHTTPRequestHandler):
 def do_GET(self):
 self.send_response(200)
 self.send_header("Content-Type", "text/plain")
 self.end_headers()
 self.wfile.write("Hello World!".encode("utf8"))


if __name__ == "__main__":
 httpd = HTTPServer(("localhost", 65012), HelloWorldHTTPRequestHandler)
 httpd.timeout = 1

 while True:
 try:
 print("handle_request")
 httpd.handle_request()

 except KeyboardInterrupt:
 break

Example code output

handle_request
handle_request
handle_request
handle_request
127.0.0.1 - - [22/Feb/2023 14:49:14] "GET / HTTP/1.1" 200 -
handle_request
127.0.0.1 - - [22/Feb/2023 14:49:29] "GET / HTTP/1.1" 200 -
handle_request
handle_request
handle_request
handle_request
handle_request
handle_request
handle_request
handle_request
handle_request
KeyboardInterrupt

Linked PRs

• gh-102156: Fix HTTPServer timeout ignored in keep-alive connections #137806

[longvhdeveloper]

@corona10 I'll take it

[corona10]

@corona10 I'll take it

Please go ahead

[longvhdeveloper]

@corona10 I debug and see problem in here socketserver.py of https://github.com/python/cpython/blob/main/Lib/socketserver.py , code example above will call method "handle_timeout", but inside of it just pass
https://github.com/python/cpython/blob/main/Lib/socketserver.py#L326

I think if we want handle it we can edit method as below

try:
 request, client_address = self.get_request()
 self.finish_request(request, client_address)
 except OSError:
 pass

How do you think about it ?

[corona10]

@longvhdeveloper
Thank you for hardwork! I will take a soon :)

[nipil]

Hello, any update on this issue ?

I reproduced similar problems with the sample code below (using 3.11.9) with SIGINT trying to trigger a clean shutdown.

• using handle_request() with a self.timeout set :

1. if i SIGINT before sending any request, handle_request() timeouts fine, and everything shuts down properly
2. if i send any amount of request in a browser, before i SIGINT
   2a. handle_request() timeouts do not happen anymore, and i cannot shut down :
   2b. only if i refresh browser (same tcp connection) then handle_request() processes a last request and returns
   2c. or if i close the browser (same tcp donnection) then handle_request() returns directly

• using serve_forever() (with no self.timeout!) and a self.shutdown() from another thread

1. if i SIGINT before sending any request, self.shutdown() stops serve_forever() fine, and it shuts down properly
2. if i send any amount of request in a browser, before i SIGINT
   2a. self.shutdown() does not return, and i cannot shut down :
   2b. only if i refresh browser then serve_forever() does last request and self.shutdown() returns
   2c. or if i close the browser then serve_forever() returns directly, and self.shutdown() returns

So both steps 2b and 2c indicate that the server is held "pending" due to an (idle ?) established TCP connection with a client. As far as i understand, there is something preventing the server from dropping the active connection for successful shutdown. I believe that a server should not be held running due to client-established connections, and should be able to decide on its own to shut down, so the above behaviour is not really acceptable.

PS: i tried to use close_connection but could not make it work or it did not change anything. And i think it would not help with the "already blocking" request anyway.

Here is the code i use to reproduce the problem (please uncomment one line or the other at the bottom to pick the working mode)

from concurrent.futures import ThreadPoolExecutor
from http.server import SimpleHTTPRequestHandler, HTTPServer
from threading import Event, current_thread


def t_forever(http_server, _):
 print(".thread", current_thread())
 http_server.serve_forever()
 print('.exit')


def t_single(http_server, trigger):
 print(".thread", current_thread())
 while not trigger.is_set():
 print('-')
 http_server.handle_request()
 print('.exit')


def main(thread, shutdown, timeout):
 trigger = Event()
 print(",main", current_thread())
 HTTPServer.allow_reuse_address = True
 with HTTPServer(('localhost', 8080), SimpleHTTPRequestHandler) as http_server:
 if timeout:
 http_server.timeout = 1
 with ThreadPoolExecutor(max_workers=1) as executor:
 future = executor.submit(thread, http_server, trigger)
 while True:
 try:
 if not future.running():
 print(',break')
 break
 if trigger.is_set() and shutdown:
 print(',shutdown')
 http_server.shutdown()
 print(',endif')
 print(',result')
 return future.result(timeout=1)
 except TimeoutError:
 print(',')
 except KeyboardInterrupt:
 print(',sigint')
 trigger.set()


if __name__ == '__main__':
 # main(t_single, False, True)
 main(t_forever, True, False)

Thanks in advance for your help on allowing a "clean shutdown" with client-idle connections.
Nicolas

[ArpaAP]

@corona10 Let me try to fix this issue.

[JooJooBee666]

I'm currently having major issues due to this with one of my projects. Is there a recommended workaround?