Python 3.12 breaks backwards compatibility for logging configuration

[egor-tensin]

Bug report

Bug description:

This worked fine on previous versions:

import logging
import logging.config
import logging.handlers
import multiprocessing as mp


def main():
 config ={
 'version': 1,
 'handlers':{
 'sink':{
 'class': 'logging.handlers.QueueHandler',
 'queue': mp.get_context('spawn').Queue(),
 },
 },
 'root':{
 'handlers': ['sink'],
 'level': 'DEBUG',
 },
 }
 logging.config.dictConfig(config)


if __name__ == '__main__':
 main()

With Python 3.12, it drops an error:

Traceback (most recent call last):
 File "/home/egor/.pyenv/versions/3.12-dev/lib/python3.12/logging/config.py", line 581, in configure
 handler = self.configure_handler(handlers[name])
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/egor/.pyenv/versions/3.12-dev/lib/python3.12/logging/config.py", line 786, in configure_handler
 raise ValueError('No handlers specified for a QueueHandler')
ValueError: No handlers specified for a QueueHandler

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "/home/egor/workspace/personal/cimple/../test.py", line 25, in <module>
 main()
 File "/home/egor/workspace/personal/cimple/../test.py", line 21, in main
 logging.config.dictConfig(config)
 File "/home/egor/.pyenv/versions/3.12-dev/lib/python3.12/logging/config.py", line 912, in dictConfig
 dictConfigClass(config).configure()
 File "/home/egor/.pyenv/versions/3.12-dev/lib/python3.12/logging/config.py", line 588, in configure
 raise ValueError('Unable to configure handler '
ValueError: Unable to configure handler 'sink'

More than that, even the example in the Logging Cookbook is broken now: https://docs.python.org/3/howto/logging-cookbook.html#a-more-elaborate-multiprocessing-example (fails with the same error).

Version info:

# python -VV
Python 3.12.0+ (heads/3.12:f108785, Nov 1 2023, 19:47:19) [GCC 13.2.1 20230801]

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

• gh-111615: Fix regression in QueueHandler configuration. #111638
• [3.12] gh-111615: Fix regression in QueueHandler configuration. (GH-111638) #113507

[vsajip]

Thanks for the report. This is due to a fix for gh-93162 , which allows proper configuration of a QueueListener and QueueHandler together. I'm sorry about the breakage - it appears not enough consideration was given to configuring a Queuehandler without configuring a QueueListener. This was of course the default case before the change went in, and the cookbook example needs to be updated. How are you configuring the QueueListener in your environment?

[egor-tensin]

I'm using QueueListener/QueueHandler w/ multiprocessing to centralize logging in the parent process logger, which I think is common advise on the net. The complete setup is:

@contextmanager
def configure_logging_in_child(queue):
 config ={
 'version': 1,
 'handlers':{
 'sink':{
 'class': 'logging.handlers.QueueHandler',
 'queue': queue,
 },
 },
 'root':{
 'handlers': ['sink'],
 'level': 'DEBUG',
 },
 }
 logging.config.dictConfig(config)
 try:
 yield
 except Exception as e:
 logging.exception(e)

def child_main(log_queue):
 with configure_logging_in_child(log_queue):
 # Do some actual work.
 ...

def main():
 ...
 ctx = multiprocessing.get_context('spawn')
 queue = ctx.Queue()
 listener = logging.handlers.QueueListener(queue, logging.getLogger())
 listener.start()
 # There's actually a list of child processes.
 args = (queue,)
 child = ctx.Process(target=child_main, args=args)
 child.start()
 child.join()
 ...

To be honest, I don't really understand how the queue & listener can be configured in one call, since I configure them separately in the parent & child processes.

[vsajip]

To be honest, I don't really understand how the queue & listener can be configured in one call

Well, they can be if you're using threads rather than processes.

[egor-tensin]

Well, they can be if you're using threads rather than processes.

I understand. I cannot use Python threads though, since I need actual parallel execution (for a number of child processes).

I use a similar pattern in a couple of other projects, where I also must use multiprocessing, since the child process needs to drop permissions, execute as a different user, etc.

[vsajip]

Well, the change in the linked PR should resolve this, but it will have to wait for the next release of Python 3.12.

[egor-tensin]

Well, the change in the linked PR should resolve this, but it will have to wait for the next release of Python 3.12.

Thanks, sounds good!