Dict memory leak in Python 3.12.x / msgpack 1.0.x#612
Description
Hello,
I think I've found a second Python 3.12 memory leak (having read the issue for the other one, and tried upgrading to msgpack 1.0.8 to fix).
This issue occurs with dicts, and key uniqueness seems to be a major consideration. (A bunch of dicts with the exact same keys don't cause the problem -- some kind of key caching issue maybe?)
Here's a repro:
import psutil import msgpack # Set up a list of packed dicts p = [ msgpack.packb({f'{p}.{r}': True for r in range(1000)} ) for p in range(1000) ] # Check initial state print(f'initial mem:{psutil.Process().memory_info().rss}') # Loop and unpack i = 0 for x in p: y = msgpack.unpackb(x) i += 1 if i % 100 == 0: print(f'subsequent mem:{psutil.Process().memory_info().rss}') Output, msgpack 1.0.8 on Python 3.12.3:
initial mem: 24571904 subsequent mem: 34029568 subsequent mem: 42627072 subsequent mem: 47493120 subsequent mem: 60088320 subsequent mem: 64684032 subsequent mem: 69550080 subsequent mem: 89878528 subsequent mem: 94744576 subsequent mem: 99340288 subsequent mem: 104206336 Output, msgpack 0.6.2 on Python 3.12.3 or msgpack 1.0.8 on Python 3.11.9 (slightly different numbers but same effect):
initial mem: 22761472 subsequent mem: 23089152 subsequent mem: 23089152 subsequent mem: 23089152 subsequent mem: 23089152 subsequent mem: 23089152 subsequent mem: 23089152 subsequent mem: 23089152 subsequent mem: 23089152 subsequent mem: 23089152 subsequent mem: 23089152 p.s. apologies for using psutil in the script, I didn't remember until later that it isn't a Python built-in.