tarfile in stream mode always set zlib compression level to 9

[PatrikDufresne]

BPO	26253
Nosy	@gustaebel, @vadmium, @wimglenn, @zhangyangyu, @jarondl
PRs	bpo-26253: Add compressionlevel to tarfile stream #2962

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2016-02-01.00:55:23.398>
labels = ['3.7', 'type-feature', 'library']
title = 'tarfile in stream mode always set zlib compression level to 9'
updated_at = <Date 2017-08-01.17:03:47.325>
user = 'https://bugs.python.org/PatrikDufresne'

bugs.python.org fields:

activity = <Date 2017-08-01.17:03:47.325>
actor = 'jarondl'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2016-02-01.00:55:23.398>
creator = 'Patrik Dufresne'
dependencies = []
files = []
hgrepos = []
issue_num = 26253
keywords = []
message_count = 6.0
messages = ['259304', '259308', '259987', '292628', '297220', '299622']
nosy_count = 6.0
nosy_names = ['lars.gustaebel', 'martin.panter', 'wim.glenn', 'xiang.zhang', 'Patrik Dufresne', 'jarondl']
pr_nums = ['2962']
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue26253'
versions = ['Python 3.7']

[PatrikDufresne]

When using tarfile.open(mode='w|gz'), the compression level is hard-coded to 9. Seed _Stream._init_write_gz():
self.zlib.compressobj(9,

1. In regards to zlib, I would start by replacing the value of 9 by zlib.Z_DEFAULT_COMPRESSION. This is the default value and zipfile is using it. Why using something different.

2. Then, I would also love to control the compression level when calling tarfile.open()

[vadmium]

It looks like the default has been hard-coded to 9 ever since tarfile was added to Python. The gzip module is also hard-coded to 9 since it was added. If tarfile is changed, maybe gzip should too.

Why would you want to use zlib’s default (apparently 6)? Memory usage or speed perhaps? If we do change the default, maybe it is best to only do it in 3.6. I don’t see it as a bug fix, and there is a chance it could break someone’s code.

To be able to control the compression level, perhaps you can already do it by wrapping the tar stream with GzipFile (untested):

gz_writer = GzipFile(fileobj=raw_writer, mode="wb", compresslevel=...)
tar_writer = tarfile.open(fileobj=gz_writer, mode="w|")
tar_writer.addfile(...)
tar_writer.close()
gz_writer.close()

If the default is changed, it certainly makes sense to add an easy compression level parameter, to be able to restore the old behaviour.

[vadmium]

Actually it’s not really obvious from the signatures, but in the middle of the tarfile.open() documentation it says “. . . tarfile.open() accepts the keyword argument _compresslevel_”, so it should already be possible.

[zhangyangyu]

*compresslevel* takes effect for modes 'w:gz', 'r:gz', 'w:bz2', 'r:bz2', 'x:gz', 'x:bz2'. For stream modes, 'r|gz', 'w|gz', 'r|bz2', 'w|bz2', the *compresslevel* doesn't make sense. It seems not hard to make it possible but I'm not sure it's worth it or there is any reason it's hard-coded.

[wimglenn]

This issue also got me. compresslevel kwarg works fine for tarfile.open(..., mode='w:gz') but raises exception for tarfile.open(..., mode='w|gz')

I want to use stream compression, and compresslevel=1 is more than enough for my use case, the default of 9 is way too slow.

[jarondl]

I have submitted a PR on GitHub #2962