zipfile regression: When writing a zip64 entry to an unseekable file, the local file header compressed/uncompressed fields are not set to 0

[chenxiaolong]

Bug report

When creating a zip file with a zip64 entry in a streaming way (unseekable file), my understanding from the spec is that the headers should be set up so that:

1. The local file header general flags field has bit 3 set (§4.4.4)
2. The local file header crc32, compressed size, and uncompressed size fields are set to 0 (§4.4.4)
3. The local file header has a zip64 (0x0001) extra record so that data descriptor sizes are interpreted as 64-bit integers (§4.3.9.2)
4. A data descriptor header exists (§4.3.9.1)

zipfile normally does all this, but (2) seems to have broken after #103861/#103863. With that change, the local file header's compressed and uncompressed sizes are set to 0xffffffff instead of 0x00000000. (0xffffffff is correct for the usual case where files are seekable and data descriptors are not used.)

From initial testing, it seems like this is all that's needed to fix the issue:

--- zipfile.py	2023-06-08 22:29:05.000000000 -0400
+++ zipfile2.py	2023-06-28 17:50:45.841709476 -0400
@@ -463,8 +463,9 @@
 fmt = '<HHQQ'
 extra = extra + struct.pack(fmt,
 1, struct.calcsize(fmt)-4, file_size, compress_size)
- file_size = 0xffffffff
- compress_size = 0xffffffff
+ if not (self.flag_bits & _MASK_USE_DATA_DESCRIPTOR):
+ file_size = 0xffffffff
+ compress_size = 0xffffffff
 min_version = ZIP64_VERSION
 
 if self.compress_type == ZIP_BZIP2:

To reproduce

import zipfile


class UnseekableFile:
 def __init__(self, fp):
 self.fp = fp

 def write(self, data):
 return self.fp.write(data)

 def flush(self):
 self.fp.flush()


with open('test.zip', 'wb') as f_raw:
 with zipfile.ZipFile(UnseekableFile(f_raw), 'w') as z:
 with z.open('foobar', 'w', force_zip64=True) as f:
 f.write(b'Hello, world!')

The resulting file looks like this, as reported by zipdetails (green=good, red=problematic).

 0000 LOCAL HEADER #1 04034B50
 0004 Extract Zip Spec 2D '4.5'
 0005 Extract OS 00 'MS-DOS'
+0006 General Purpose Flag 0008
+ [Bit 3] 1 'Streamed'
 0008 Compression Method 0000 'Stored'
 000A Last Mod Time 00210000 'Mon Dec 31 19:00:00 1979'
+000E CRC 00000000
-0012 Compressed Length FFFFFFFF
-0016 Uncompressed Length FFFFFFFF
 001A Filename Length 0006
 001C Extra Length 0014
 001E Filename 'foobar'
+0024 Extra ID #0001 0001 'ZIP64'
+0026 Length 0010
+0028 Uncompressed Size 0000000000000000
+0030 Compressed Size 0000000000000000
 0038 PAYLOAD Hello, world!
 
+0045 STREAMING DATA HEADER 08074B50
+0049 CRC EBE6C6E6
+004D Compressed Length 000000000000000D
+0055 Uncompressed Length 000000000000000D
 
 005D CENTRAL HEADER #1 02014B50
 0061 Created Zip Spec 2D '4.5'
 0062 Created OS 03 'Unix'
 0063 Extract Zip Spec 2D '4.5'
 0064 Extract OS 00 'MS-DOS'
 0065 General Purpose Flag 0008
 [Bit 3] 1 'Streamed'
 0067 Compression Method 0000 'Stored'
 0069 Last Mod Time 00210000 'Mon Dec 31 19:00:00 1979'
 006D CRC EBE6C6E6
 0071 Compressed Length 0000000D
 0075 Uncompressed Length 0000000D
 0079 Filename Length 0006
 007B Extra Length 0000
 007D Comment Length 0000
 007F Disk Start 0000
 0081 Int File Attributes 0000
 [Bit 0] 0 'Binary Data'
 0083 Ext File Attributes 01800000
 0087 Local Header Offset 00000000
 008B Filename 'foobar'
 
 0091 END CENTRAL HEADER 06054B50
 0095 Number of this disk 0000
 0097 Central Dir Disk no 0000
 0099 Entries in this disk 0001
 009B Total Entries 0001
 009D Size of Central Dir 00000034
 00A1 Offset to Central Dir 0000005D
 00A5 Comment Length 0000
 Done

Your environment

• CPython versions tested on: 3.11.4
• Operating system and architecture: Alpine Linux x86_64

[chenxiaolong]

I think my understanding of the spec was incorrect. When I read this in §4.4.4:

 Bit 3: If this bit is set, the fields crc-32, compressed 
 size and uncompressed size are set to zero in the 
 local header. The correct values are put in the 
 data descriptor immediately following the compressed
 data. 

I interpreted "set to zero in the local header" as in the two 32-bit size fields in the local header. I believe the spec considers the zip64 extra record to be part of the local header, so the correct interpretation is that the zip64 extra record's sizes are set to 0. This is what Python is doing now.

It also matches what the zip command does:

dd if=/dev/zero bs=1M count=5000 | zip | cat > test.zip

The resulting zip looks like this:

000000 LOCAL HEADER #1 04034B50
000004 Extract Zip Spec 2D '4.5'
000005 Extract OS 00 'MS-DOS'
000006 General Purpose Flag 0008
 [Bits 1-2] 0 'Normal Compression'
 [Bit 3] 1 'Streamed'
000008 Compression Method 0008 'Deflated'
00000A Last Mod Time 56E37E59 'Mon Jul 3 11:50:50 2023'
00000E CRC 00000000
000012 Compressed Length FFFFFFFF
000016 Uncompressed Length FFFFFFFF
00001A Filename Length 0001
00001C Extra Length 0014
00001E Filename '-'
00001F Extra ID #0001 0001 'ZIP64'
000021 Length 0010
000023 Uncompressed Size 0000000000000000
00002B Compressed Size 0000000000000000
000033 PAYLOAD

4DA380 STREAMING DATA HEADER 08074B50
4DA384 CRC 6AA043F2
4DA388 Compressed Length 00000000004DA34D
4DA390 Uncompressed Length 0000000138800000

Given that, I'm closing this issue since it's a misunderstanding on my part.