shutil.copy() inefficient implementation in Windows

[sfmc]

BPO	44579
Nosy	@pfmoore, @tjguk, @zware, @eryksun, @zooba

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-07-07.12:11:29.301>
labels = ['3.8', 'OS-windows', 'performance']
title = 'shutil.copy() inefficient implementation in Windows'
updated_at = <Date 2021-07-07.14:39:41.523>
user = 'https://bugs.python.org/sfmc'

bugs.python.org fields:

activity = <Date 2021-07-07.14:39:41.523>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Windows']
creation = <Date 2021-07-07.12:11:29.301>
creator = 'sfmc'
dependencies = []
files = []
hgrepos = []
issue_num = 44579
keywords = []
message_count = 3.0
messages = ['397076', '397077', '397091']
nosy_count = 6.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'sfmc']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue44579'
versions = ['Python 3.8']

Linked PRs

• gh-88745: Add _winapi.CopyFile2 and update shutil.copy2 to use it #105055
• [3.12] gh-88745: Add _winapi.CopyFile2 and update shutil.copy2 to use it (GH-105055) #105086

[sfmc]

In Windows shutil.copy() uses _copyfileobj_readinto which copies file in user mode.
In Windows there is an fast API to copy file in kernel mode: CopyFile (see https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-copyfile).

[eryksun]

In Windows there is an fast API to copy file in kernel mode: CopyFile

The possibility of calling CopyFileEx() for shutil.copy2() is discussed in bpo-30044. Note that CopyFileEx() is a high-level Windows API function, not a "kernel mode" copy. It opens the source and destination files and makes multiple system calls in order to copy file data and metadata (e.g. system calls such as NtOpenFile, NtCreateFile, NtReadFile, NtWriteFile, NtQueryInformationFile, NtSetInformationFile, NtQueryEaFile, NtSetEaFile, NtQuerySecurityObject, etc). This includes copying the primary data stream, alternate data streams, file attributes, extended file attributes, and security resource attributes.

[zooba]

Note that CopyFileEx() is a high-level Windows API function, not a "kernel mode" copy.

This is true today, but could change whenever Windows feels like changing it. If we switch to the native API then we'll get any advantage there automatically.

The only challenge is in managing changes to semantics (that is, anything extra we do to "match" Unix that isn't normally how copies work on Windows - personally, I'd rather be more native on Windows anyway).