Welcome, guest | Sign In | My Account | Store | Cart

Copy huge files on Windows avoiding system cache misbehaviour (Python recipe) by Christos Georgiou
ActiveState Code (http://code.activestate.com/recipes/496694/)

This recipe uses pywin32 in order to copy files making use of the win32file.FILE_FLAG_SEQUENTIAL_SCAN, which makes much more efficient use of the system cache (only a few megabytes are consumed instead of caching the whole file).

Python, 30 lines
Download
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
import win32file

def copy_big_file(srcname, dstname):
	hin= win32file.CreateFile(
		srcname,
		win32file.GENERIC_READ,
		0, # win32file.FILE_SHARE_READ,
		None,
		win32file.OPEN_EXISTING,
		win32file.FILE_FLAG_SEQUENTIAL_SCAN,
		0)
##	print "type of hin=%r" % type(hin)
	hou= win32file.CreateFile(
		dstname,
		win32file.GENERIC_WRITE,
		0, # win32file.FILE_SHARE_READ,
		None,
		win32file.CREATE_ALWAYS,
		win32file.FILE_FLAG_SEQUENTIAL_SCAN,
		0)
	while 1:
		rc, buffer= win32file.ReadFile(hin, 65536)
		if not buffer: break
		if rc == 0:
			win32file.WriteFile(hou, buffer)
		else:
			print "rc=%d" % rc
			break
	hin.Close()
	hou.Close()

Copying files around won't swap out all your running applications from now on :)

Create a command line script or embed it in your GUI file browser.

Reference: http://support.microsoft.com/default.aspx?scid=kb;EN-US;98756

Tags: files

1 comment

Kai Duebbert 17 years, 3 months ago  # | flag

Not quite sure what's better than win32file.CopyFile. I've tried this function and did some benchmarks with a 5GB file. The win32file.CopyFile function was consistently 40-50% faster than this function and over 300% faster than shutil.copyfile.

In which situations is your function faster than win32file.CopyFile?