Direct Disk Reads

[keshavvinayak01]

Is there a way to use the Image.open API call with the O_DIRECT flag?

I'm working on a project with application caching and I want to prevent all file-backed pages to load into the page cache (wasteful memory in my case) while calling the Image.open API . Let me know where I could modify the code-base to add this specific feature, if possible.

Thanks.

[radarhere]

If you look at the source code for Image.open,

Pillow/src/PIL/Image.py

Line 3050 in 65c832d

def open(fp, mode="r", formats=None):

the first parameter determines how the file is opened.

Pillow/src/PIL/Image.py

Lines 3060 to 3063 in 65c832d

	:param fp: A filename (string), pathlib.Path object or a file object.
	The file object must implement ``file.read``,
	``file.seek``, and ``file.tell`` methods,
	and be opened in binary mode.

If it is a filename, then Python is used to open the file, otherwise the file object passed in is used for subsequent read operations.

Pillow/src/PIL/Image.py

Lines 3099 to 3109 in 65c832d

	if filename:
	fp = builtins.open(filename, "rb")
	exclusive_fp = True
	try:
	fp.seek(0)
	except (AttributeError, io.UnsupportedOperation):
	fp = io.BytesIO(fp.read())
	exclusive_fp = True
	prefix = fp.read(16)

See also https://pillow.readthedocs.io/en/stable/handbook/tutorial.html#reading-from-an-open-file

So I suspect you don't need to modify Pillow - you just need to open your file with O_DIRECT in Python, and then provide Image.open with an object to use to read data from it.

[keshavvinayak01]

This is what I tried as per your suggestion:

def pil_loader(path):
    image_fd = os.open(path, os.O_RDONLY | os.O_DIRECT | os.O_SYNC)
    image_mmap = mmap.mmap(image_fd, os.path.getsize(path), offset=0, access=mmap.ACCESS_READ)
    img = Image.open(image_mmap)
    to_ret = img.convert('RGB')
    os.close(image_fd)
    return to_ret

This still leads to the size of buff/cache (Obtained from free) to increase, as well as the vmtouch utility also showed the images to be loaded into the cache.

Let me know what I should change.

[radarhere]

If you exclude Pillow for the moment, does the code you've provided perform in the way that you want?

def pil_loader(path):
    image_fd = os.open(path, os.O_RDONLY | os.O_DIRECT | os.O_SYNC)
    image_mmap = mmap.mmap(image_fd, os.path.getsize(path), offset=0, access=mmap.ACCESS_READ)
    image_mmap.read()

If not, then this is a general Python problem, not something specific to Pillow.

[keshavvinayak01]

No it does not. Indeed it looks like a python specific problem, very strange.

[keshavvinayak01]

For future reference, I got it working. The following code achieves non-buffered reads:

image_fd = os.open(path, os.O_RDONLY | os.O_DIRECT)
fo = os.fdopen(image_fd, 'rb+', buffering=0)
image_mmap = mmap.mmap(-1, (1 << 24))
fo.readinto(image_mmap)
img = Image.open(image_mmap)

Can read about fdopen and mmap to understand why this works, thanks :)

[radarhere]

Thanks for posting the solution!