Use os.listdir() instead of os.scandir() in FileSession during clean-up
#2010
Conversation
os.scandir() is more effcient than os.listdir() and is generator based.
| @@ -578,15 +578,16 @@ def clean_up(self): | |||
| """Clean up expired sessions.""" | |||
| now = self.now() | |||
| # Iterate over all session files in self.storage_path | |||
| for fname in os.listdir(self.storage_path): | |||
| for entry in os.scandir(self.storage_path): | |||
There was a problem hiding this comment.
If we're changing this, let's go for the higher level pathlib interface already rather then working with the low-level os API.
There was a problem hiding this comment.
Surprisingly, when I attempted later on to switch from using os.scandir() to pathlib's iterdir(), the performance significantly deteriorated. I am currently using a Windows server, and the calls to os.listdir() or path.iterdir() hardly ever return, whereas scandir() was able to return and start iterating within approximately 30 seconds. (I have about 500K session files) So I would suggest sticking to os.scandir() for better performance.
There was a problem hiding this comment.
pathlib is the interface of choice for maintainability reasons. Also, using file-based sessions was never intended for high-performance use-cases specifically because file system scans are dependent on the number of files in directories and the lookups aren't exactly linear. Having a 30-second scan suggests that other file system operations during normal runtime aren't quite fast either.
So I'd say it's an architecture/deployment problem on the app side, not the framework. A typical solution is to use session storages that fit better for this purpose — like memcached or redis.
Could you post the timings/metrics showing the difference? And perhaps, report this issue on the upstream CPython tracker?
webknjaz
changed the title
os.listdir() to os.scandir()
jiangwen365
changed the title
os.listdir() to os.scandir()os.listdir() to os.scandir() in FileSession clean up
webknjaz
changed the title
os.listdir() to os.scandir() in FileSession clean upos.listdir() instead of os.scandir() in FileSession during clean-up
os.scandir()is more effcient thanos.listdir()and is generator based.What kind of change does this PR introduce?
What is the related issue number (starting with
#)What is the current behavior? (You can also link to an open issue here)
What is the new behavior (if this is a feature change)?
when cleaning file-based session storage,
os.listdir()returns a huge list of file names for very large session directories. so switching toos.scandir()function – a better and faster directory iterator should be beneficial.Other information:
Checklist:
and description in grammatically correct, complete sentences