Preload deep learning model

[Houd1ny]

I have deep learning model that loads into GPU
I want to preload and reuse it every time I execute job.
I have found https://python-rq.org/docs/workers/#performance-notes, but don't know how to access objects from job code
custom_worker.py

import sys
from rq import Connection, Worker, SimpleWorker

import cv2
import numpy as np
from cv.cnn_model import init_model, predict
__model = init_model("config.json", "weights.data")#loads to GPU

with Connection():
    qs = sys.argv[1:] or ['default']

    w = SimpleWorker(qs)
    w.work()

job.py

def detect(image_path):    
    image = cv2.imread(image_path)
    data = predict(__model, image) #should use preloaded __model
    return data

[Houd1ny]

I have found a solution using a custom worker to pass loaded model to job function
but this looks ugly!

class CustomWorker(SimpleWorker):
    def __init__(self, *args, **kwargs):
    	self.__model = init_model("config.json", "weights.data")
        SimpleWorker.__init__(self, *args, **kwargs)

    def execute_job(self, job, queue):
        job.args = (job.args[0], self.__model)
        return self.perform_job(job, queue)

job.py

def detect(image_path, model):    
    image = cv2.imread(image_path)
    data = predict(model, image) #used preloaded model
    return data

[hnykda]

TBH I don't know if you could do any better. This is probably how I would do it as well. Why do you think it's ugly?

[Houd1ny]

@hnykda It is ugly because I`m violating encapsulation of Worker class changing its args variable. Originally it should represent values passes from the caller "result = q.enqueue(predict, "1.jpeg")" There only one parameter!
but I have found better solution

[Houd1ny]

Finally, I understood how this should work. RQ uses importlib.import_module to load function. So every time you call the function it loads jobs file. So as I understand to fix this, you should load this file BEFORE worker execution.

import library_that_you_want_preloaded

should be replaced with

from job import predict

not this

import cv2
import numpy as np
from cv.cnn_model import init_model, predict
__model = init_model("config.json", "weights.data")#loads to GPU

My final script looks like this
job.py

import cv2
import numpy as np
from cv.cnn_model import init_model, predict


def init_weights():
    global __model
    __model =init_model("config.json", "weights.data")

def predict(image_path):    
    image = cv2.imread(image_path)
    data = predict(__model, image)
    return data

worker.py

import sys
from rq import Connection, SimpleWorker
from job import predict, init_weights

with Connection():
    qs = sys.argv[1:] or ['default']
    init_weights()#I initialize model before workers start
    w = SimpleWorker(qs)#I use SimpleWorker because it does not fork
    w.work()

[xNul]

@Houd1ny I spent many hours trying to figure this out with no luck until I found this issue. Thanks for the help!

[mihailyanchev]

@Houd1ny thanks for the insight, this approached worked for me as well.

Sorry for opening the topic again, but I had an additional question.
I also found that using a normal worker with fork crashed on both MacOS and Ubuntu (haven't checked Windows yet) and using a SimpleWorker fixed the issue. Do you know if there are any caveats about using the SimpleWorker? From the class docstring of Worker it says that it spawns a fail-safe child process (I guess the work horse), so I guess SimpleWorker is not as fail-safe.

[mihailyanchev]

By the way, there is also a similar solution here: #720 where you override the work method of a worker. It works with both the normal Worker and the SimpleWorker.

class CustomWorker(Worker):
"""Custom worker class for pre-loading models."""

def work(self, *args, **kwargs):
     """Override the work method to preload a model."""
     model.load() # some model loading
     return super(Worker, self).work(*args, **kwargs)```

[Harsh-Maheshwari]

@mihailyanchev @Houd1ny

If we add the model.load() inside the work function, the model will be loaded every time right ? Can we add it in the __init__ function of the CustomWorker ? So that If I keep adding jobs to the Queue model will not have to be loaded every time?

[mihailyanchev]

@Harsh-Maheshwari You are correct. I did not find a way to have it work with the __init__ function. But what worked is using a singleton class, which is instantiated only once and every next time you are fetching the same instance.

class ModelClass():
    def __init__(self):
    # load some model objects

    @classmethod
    @lru_cache
    def load(cls):
        return(cls)

Then you can have the CustomWorker load it only once at startup and then fetch the same instance every single time:

import ModelClass

class CustomWorker(SimpleWorker):
    """Custom worker class for pre-loading models."""

   def work(self, *args, **kwargs):
       """Override the work method to preload a model."""
       _ = ModelClass.load() # some model loading
       return super(SimpleWorker, self).work(*args, **kwargs)

Note: use the SimpleWorker instead of the Worker class to inherit from. I ran into some crashed due to forking of the Worker.