Flask-SocketIO emit not working in a start_background_task

[shrinath-gupta-ril]

Hi,
I'm using gunicorn and eventlet. The socketio.emit is not working in start_background_task for client while I can see them in server logs as Sending packet MESSAGE the client is not receiving this event. I think it is related to eventlet but I'm not able to figure it out.

Functional Requirement: An endpoint which returns an event_id and to which our client can listen to.

app.py
@app.route("/my_route", methods=["POST"])
def my_route():
    required_file_params = ["file_one", "file_two"]
    if not validateRequest(required_file_params, request.files):
        return create_bad_request_response()
    event_id: str = create_event_id()
    file_one = request.files["file_one"]
    file_one_save_path: str = create_output_file_path(f"{event_id}-file-1.json")
    file_one.save(file_one_save_path)
    file_two = request.files["file_two"]
    file_two_save_path: str = create_output_file_path(f"{event_id}-file-2.json")
    file_two.save(file_two_save_path)
    output_file_path: str = create_output_file_path(f"{event_id}-output-file.json")
    socketio.start_background_task(
        Service.create_output,
        socketio,
        event_id,
        file_one_save_path,
        file_two_save_path,
        output_file_path,
    )
    return jsonify({"event_id": event_id})

service.py

   @staticmethod
def create_output_file(socketio,event_id:str,file_one_path:str,strings_file_two_path:str,output_file_path:str):
    try:
        create_log_file("log.txt")
        socketio.emit(event_id,EventDetails(status=EventStatus.STARTED).to_Json())
        file_one_json = load_json_from_file(file_one_path)
        file_two_json = load_json_from_file(file_two_path)
        if file_one_json and file_two_json:
            file_one_utils = FileOneUtils(file_one_json)
            socketio.emit(event_id,EventDetails(status=EventStatus.IN_PROGRESS).to_Json())
            # LONG RUNNING PROCESS
            output = file_one_utils.performOperation(file_two_json)
            isOutputFileSaved = save_json(output,output_file_path)
            if isOutputFileSaved:
                socketio.emit(event_id,EventDetails(status=EventStatus.COMPLETED,saved_files=[SavedFile("troubleshoot.json",output_file_path,"json")]).to_Json())
                return
            else:
                socketio.emit(event_id,EventDetails(status=EventStatus.FAILED,error_messages=["Something went wrong!"]).to_Json())
                return
        else:
            socketio.emit(event_id,EventDetails(status=EventStatus.FAILED,error_messages=["Invalid troubleshot or strings file!"]).to_Json())
    except Exception as exception:
        socketio.emit(event_id,EventDetails(status=EventStatus.FAILED,error_messages=[repr(exception)]).to_Json())

[miguelgrinberg]

Long running processes do not work well with eventlet. I recommend you try installing eventlet and switching to the threading async mode to see if things work better.

[shrinath-gupta-ril]

Thanks @miguelgrinberg for your prompt attention. I tried to use threading async mode by setting the following

socketio = SocketIO(app, cors_allowed_origins="*",logger=True, engineio_logger=True,async_mode="threading")
This results in slower http response for example the above endpoint took 29 seconds to respond.
One thing I observed is that given I have not set threading async mode if I add socket.sleep(0) before long running task the initial emits of event start and in progress are received on client but the emit after long running task is not receive I think because of sleep.
Can you please help?

[miguelgrinberg]

You are using eventlet, which does not work well with long running tasks. Your two options are 1) add frequent sleep(0) calls while the long task runs, or 2) use threading instead of eventlet.

[shrinath-gupta-ril]

Adding socketio.sleep(0) in long running process fixes the issue. Thanks @miguelgrinberg