[Bug]: Issue with project delete (with RedisTarget)

[george0st]

MLRun Version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of MLRun CE.

Reproducible Example

import mlrun
import mlrun.feature_store as fstore
from mlrun.datastore.targets import RedisNoSqlTarget,SQLTarget, CSVTarget, ParquetTarget
from mlrun.data_types.data_types import ValueType
import pandas as pd


def test_delete(project_name):
    feature_name = "basic_party"

    mlrun.set_env_from_file("mlrun-nonprod.env")
    project = mlrun.get_or_create_project(project_name, context='./', user_project=False)

    feature_name = "basic-party"
    feature_set = fstore.FeatureSet(feature_name, entities=[fstore.Entity("party-id", value_type=ValueType.INT64)],
                                    engine="storey")
    feature_set.add_feature(fstore.Feature(name="party-type"))
    feature_set.set_targets(targets=[RedisNoSqlTarget(path="redis://localhost:6379")],
                            with_defaults=False)
    feature_set.save()
    data = {"party-id": [1, 2, 3],
            "party-type": ["a1", "b2", "c3"]}
    dataFrm = pd.DataFrame(data)
    fstore.ingest(feature_set, dataFrm, overwrite=False)

    mlrun.get_run_db().delete_project(project_name, mlrun.common.schemas.DeletionStrategy.cascade)

if __name__ == '__main__':
    test_delete("qqq")

Issue Description

It is not possible to delete project (via mlrun.get_run_db().delete_project) with RedisTarget.

NOTE: The part of code delete_project takes 5 minutes and after that all seems fine (without error/exception) but project still exists in UI.

Expected Behavior

It will be possible to delete project without delay.

NOTE: MLRun 1.5.2 works correctly.

Installation OS

Windows

Installation Method

Docker

Python Version

3.9.10

MLRun Version

1.6.0

Additional Information

No response

[xsqian]

@george0st I can confirm that in a 1.5.2 environment, the project with RedisTarget can be deleted successfully. I will find a 1.6.0 environment to produce your issue shortly. Thanks!

[george0st]

Hi @xsqian , only note, I tested the issue with new MLRun 1.6.1 also (not only under MLRun 1.6.0) and the issue still exist.

NOTE: I saw in cycle these exceptions, see Exception('Background task not in terminal state. name=e74966f3-21b3-401d-9b03-36d593e35ee9, state=running') in the function _wait_for_background_task_to_reach_terminal_state see httpdb.py

[george0st]

See discussion on Slack https://mlopslive.slack.com/archives/C014XCMNY4Q/p1709025666498859

[george0st]

Relation to the possible solution #5258

[alonmr]

Hi @george0st thank you for submitting the issue.
I created a CE environment with minikube and mlrun 1.6.1 and I was able to delete the project successfully with no issues.
Why do you think it is related to RedisNoSqlTarget ?
Either way, the way project deletion works is that the client is querying the background task that deletes the project and waits for it to reach terminal state. Please check the mlrun api chief logs. If any error occur is should be evident there.

[alonmr]

I see you already added the logs in slack. It is possible that the api chief is waiting for some nuclio function deletion that never happens. Do you have any dangling nuclio functions?

[george0st]

Hi @alonmr , I tested two projects one with context.log_model and one without nuclio functions (only feature sets and feature vectors) and I have issue with delete both projects. The xsqian, caught the same issue (when he simulated my issues) also (see the note from him in slack). I tested the solution in desktop docker.

NOTE: it does not have relation to Redis solution. In case of interest, I can show you details directly from debugger, let me know.

[george0st]

Note: the same issue also under MLRun 1.6.2