Why do Input Adapters not take effect in test cases ?

[jiyer2016]

I have a simple integration test case that looks like this:

    def test_rank_items(self):
        service = Service()
        service.pack('model', self.model)
        X = [[0, 0], [2, 2], [0, 2]]
        y_true = [0.0, 1.0, 1.0]
        y_pred = service.classify(X)
        self.assertTrue(np.allclose(y_true, y_pred))

This is the Service code.

@bentoml.env(auto_pip_dependencies=True)
@bentoml.artifacts([SklearnModelArtifact('model')])
class Service(bentoml.BentoService):

    @bentoml.api(input=DataframeInput())
    def classify(self, X):
        print(type(X)) # prints <class 'list'> in test cases whereas <class 'pandas.core.frame.DataFrame'> when served via  flask 
        return self.artifacts.model.predict(X)

The question is about this piece of code:
print(type(X)) # prints <class 'list'> in test cases whereas <class 'pandas.core.frame.DataFrame'> when served via flask

Is this on expected lines ?
I was hoping the annotation takes effect in the exact same way in both integration test cases and the real world.

Should I be explicitly passing a pandas dataframe to the API method in the test cases ?

I was hoping NOT because even the minimal BentoService IrisClassifier example in the Getting Started guide seems to suggest that input lists are converted into Dataframes.

@bentoml.env(auto_pip_dependencies=True)
@bentoml.artifacts([SklearnModelArtifact('model')])
class IrisClassifier(bentoml.BentoService):

    @bentoml.api(input=DataframeInput())
    def predict(self, df):
        return self.artifacts.model.predict(df)

# Create a iris classifier service with the newly trained model
iris_classifier_service = IrisClassifier()
iris_classifier_service.pack("model", clf)

# Test invoking BentoService instance
iris_classifier_service.predict([[5.1, 3.5, 1.4, 0.2]])

Could I be doing something wrong ?

What is the best strategy for coding integration tests ?

[parano]

Hey @jiyer2016 - I think the other thread here #1067 probably answered your question here. Adapters do not take effect when calling directly with the Python API. DataframeInput for example, will need the user to provide a pandas.DataFrame object as its input, the quick start example with iris classifier should be used as follow:

import pandas as pd
iris_classifier_service.predict(pd.DataFrame([[5.1, 3.5, 1.4, 0.2]]))

We could potentially make this work and convert arrays to Dataframe for the user, and we actually tried out a version with that approach when building the initial version of BentoML. But users tend to find that approach more confusing, as they expect the BentoService class method should just work the same way how a regular python class method works when being invoked.

We can potentially add a special method for invoking APIs from Python and adding an additional method to InputAdapter class for converting a Python object into the argument expected by the user-defined API function.

E.g.

iris_classifier_service.predict.run([[5.1, 3.5, 1.4, 0.2]]) 

Although I'm not sure how much value that would add. Thoughts?

[jiyer2016]

@parano - For my test cases, I'll just go ahead and do something like the below:
iris_classifier_service.predict(pd.DataFrame([[5.1, 3.5, 1.4, 0.2]]))

That resolves my test cases.
I don't necessarily think we need the special method for converting the python object into the argument expected by the API as long as we can make the user aware that the type conversion isn't automatic (in direct invocations)