Model not training at all when using package

[michagast]

Hey all,

For my own feature extraction method I want to use a package that can detect proper nouns in a text. This feature is very useful for what I am trying to achieve, which is text classification of emails. However, when using this feature within ASReview the model doesn't learn and always predicts the same class for everything. For the feature I am using the averaged pereceptron tagger from the NLTK package link.

The code below is what I am using to import the function:

from nltk.tag import pos_tag   
nltk.download('averaged_perceptron_tagger')

The rest of the code can be found here. I'm quite lost as to why this is happening, so any idea on what could be the cause of this would be very helpful.

Note that I am manually extracting the models each time from the ASReview package using the following snippet of code that is placed within here

with open('model_unbalanced.pkl', 'wb') as file:
    # A new file will be created
    pickle.dump(self.classifier, file)
    print("Succesfully saved model")

Thanks in advance for the help,
Micha

[PeterLombaers]

Hey Micha,
Just to get the picture clear, if I understand what you are saying correctly, you are doing two things:

• You use a feature extraction method that uses a POS tagger to create a feature matrix.
• You save the classifier at every iteration of the review.

Are you saying that the POS tagger predicts the same label all the time? Or does ASReview present you with the same record everytime when you use this feature matrix?

[PeterLombaers]

I'm still a little confused, and maybe the confusion is in how we use 'classifier'. Are you saying that if you let the ASReview classifier (for example NaiveBayesClassifier) predict the labels, then all predictions are 1? In that case you might be slightly misunderstanding what happens under the hood. In ASReview, the classifier doesn't actually classify, but it only produces prediction scores. These are then are used to put the records in a ranked order. ASReview does not interpret these prediction scores (for example by saying prediction_score > 0.5 = relevant), all it does is present the user the highest ranked record. In the end it is the user that does the actual classification by marking the record as relevant or not. So the classifier never predicts a label, it only gives a score.

If in your case the prediction scores are all closer to 1 than to 0, that does not have to be a problem in principle. As long as the relevant records get a higher score than the irrelevant records, ASReview will still present you the relevant records before the irrelevant records. Interpreting the prediction scores is problematic if you are always labeling the record with the highest prediction score, because in that case the training data of the model is not a random sample of the dataset.

Hope this helps, and if I still misunderstood your question, just let me know!

[PeterLombaers]

By the way, this is also why we don't evaluate the performance of different models in ASReview by calculating the prediction accuracy of the final model on the dataset, or some similar metric. Instead we run simulations, and calculate metrics such as WSS (work saved over sampling ).

[michagast]

Thanks for the quick responses @PeterLombaers.
To clarify, I am not performing systematic reviews in ASReview. I am just using it as a tool to perform active learning with to get a good classifier. With that being said, WSS is not the best criterion for me I think. Instead, I want to know what the accuracy, precision and recall are for the model if it predicts new instances. To do this I have added a function to the Logistic Regression model within ASReview which instead of probabilities outputs labels. This function is as follows:

    def predict(self,X):
        return self._model.predict(X)

With this function I am able to generate labels from the model, instead of just probabilties. Using this function I can predict labels for data outside ASReview using pickle. And this is where the model is predicting everything either label 0 or 1, when the feature that makes use of the POS_tagger is part of the feature matrix that this model is trained on.

Thanks in advance for the answer,
Micha

[PeterLombaers]

Ah, I see! Your problem probably isn't with the functioning of the classifier. They are mostly just implementations of sklearn classifiers, so they should behave correctly. Maybe the set on which you predict is just quite different from the set on which you train, or the model got many more 1's than 0's in the training data so now it only predicts 1's, or you need to adjust the hyperparameters, or.... It's really more of a general machine learning question though, and doesn't have very much to do with ASReview itself.

[michagast]

This indeed seems to be the case, thanks for thinking with me though!