How to vectorize text?

[gilnahmias]

Hi, thanks for the great library!

My CSV has text in some of the columns. Some of them are categorical (e.g. month of the year) and some have free text (e.g., book title). Looks like SharpLearning.InputOutput.Csv.CsvRowExtensions.ToF64Matrix is trying to parse stringified numbers. What if my CSV consists of non-number values? Is there a recommended way or should I wire another lib to do TF-IDF/word2vec/char embedding/etc?

[mdabros]

Hi @gilnahmias ,

Thanks for using SharpLearning, I am glad you like the library.

There are some limited methods for handling categorical data in the SharpLearning.FeatureTransformations project. You can see how it works together with the CsvParser in this tests: MapCategoricalFeaturesTransformer

However, there is currently no support for more advaned stuff like TF-IDF/word2vec/char embedding/etc. So here you would need a separate library. Currently i am not sure what the best option in C#/.Net is.

Microsoft has recently released Machine Learning for .Net, they have a simple example dealing with sentiment analysis on their main page, so this might be a good alternative.

If you find a good solution or library please let me know here, then we could look into creating some convenience methods for interfacing with SharpLearning.

Best regards
Mads