简介

基于java，scala，spark的机器学习相关算法

相较于python的优势

1. 大部分线上服务以java为主，因此模型服务与现有的服务结合比较方便
   
2. 可以利用spark处理大规模数据，spark对scala的支持优于python

DEMO 目录

• 逻辑回归

• 最大熵模型及自定义实现

• 决策树分类及调优总结(CART树)

• spark GBDT算法

• spark 随机森林算法

• 线性支撑向量机

• xgboost for scala

• xgboost for spark

• LDA 聚类

• spark生成tfRecord文件以及python读取的demo

• 二分类的ROC曲线、PR曲线、阈值与PR曲线、阈值F1曲线绘制

• 多分类评估指标计算,精确度、加权准确率召回率、F1值、各类别的准确率-召回率-F1值

• Ansj分词

• JieBa分词

• 自定义简易分词

• 数据标准化与归一化

• TF-IDF计算

• 生成词向量

• 连续特征离散化

• 计算编辑距离

• SimHash算法

• PCA降维

• TSNE降维

• fastText for java训练

• fastText for java 词向量模型使用

• java 绘制点、线、柱状图

• deeplearning4j - 单机前向传播神经网络

• deeplearning4j - 单机卷积神经网络

• deeplearning4j - spark版本卷积神经网络

• deeplearning4j - 单机循环神经网络

• deeplearning4j - spark版本循环神经网络

• deeplearning4j - 词向量使用

开源引用

License

BSD

References

(From fastText's references)

Please cite 1 if using this code for learning word representations or 2 if using for text classification.

Enriching Word Vectors with Subword Information

[1] P. Bojanowski*, E. Grave*, A. Joulin, T. Mikolov, Enriching Word Vectors with Subword Information

@article{bojanowski2016enriching,
  title={Enriching Word Vectors with Subword Information},
  author={Bojanowski, Piotr and Grave, Edouard and Joulin, Armand and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1607.04606},
  year={2016}
}

Bag of Tricks for Efficient Text Classification

[2] A. Joulin, E. Grave, P. Bojanowski, T. Mikolov, Bag of Tricks for Efficient Text Classification

@article{joulin2016bag,
  title={Bag of Tricks for Efficient Text Classification},
  author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1607.01759},
  year={2016}
}

FastText.zip: Compressing text classification models

[3] A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jégou, T. Mikolov, FastText.zip: Compressing text classification models

@article{joulin2016fasttext,
  title={FastText.zip: Compressing text classification models},
  author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Douze, Matthijs and J{\'e}gou, H{\'e}rve and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1612.03651},
  year={2016}
}

(* These authors contributed equally.)